KR102354704B1 - Automatic color map expression method by removing outlier temperature for thermal image - Google Patents

Abstract

Disclosed is an automatic color map expression method through outlier temperature removal. A method for expressing a color map according to an embodiment includes: obtaining 3D thermal image information; recognizing objects using a deep learning-based learning model learned to recognize objects from the obtained three-dimensional thermal image information; removing outliers through comparison of temperature information between the recognized objects; adjusting a color map for 3D thermal image information based on the temperature information from which the outlier is removed; and correcting 3D thermal image information based on the adjusted color map.

Description

Automatic color map expression method by removing outlier temperature from thermal image {AUTOMATIC COLOR MAP EXPRESSION METHOD BY REMOVING OUTLIER TEMPERATURE FOR THERMAL IMAGE}

The following description relates to a method and system for adjusting the color map of a thermal image through outlier temperature removal.

When an image is taken through an infrared camera, it is often seen with a general visible light camera because of the extreme temperature (cryogenic temperature, extremely high temperature) of a small area, but it is difficult to identify surrounding objects with an infrared camera. 1 is an example of an infrared image due to an extreme temperature. As shown in FIG. 1 , a visibility problem occurs, in which only the heating object is visible and the surrounding objects are not visible.

2 is an example for explaining a method of adjusting a color map for an image. When a visibility problem occurs, the user arbitrarily adjusts the expression range of the color map through the camera settings or manually manipulates the visualized temperature range to solve the problem. When the user manually operates the camera, the camera can be operated only in a specific situation, and the temperature value at which the range of the color map is expressed is limited within the range set by the user. In addition, in an environment in which the temperature continuously changes, the temperature range must be manually operated again, and when a user other than the user of the camera monitors the camera, there is a risk that an image expressed with an incorrect color map may be captured.

It is possible to provide a method and system for automatically removing outliers through temperature comparison of objects extracted from 3D thermal image information.

It is possible to provide a method and system for automatically adjusting a color map of 3D thermal image information through outlier temperature removal.

The color map expression method includes: obtaining three-dimensional thermal image information; recognizing objects using a deep learning-based learning model learned to recognize objects from the obtained three-dimensional thermal image information; removing outliers through comparison of temperature information between the recognized objects; adjusting a color map for 3D thermal image information based on the temperature information from which the outlier is removed; and correcting 3D thermal image information based on the adjusted color map.

The step of recognizing the object may include: the obtained 3D thermal image based on RGB data included in the obtained 3D thermal image information, infrared sensor values of the 3D thermal image information, and measurement information metadata The method may include converting information into temperature information, and extracting temperature information of an object based on the 3D thermal image information converted into the temperature information.

The step of recognizing the object may include displaying a temperature distribution of the 3D thermal image information converted into the temperature information as a histogram, and a box plot on the temperature distribution of the 3D thermal image information shown through the histogram. may include the step of creating

The step of removing the outlier may include removing an outlier existing outside the interquartile range through a box plot generated with respect to the temperature distribution of the 3D thermal image information, and using the removed outlier as a maximum value and a minimum value within the interquartile range. It may include the step of substituting with

The step of removing the outlier includes calculating a maximum value and a minimum value within the quartile by setting a margin value outside the quartile range and multiplying the difference value between the first quartile and the third quartile by the set margin value. can do.

The color map expression method includes obtaining three-dimensional thermal image information by using a three-dimensional thermal imaging camera device, and the obtaining of the three-dimensional thermal image information includes at least one including a thermal imaging camera obtaining stereo image information photographed using the above cameras; calculating distance information of an object based on the obtained stereo image information; and mapping the calculated distance information of the object or the obtained stereo image information with thermal image information to generate a 3D thermal imaging camera image.

The three-dimensional thermal imaging camera device includes a combination of a plurality of CCD cameras including a first camera and a second camera and at least one thermal imaging camera, wherein the generating includes the plurality of CCDs The stereo image information photographed using the camera is mapped with the thermal image image information photographed using the thermal imaging camera, and the photographed thermal image information is convolved in a deep learning-based learning model and then pooled. ) and de-convolve the thermal image information considered as the pooled image after the convolution through the deep learning-based learning model to generate a three-dimensional thermal image. may include

In the generating step, the deep learning-based learning model is trained using the color information included in the thermal image information, and the stereo image information obtained through the CCD camera is applied to the learned deep learning-based learning model. The method may include mapping the stereo image information with color information included in the thermal image information through deconvolution according to input.

The three-dimensional thermal imaging camera device includes a combination of a plurality of CCD cameras including a first camera and a second camera and at least one thermal imaging camera, wherein the generating includes the plurality of CCDs The method may include generating 3D thermal image information by mapping object distance information calculated from stereo image information captured using a camera to the thermal image information.

The three-dimensional thermal imaging camera device includes a combination of a plurality of CCD cameras including a first camera and a second camera and at least one thermal imaging camera, wherein the generating includes the plurality of CCDs When the resolution of the camera and the at least one thermal imaging camera is the same, the distance information of the object calculated from the stereo image information obtained using two or more of a CCD camera or a thermal imaging camera is mapped to the thermal image information It may include generating 3D thermal image information.

The color map representation system includes: an image acquisition unit configured to acquire three-dimensional thermal image information; an object recognition unit for recognizing objects using a deep learning-based learning model learned to recognize objects from the obtained three-dimensional thermal image information; an outlier removal unit for removing an outlier through comparison of temperature information between the recognized objects; a color map adjusting unit that adjusts a color map for 3D thermal image information based on the temperature information from which the outlier is removed; and an image correction unit for correcting 3D thermal image information based on the adjusted color map.

The object recognition unit is configured to convert the obtained 3D thermal image information based on RGB data included in the obtained 3D thermal image information, infrared sensor values of the 3D thermal image information, and measurement information metadata. information, and the temperature information of the object may be extracted based on the 3D thermal image information converted into the temperature information.

The object recognition unit represents a temperature distribution for the three-dimensional thermal image information converted into the temperature information as a histogram, and generates a box plot on the temperature distribution of the three-dimensional thermal image information displayed through the histogram. can

The outlier removal unit removes an outlier existing outside the interquartile range through a box plot generated with respect to the temperature distribution of the three-dimensional thermal image information, and replaces the removed outlier with the maximum and minimum values within the interquartile range. can

The outlier removing unit may set a margin value outside the quartile range and calculate a maximum value and a minimum value within the quartile by multiplying the difference value between the first quartile and the third quartile by the set margin value.

When the temperature information between objects recognized from the 3D thermal image information is too low compared to the temperature information of specific objects, so that it is difficult to distinguish them by color, the outlier can be automatically removed.

In addition, the visibility problem can be solved by adjusting the color map of the 3D thermal image information through the removal of the outlier temperature.

In addition, the image quality can be improved by correcting the 3D thermal image information through color map adjustment to solve the problem that the image was darkened and difficult to analyze in the case of extreme fever in a very small area.

1 is an example of an infrared image due to an extreme temperature.
2 is an example for explaining a method of adjusting a color map for an image.
3 is a diagram for explaining the general operation of a color map representation system according to an embodiment.
4 is a block diagram illustrating a configuration of a color map representation system according to an embodiment.
5 is a flowchart illustrating a color map expression method performed in a color map expression system according to an embodiment.
6 is an example illustrating a temperature distribution for 3D thermal image information as a histogram in the color map representation system according to an embodiment.
7 is a diagram for describing a quartile range in a color map representation system according to an embodiment.
8 is an example illustrating a box plot in a color map representation system according to an embodiment.
9 is an example of adjusting a color map of 3D thermal image information in the color map expression system according to an embodiment.
10 is a flowchart illustrating a method of generating 3D thermal image information in a color map representation system according to an exemplary embodiment.
11 is an example for explaining generation of 3D thermal image information in the color map representation system according to an embodiment.
12 is another example for explaining generation of 3D thermal image information in the color map representation system according to an embodiment.
13 is an example for explaining a deep learning-based learning model of a color map representation system according to an embodiment.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings.

3 is a diagram for explaining the general operation of a color map representation system according to an embodiment.

The color map representation system may acquire 3D thermal image information. The color map expression system may acquire image information of a 3D thermal imaging camera photographed using a 3D thermal imaging camera. At this time, the 3D thermal imaging camera may provide the 3D thermal imaging camera image information by itself, or by performing image processing technology on the image information photographed using at least one camera, the 3D thermal imaging camera image information can provide Referring to FIG. 10 , it is a flowchart illustrating a method of generating image information of a 3D thermal imaging camera. The three-dimensional thermal imaging camera may acquire stereo image information photographed using at least one or more cameras including a thermal imaging camera (1010), and calculate distance information of an object based on the obtained stereo image information (1020) ), the calculated distance information of the object or the obtained stereo image information may be mapped with thermal image information to generate a 3D thermal camera image ( 1030 ).

11 is an example for explaining generation of 3D thermal image information. The 3D thermal imaging camera apparatus may acquire stereo image information by photographing using the plurality of cameras 1110 and 1111 . In this case, different types or a plurality of cameras of the same type may be used. For example, the 3D thermal imaging camera device may be configured in a form in which a plurality of CCD cameras and at least one thermal imaging camera are combined. The 3D thermal imaging camera device may include a plurality of CCD cameras (modules) and at least one thermal imaging camera.

A CCD camera is a digital camera, and is a device that converts an image into an electric signal using a charge-coupled device (CCD), and stores digital data in a storage medium such as a flash memory. CCD cameras have the advantage of good image quality and excellent noise and afterimage processing effects. The 3D thermal imaging camera device may convert the data format of stereo image information obtained from the CCD camera into YUV or RGB mode. For example, the stereo image information obtained from the 3D thermal imaging camera device is buffered by outputting 8 bits of Y (luminance signal) and C (color difference signal) in YCbCr 8 bits, and thermal image information and image data. After tilting to adjust the position, it may be fused with the thermal image information.

A thermal imaging camera is a camera device that measures the temperature of an object (object, person, etc.). The thermal imaging camera can represent the temperature of the object reflected through the camera lens as a color image without bringing the camera into contact with the object. In this case, an infrared sensor may be present in the thermal imaging camera. In a thermal imaging camera, an object emits infrared rays above absolute zero. At this time, the infrared wavelength band is transmitted in the air, so radiant energy with minimized heat loss is transmitted to the thermal imaging camera detector, and the detector is converted into an electrical signal to change the temperature of the object. can be converted to a color image. A thermal imaging camera can display the temperature elevation of an object through the thermal resolution function in the infrared sensor. For example, the thermal imaging camera may display white or yellow when the temperature of an object is higher than or equal to a preset standard, and as the temperature decreases, go through orange and red, and display a very low area in purple or black. For example, thermal image information captured by a thermal imaging camera is converted into an image corresponding to a temperature value through sampling and data conversion to generate thermal image information. Sampling that converts the raw data input once into 14-bit data, NUC (non-uniformity correction) that corrects the non-uniform value of each pixel of the thermal image, and DPC that corrects using normal pixels around the dead pixel (dead pixel correction), NR (noise reduction), BIT conversion that converts 14-bit data to 8-bit data, Upscale that magnifies 2 times with digital zoom, and Color Palette that applies color corresponding to temperature to black and white thermal image After roughing, Y (luminance signal) and C (color difference signal) are converted into 8 bits each and fused with the converted stereo image information.

The 3D thermal imaging camera apparatus may generate a 3D thermal imaging camera image by mapping the calculated distance information of the object or the acquired stereo image information with thermal image information. In this case, stereo image information from which depth information can be acquired may be acquired.

Specifically, referring to FIG. 11 , the 3D thermal imaging camera apparatus may acquire stereo image information captured by using at least one camera including a thermal imaging camera. For example, the 3D thermal imaging camera device may be configured in a form in which a plurality of CCD cameras including a first camera and a second camera and a thermal imaging camera are combined. In this case, in the CCD camera, the first camera may photograph the left side and the second camera may photograph the right side. Stereo image information photographed on the left and stereo image information photographed on the right may be acquired. Alternatively, the first camera and the second camera may photograph the upper and lower sides as well as the left and right.

The 3D thermal imaging camera apparatus may determine whether resolutions of a plurality of CCD cameras and at least one thermal imaging camera are the same. When the resolution of the plurality of CCD cameras and the thermal imaging camera is the same, the 3D thermal imaging camera device may display object distance information calculated from stereo image information obtained using two or more of the CCD camera or the thermal imaging camera. It is possible to generate 3D thermal image information by mapping the image information. For example, the 3D thermal imaging camera apparatus may acquire stereo image information photographed using a plurality of thermal imaging cameras. Alternatively, the 3D thermal imaging camera apparatus may acquire stereo image information photographed using a thermal imaging camera and a CCD camera. Alternatively, the 3D thermal imaging camera apparatus may acquire stereo image information photographed using a plurality of CCD cameras. The three-dimensional thermal imaging camera device calculates distance information of an object based on the obtained stereo image information, and maps the calculated distance information of the object or the obtained stereo image information with thermal image information to obtain a three-dimensional thermal imaging camera image can create

As follows, the 3D thermal imaging camera apparatus may calculate distance information of an object from stereo image information. The three-dimensional thermal imaging camera apparatus can remove radial distortion and tangential distortion of lenses of each CCD camera/thermal imaging camera. The 3D thermal imaging camera device can adjust the angle and distance between two cameras. The 3D thermal imaging camera device may detect a corresponding point having the same characteristics in the left stereo image information and the right stereo image information, and derive a disparity map through the detected corresponding point. The 3D thermal imaging camera device may convert a parallax map into distance information using trigonometry based on a state in which the camera is placed. This process is called re-projection, and through this, a depth map can be obtained. The 3D thermal imaging camera apparatus may perform stereo adjustment on stereo image information obtained through a plurality of cameras. For example, two cameras can re-project the image plane and create it to be positioned on the same plane, and the rows of images to be created to be aligned in a precise front-parallel configuration.

The 3D thermal imaging camera apparatus may use a stereo geometric equation that the depth of a point in 3D space is inversely proportional to the gap between the left and right cameras for the point. When the 3D thermal imaging camera device calibrates two cameras, it is possible to find the difference between the simultaneous images captured by the left and right cameras for that point by re-formulating the object that finds the distance to the object.

The 3D thermal imaging camera apparatus may map object distance information calculated from stereo image information to thermal image information captured by using the thermal imaging camera. In advance, the 3D thermal imaging camera apparatus may be mapped without a separate data synchronization process by synchronizing the basic setting information of the CCD camera and the thermal imaging camera in the same way. Alternatively, the 3D thermal imaging camera apparatus may synchronize data formats of the image information obtained from each of the CCD camera and the thermal imaging camera and then map them. The 3D thermal imaging camera apparatus may generate a 3D thermal imaging camera image by mapping object distance information to thermal image information.

11 , the 3D thermal imaging camera apparatus may numerically display the distance from each object to the camera. The 3D thermal imaging camera apparatus may calculate distance information from the selected object to the camera from the obtained stereo image information, and may display the calculated distance information on the 3D thermal image. For example, the user may select an object for which distance information is to be obtained from the 3D thermal image. The 3D thermal imaging camera apparatus may calculate distance information between the object selected by the user and the camera and display the calculated distance information on the 3D thermal imaging image. In addition, the 3D thermal imaging camera device may display temperature information for each object.

Referring to FIG. 12 , it is another example for explaining generation of 3D thermal image information. The 3D thermal imaging camera apparatus may acquire stereo image information photographed using at least one camera including a thermal imaging camera. The 3D thermal imaging camera device may be configured in a form in which a plurality of CCD cameras including a first camera and a second camera and a thermal imaging camera are combined. In this case, in the CCD camera, the first camera may photograph the left side and the second camera may photograph the right side. Stereo image information photographed on the left and stereo image information photographed on the right may be acquired. Alternatively, the first camera and the second camera may photograph the upper and lower sides as well as the left and right.

The 3D thermal imaging camera apparatus may generate a 3D thermal imaging camera image by mapping stereo image information photographed using a plurality of CCD cameras with thermal image information photographed using the thermal imaging camera. Accordingly, the 3D thermal imaging camera apparatus can overcome the disadvantage of the low resolution of the thermal imaging camera. As shown in FIG. 12 , for example, thermal image information may be acquired by photographing a user's head with a thermal imaging camera.

Specifically, referring to FIG. 13 , the three-dimensional thermal imaging camera device deconvolves the convolution in the deep learning-based learning model 500 to convert the thermal image to high-resolution stereo image information. Image information can be mapped. The 3D thermal imaging camera device considers thermal image information as a pooled image after convolution in a deep learning-based learning model. The 3D thermal imaging camera device may deconvolve the pooled image (ie, thermal image information) after convolution in the deep learning-based learning model 500 through the deep learning-based learning model 500 . For example, the 3D thermal imaging camera apparatus may train the learning model 500 using color information (temperature distribution) included in thermal image information. As the 3D thermal imaging camera apparatus inputs stereo image information obtained through the CCD camera into the learning model 500, color information included in the thermal image information may be mapped through deconvolution. The 3D thermal imaging camera apparatus may generate a 3D thermal image by mapping the stereo image information with color information of the thermal image information. In this case, the deep learning-based learning model 500 may be configured based on CNN or GAN (Generative Alternative Neural Network). In general, the feature data can be obtained by inputting input data to the deep learning-based learning model 500 and repeatedly performing convolution and pooling on the input data. In this case, the deep learning-based learning model proposed in the embodiment may be configured in a form including a deconvolution layer for deconvolution as well as a convolution and pooling layer.

A convolutional neural network (CNN) is a type of machine learning in which a model directly classifies an image, video, text, or sound, and is most often used in deep learning. For example, a CNN may be composed of at least one or more convolutional layers, pooling layers, FC layers, and softmax. GAN is one of the machine learning methods that automatically generate images, videos, and voices that are close to reality while a generative model and a discriminant model compete. If real examples in a specific field are presented, the computer program learns the commonalities that are not well revealed in the presented examples, so that very sophisticated counterfeits can be easily and quickly created. GAN consists of a generative model that learns probability distribution and a discriminant model that distinguishes different sets. The generative model (or generator) is trained to deceive the discriminant model as much as possible by making fake examples, and the discriminant model (or discriminant) is trained to distinguish the fake examples presented by the generative model from the real examples as accurately as possible. This method of training a generative model to deceive the discriminant model is called an adversarial process. GAN is the process of developing a generative model and a discriminant model through an antagonistic process, and it creates a similar product (a fake that looks like the real thing) that is very similar to the real example. Practical examples collected from the same field have common characteristics. However, in general, it is difficult for a person to directly analyze and find, so a computer automatically finds common features using statistical techniques. In particular, since one of the machine learning methods to find common features that are difficult for humans to find is a deep neural network, a deep neural network is used to implement a GAN.

The color map expression system may recognize objects using a deep learning-based learning model learned to recognize objects from 3D thermal image information. The color map expression system converts the 3D thermal image information obtained based on the RGB data included in the acquired 3D thermal image information, the infrared sensor value of the 3D thermal image information, and the measurement information metadata into temperature information. can be converted The color map expression system may extract temperature information of an object based on 3D thermal image information converted into temperature information. In this case, the color map expression system may recognize objects from 3D thermal image information obtained using a deep learning-based learning model learned to recognize objects. The color map representation system can train a deep learning-based learning model using a data set for recognizing an object. In this case, the data set may be image information (eg, an image, an image, a thermal image, a thermal image, etc.) for recognizing an object. Deep learning-based learning models can be configured in various forms such as CNN and RNN. For example, the color map representation system may extract temperature information of an object, or may extract temperature information for each pixel.

을 설정하여 제1사분위와 제3사분위의 차이의 값에 설정된 마진값을 곱하여 사분위 안의 최대값과 최소값을 계산할 수 있다. 예를 들면, 처음의 마진값은 1.5라는 이상치를 제거하는 기본의 사분위 범위 방식이 기본 수치로 설정되어 있을 수 있다. The color map representation system may remove outliers by using the extracted temperature information. For example, the color map representation system may remove outliers through comparison of temperature information between recognized objects. As the color map representation system compares the temperature information between the recognition objects, when the temperature information of the recognized objects is too low compared to specific objects to be difficult to distinguish by color, the outlier may be removed. Referring to FIG. 6 , the color map representation system may represent a temperature distribution for 3D thermal image information as a histogram. For example, the color map representation system may represent the extracted temperature information for each pixel as a histogram, or may represent the extracted temperature information of the object as a histogram. As shown in FIG. 6 , it can be confirmed that the data is mainly concentrated within 0 to 30 degrees of the temperature distribution. Referring to FIG. 8 , the color map expression system may generate a box plot on the temperature distribution of 3D thermal image information displayed through a histogram. Box plots can be used for comparison between groups or for detecting outliers. For example, a boxplot finds the first, second, and third quartiles, draws a box shape, draws a line 1.5 times the distance between the first and third quartiles on both sides, and , find the minimum and maximum points and draw a bar. You can connect the box and each peak and minimum with a dotted line, and mark the rest of the data with a circled dot. As shown in FIG. 8 , it can be confirmed that the outliers are located outside of 30 degrees. Accordingly, the color map representation system can replace the values outside the interquartile range (IQR) that can be seen in all devices in the 3D image information with the maximum value and the minimum value within the quartile. Referring to FIG. 7 , it is a diagram for explaining a quartile range. A quartile means three positions (25%, 50%, 75%) when data is divided into equal parts. The first quartile (Q1) means 25% points, the second quartile (Q2) means 50% points, and the third quartile (Q3) means 75% points, and the quartile range is from the 3rd quartile to the 1st quartile. It means the range minus the quartile. The color map representation system removes outliers outside the interquartile range through the box plot generated for the temperature distribution of 3D thermal image information, and replaces the removed outliers with the maximum and minimum values within the interquartile range. . At this time, the color map representation system is a margin value outside the interquartile range.

By setting , the maximum value and minimum value within the quartile can be calculated by multiplying the value of the difference between the first quartile and the third quartile by the set margin value. For example, an initial margin value of 1.5 may be set as a default value using a basic interquartile range method that removes an outlier.

Referring to FIG. 9 , it is an example of adjusting the color map of 3D thermal image information. The color map expression system may adjust the color map for the 3D thermal image information based on the temperature information from which the outlier is removed. For example, the color map representation system may normalize temperature values into pixel values of 0 to 255 based on temperature information from which outliers are removed. In this case, standardization may be performed through the Min-Max algorithm. Also, the color map expression system may perform standardization and normalization operations. The color map representation system may correct 3D thermal image information based on the adjusted color map.

4 is a block diagram for explaining the configuration of a color map representation system according to an embodiment, and FIG. 5 is a flowchart for explaining a color map representation method performed in the color map representation system according to an embodiment.

The color map expression system 100 may include an image acquisition unit 410 , an object recognition unit 420 , an outlier removal unit 430 , a color map adjustment unit 440 , and an image correction unit 450 . The components of the color map representation system 100 may be representations of different functions performed by the processor according to instructions provided by the program code stored in the color map representation system 100 . The color map representation system 100 and components of the color map representation system 100 may control the color map representation system 100 to perform steps 510 to 550 included in the color map representation method of FIG. 5 . have. In this case, the color map representation system 100 and the components of the color map representation system 100 may be implemented to execute instructions according to the code of the operating system included in the memory and the code of at least one program.

The processor may load the program code stored in the file of the program for the color map representation method into the memory. For example, when a program is executed in the color map representation system 100, the processor may control the color map representation system to load a program code from a file of the program into the memory according to the control of the operating system. In this case, the color map expression system 100 and the image acquisition unit 410, the object recognition unit 420, the outlier removal unit 430, the color map adjustment unit 440 and the image information included in the color map expression system 100 are included. Each of the governments 450 may be different functional representations of the color map representation system 100 for executing subsequent steps 510 to 550 by executing instructions of a corresponding portion of the program code loaded into the memory.

In operation 510 , the image acquisition unit 410 may acquire 3D thermal image information. The image acquisition unit 410 acquires stereo image information photographed using at least one or more cameras including a thermal imaging camera, calculates distance information of an object based on the acquired stereo image information, and calculates the distance of the object Information or the obtained stereo image information may be mapped with thermal image information to generate a 3D thermal imaging camera image. For example, the 3D thermal imaging camera device may be configured in a form in which a plurality of CCD cameras including a first camera and a second camera and at least one thermal imaging camera are combined. The image acquisition unit 410 maps the stereo image information photographed using a plurality of CCD cameras with the thermal image image information photographed using the thermal imaging camera, and the photographed thermal image image information in a deep learning-based learning model. After convolution, the image is regarded as a pooled image, and the thermal image information that is considered as the pooled image after convolution is de-convolved through a deep learning-based learning model to form a three-dimensional column. Images can be created. The image acquisition unit 410 learns a deep learning-based learning model using color information included in the image thermal image information, and inputs the stereo image information acquired through the CCD camera to the learned deep learning-based learning model. Accordingly, stereo image information may be mapped with color information included in thermal image information through deconvolution. As another example, the 3D thermal imaging camera device may be configured in a form in which a plurality of CCD cameras including a first camera and a second camera and at least one thermal imaging camera are combined. The image acquisition unit 410 may generate 3D thermal image information by mapping object distance information calculated from stereo image information photographed using a plurality of CCD cameras to thermal image information. As another example, the 3D thermal imaging camera device may be configured in a form in which a plurality of CCD cameras including a first camera and a second camera and at least one thermal imaging camera are combined. When the resolution of the plurality of CCD cameras and the at least one thermal imaging camera are the same, the image acquisition unit 410 is a distance of an object calculated from stereo image information acquired using two or more of the CCD camera or the thermal imaging camera. The information may be mapped to the thermal image information to generate 3D thermal image information.

In operation 520 , the object recognition unit 420 may recognize objects using a deep learning-based learning model learned to recognize objects from the obtained 3D thermal image information. The object recognition unit 420 converts the 3D thermal image information obtained based on the RGB data included in the acquired 3D thermal image information, infrared sensor values of the 3D thermal image information, and measurement information metadata to the temperature. It is converted into information, and temperature information of an object may be extracted based on the 3D thermal image information converted into temperature information. The object recognition unit 420 represents the temperature distribution for the three-dimensional thermal image information converted into the temperature information as a histogram, and generates a box plot on the temperature distribution of the three-dimensional thermal image information displayed through the histogram. can

In operation 530, the outlier removal unit 430 may remove the outlier by comparing temperature information between the recognized objects. The outlier removal unit 430 removes outliers that exist outside the interquartile range through a box plot generated with respect to the temperature distribution of the 3D thermal image information, and replaces the removed outliers with the maximum and minimum values within the interquartile range. can The outlier removal unit 430 may set a margin value outside the quartile range, multiply the value of the difference between the first quartile and the third quartile by the set margin value, and calculate the maximum value and the minimum value within the quartile.

In operation 540 , the color map adjuster 440 may adjust the color map for the 3D thermal image information based on the temperature information from which the outlier is removed.

In operation 550 , the image correction unit 450 may correct 3D thermal image information based on the adjusted color map.

The device described above may be implemented as a hardware component, a software component, and/or a combination of the hardware component and the software component. For example, devices and components described in the embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA). , a programmable logic unit (PLU), microprocessor, or any other device capable of executing and responding to instructions, may be implemented using one or more general purpose or special purpose computers. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. A processing device may also access, store, manipulate, process, and generate data in response to execution of the software. For convenience of understanding, although one processing device is sometimes described as being used, one of ordinary skill in the art will recognize that the processing device includes a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that can include For example, the processing device may include a plurality of processors or one processor and one controller. Other processing configurations are also possible, such as parallel processors.

Software may comprise a computer program, code, instructions, or a combination of one or more thereof, which configures a processing device to operate as desired or is independently or collectively processed You can command the device. The software and/or data may be any kind of machine, component, physical device, virtual equipment, computer storage medium or apparatus, to be interpreted by or to provide instructions or data to the processing device. may be embodied in The software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored in one or more computer-readable recording media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the embodiment, or may be known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic such as floppy disks. - includes magneto-optical media, and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like.

As described above, although the embodiments have been described with reference to the limited embodiments and drawings, various modifications and variations are possible from the above description by those skilled in the art. For example, the described techniques are performed in an order different from the described method, and/or the described components of the system, structure, apparatus, circuit, etc. are combined or combined in a different form than the described method, or other components Or substituted or substituted by equivalents may achieve an appropriate result.

Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

Claims (15)

In the color map expression method,
obtaining 3D thermal image information;
recognizing objects using a deep learning-based learning model learned to recognize objects from the obtained three-dimensional thermal image information;
removing outliers through comparison of temperature information between the recognized objects;
adjusting a color map for 3D thermal image information based on the temperature information from which the outlier is removed; and
Compensating 3D thermal image information based on the adjusted color map
including,
The step of removing the outlier is,
removing outliers outside the interquartile range through a box plot generated with respect to the temperature distribution of the 3D thermal image information, and replacing the removed outliers with maximum and minimum values within the interquartile range
A colormap representation method that includes. According to claim 1,
Recognizing the object comprises:
Converting the obtained 3D thermal image information into temperature information based on the RGB data included in the obtained 3D thermal image information, infrared sensor values of the 3D thermal image information, and measurement information metadata, extracting temperature information of an object based on the three-dimensional thermal image information converted into the temperature information
A colormap representation method that includes. 3. The method of claim 2,
Recognizing the object comprises:
A step of representing a temperature distribution for the three-dimensional thermal image information converted into the temperature information as a histogram, and generating a box plot on the temperature distribution of the three-dimensional thermal image information displayed through the histogram
A colormap representation method that includes. delete According to claim 1,
The step of removing the outlier is,
calculating a maximum value and a minimum value within the quartile by setting a margin value outside the quartile range and multiplying the difference value between the first quartile and the third quartile by the set margin value
A colormap representation method that includes. According to claim 1,
The color map expression method is,
Comprising obtaining three-dimensional thermal image information using a three-dimensional thermal imaging camera device,
The step of obtaining the 3D thermal image information includes:
acquiring stereo image information photographed using at least one camera including a thermal imaging camera;
calculating distance information of an object based on the obtained stereo image information; and
generating a three-dimensional thermal imaging camera image by mapping the calculated distance information of the object or the obtained stereo image information with thermal image information
A colormap representation method that includes. 7. The method of claim 6,
The three-dimensional thermal imaging camera device,
Including a combination of a plurality of CCD cameras including a first camera and a second camera and at least one thermal imaging camera,
The step of obtaining the 3D thermal image information includes:
The stereo image information photographed using the plurality of CCD cameras is mapped with thermal image information photographed using a thermal imaging camera, and the photographed thermal image information is convolved in a deep learning-based learning model. A three-dimensional thermal image is obtained by de-convolution through the deep learning-based learning model, which is regarded as a post-pooled image, and the thermal image information considered as a pooled image after the convolution. steps to create
A colormap representation method that includes. 8. The method of claim 7,
The step of obtaining the 3D thermal image information includes:
The deep learning-based learning model is trained using the color information included in the thermal image information, and the stereo image information obtained through the CCD camera is input to the learned deep learning-based learning model, thereby deconvolution mapping the stereo image information with color information included in the thermal image information through
A colormap representation method that includes. 7. The method of claim 6,
The three-dimensional thermal imaging camera device,
Including a combination of a plurality of CCD cameras including a first camera and a second camera and at least one thermal imaging camera,
The step of obtaining the 3D thermal image information includes:
generating 3D thermal image information by mapping object distance information calculated from stereo image information photographed using the plurality of CCD cameras to the thermal image information
A colormap representation method that includes. 7. The method of claim 6,
The three-dimensional thermal imaging camera device,
Including a combination of a plurality of CCD cameras including a first camera and a second camera and at least one thermal imaging camera,
The step of obtaining the 3D thermal image information includes:
When the resolutions of the plurality of CCD cameras and the at least one thermal imaging camera are the same, distance information of an object calculated from stereo image information obtained using two or more of a CCD camera or a thermal imaging camera is used as a thermal image. Generating 3D thermal image information by mapping the information
A colormap representation method that includes. In the color map representation system,
an image acquisition unit acquiring three-dimensional thermal image information;
an object recognition unit for recognizing objects using a deep learning-based learning model learned to recognize objects from the obtained three-dimensional thermal image information;
an outlier removal unit for removing an outlier through comparison of temperature information between the recognized objects;
a color map adjusting unit that adjusts a color map for 3D thermal image information based on the temperature information from which the outlier is removed; and
An image correction unit for correcting 3D thermal image information based on the adjusted color map
including,
The outlier removal unit,
Removes outliers outside the interquartile range through a box plot generated for the temperature distribution of the 3D thermal image information, and replaces the removed outliers with the maximum and minimum values within the interquartile range
Colormap representation system. 12. The method of claim 11,
The object recognition unit,
Converting the obtained 3D thermal image information into temperature information based on the RGB data included in the obtained 3D thermal image information, infrared sensor values of the 3D thermal image information, and measurement information metadata, Extracting temperature information of an object based on the 3D thermal image information converted into the temperature information
A color map representation system, characterized in that. 13. The method of claim 12,
The object recognition unit,
The temperature distribution of the three-dimensional thermal image information converted into the temperature information is expressed as a histogram, and a box plot is generated on the temperature distribution of the three-dimensional thermal image information displayed through the histogram.
A color map representation system, characterized in that. delete 12. The method of claim 11,
The outlier removal unit,
Setting a margin value outside the quartile range to calculate the maximum value and the minimum value within the quartile by multiplying the value of the difference between the first quartile and the third quartile by the set margin value
A color map representation system, characterized in that.

Images