KR102591607B1 - Image processing method and system using metadata extracted through reverse engineering of thermal image - Google Patents

Abstract

An image processing method and system using metadata extracted through reverse engineering of a thermal image is disclosed. Extracting metadata through reverse engineering of a thermal image according to an embodiment; And it may include performing image processing on the thermal image using the extracted metadata.

Description

Image processing method and system using metadata extracted through reverse engineering of thermal imaging images {IMAGE PROCESSING METHOD AND SYSTEM USING METADATA EXTRACTED THROUGH REVERSE ENGINEERING OF THERMAL IMAGE}

The explanation below relates to thermal image data processing technology and an image processing method and system using metadata extracted through reverse engineering of thermal image images.

A separate tool is needed to extract metadata from thermal images and process the images. For example, thermal images taken with a FLIR model require software such as exiftool for metadata extraction and FlirTool for image processing. These programs are provided with only limited functions by the manufacturer. Since there are no open APIs or libraries, batch processing of data is inevitably limited even when processing big data consisting of thermal image images.

A method and system for extracting metadata by reverse engineering a thermal image and performing image processing on the thermal image using the extracted metadata can be provided.

An image processing method performed by an image processing system includes extracting metadata through reverse engineering of a thermal image; And it may include performing image processing on the thermal image using the extracted metadata.

The extracting step analyzes header information for metadata extraction among headers included in the thermal image based on the file information of the thermal image, and uses the offset obtained through the analyzed header information to image the image. It may include accessing an Image File Directory (IFD) area and extracting metadata for the thermal image from tag information included in the accessed image file directory area.

In the extracting step, in order to access the first tag indicating the file path value, the offset of the image file directory area of the second tag obtained through the analyzed header information and the offset of the first tag indicating the file path value are obtained. It includes the step of: wherein the first tag may be a lower tag of the second tag.

The extracting step includes information on the first tag indicating the file path value through address data plus the offset of the image file directory area of the obtained second tag and the offset of the first tag indicating the file path value. A step of extracting metadata may be included.

The step of performing the image processing may include adjusting the color map of the thermal image through a tool for the image processing.

The step of performing the image processing includes reading the sensor value stored in the file information of the thermal image image, converting the read sensor value into a temperature value, and converting the sensor value stored in the image file into a thermal image using a color map. It may include the step of visualizing it as an image.

The step of performing the image processing includes dividing the color map into a plurality of sections, setting the length ratio of each section, and setting the ratio of pixels included in each section among all pixels according to the set length ratio of each section. It may include the step of obtaining the ratio value of each section representing .

The step of performing the image processing may include dividing the color map into a plurality of sections of a certain length and setting the length ratio of each section to be the same.

The step of performing the image processing is to determine the length of the section through Equation 1 ( Rn = Rn + 10 / ( W × Including the step of modifying the ratio, and in Equation 1, Rn is Xn may represent the length ratio of each section, Xn may represent the ratio value of each section, and W may represent a constant for reducing the time taken to converge.

The step of performing the image processing repeats the process of modifying the ratio value of each obtained section and the ratio of the section length until the minimum ratio among the ratio values of each obtained section exceeds a preset ratio or converges. It may include steps.

The image processing system includes a data extraction unit that extracts metadata through reverse engineering of a thermal image; and an image processing unit that performs image processing on the thermal image using the extracted metadata.

Without a separate tool for extracting metadata and processing images, it is possible to reverse engineer thermal images through API to extract metadata and process the image using the extracted metadata. In addition, batch processing of big data consisting of thermal images is possible.

It can provide the function of reading sensor values from an image, converting them to temperature values, and visualizing the sensor values as a thermal image using a color map.

Figure 1 is an example to explain the FLIR image binary structure, according to one embodiment.
Figure 2 is an example to explain the FLIR FFF header binary structure in one embodiment.
Figure 3 is an example to explain the IFD area binary structure, according to one embodiment.
Figure 4 is an example to explain the FLIR JPG file analysis and metadata extraction process in one embodiment.
Figure 5 is an example showing matching a thermal image and a real image in one embodiment.
Figure 6 is an example showing the original, an image in which a color map is applied to sensor values, and an image in which the original is reproduced by modifying the color map, according to an embodiment.
Figure 7 is a diagram showing the color map and pixel distribution analysis results of an image, according to one embodiment.
Figure 8 is a block diagram for explaining the configuration of an image processing system according to an embodiment.
Figure 9 is a flowchart for explaining an image processing method in an image processing system according to an embodiment.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings.

In the embodiment, the operation of reverse engineering a thermal image to extract metadata and processing the image using the extracted metadata will be described. Reverse engineering means analyzing the finished product in detail to trace the basic design details. In software, it means specializing in technology to convert machine code into raw source code. It is called reverse engineering in the sense of tracing back the normal process of a product. It is used to analyze how a product operates to find out how a program or security mechanism operates.

In the embodiment, an image captured by a FLIR model will be described as an example of a thermal image. Metadata can be extracted by reverse engineering from images taken with a FLIR model, and an API that processes the image using the extracted metadata can be provided.

Figure 1 is an example to explain the FLIR image binary structure, according to one embodiment. FLIR images follow the Exif format and the basic structure can be represented as shown in Figure 1. In Exif format, the main information of the image is stored in the APP1 area. At this time, there may be a plurality of APP1 areas where Exif data and FLIR data are respectively stored. Figure 1 shows that there are two APP1 areas where Exif data and FLIR data are respectively stored. If the APP1 area is divided into large categories, inside APP1, the categories are divided into sub-tags and assigned to each IFD area. In the example, we will focus on analyzing the APP1 area where FLIR data is stored.

Thermal image metadata can be classified and stored in the FLIR APP1 area with tags as shown in Table 1.

Table 1: List of FLIR thermal imaging metadata tags

Not all tags always exist in the file, and which tags exist in the file and the offset information of the tags within the APP1 area are recorded in the FLIR FFF header. The image processing system may start by reading the FFF header to extract thermal image metadata.

Figure 2 is an example to explain the FLIR FFF header binary structure in one embodiment.

Referring to Figure 2, it shows the binary of the FLIR FFF header area of the actual FLIR image. Decoded text shown in Figure 2 is a value converted from a binary value to ASCII.

The unique identifier FFF indicates that the FFF header begins. Since all offset information that appears thereafter is a relative value to the FFF header, the offset of the FFF header is essential to access metadata at the file level.

A list of tags that exist in the file after a specific byte from the offset of the FFF header and the offset of the IFD area where the tags are stored are given.

Figure 3 is an example to explain the IFD area binary structure, according to one embodiment.

Actual metadata are recorded in the IFD (Image File Directory) area of FLIR APP1. To obtain metadata, each IFD area must be accessed and metadata extracted using the offset obtained through FFF header analysis.

In the IFD area, data belonging to tags is classified into sub-tags and stored continuously. Information about this subtag can be obtained from the exiftool forum.

Referring to Figure 3, this is an example showing the process of extracting metadata of the PaletteInfo tag. In the sub-tag information table, the index means the offset in the IFD area. Using this, you can access the data of the child tag in the same way as accessing the parent tag.

To access the 'PaletteFileName' tag, which is a sub-tag of the 'PaletteInfo' tag and represents the palette file path value, the offset (0x0BAC) of the PaletteInfo tag IFD area obtained through FFF header analysis and the offset (0x0030) of 'PaletteFileName' are required. do.

Finally, you can see that information corresponding to the 'PaletteFileName' tag appears at 0x0BEC, which is the address obtained by adding the two offsets, and each metadata is extracted through the above process.

Figure 4 is an example to explain the FLIR JPG file analysis and metadata extraction process in one embodiment.

The image processing system is a JPG file recognition process, an Exif header access process, a search process for the AP1 area where thermal image metadata is stored in the JPG file, a tag list and offset information extraction process from the FFF header, and an IFD for each tag through the tag list and offset information. Area access and metadata extraction processes can be performed.

The image processing system can identify the file information of the thermal image captured by the camera. At this time, the camera may be a thermal imaging camera, or a thermal imaging camera function may be provided through a hardware or software combination of different types of cameras other than thermal imaging cameras. Additionally, a thermal image may be acquired from a thermal imaging camera, or the thermal image may be obtained through specific processing and combination of images acquired from different types of cameras. For example, an image processing system can recognize a JPG file from a thermal image. The image processing system can first read the SOI (Start Of Image) unique bits of the thermal image to check whether the file follows the .jpg format.

The imaging system has access to the thermal image Exif header. In the case of Exif format JPG files, the image processing system has a constant length because information is not stored in the JFIF area, so the Exif header can be easily accessed directly.

The image processing system can search the APP1 area where thermal image metadata is stored in the JPG file. The image processing system records the size of the area at the beginning of each segment of Exif, allowing it to quickly search the FLIR APP1 area.

The image processing system can extract tag list and offset information from the FFF header. At this time, all offset information appearing in the FFF header is a relative offset based on the FFF header, so the offset information in the FFF header is essential to access data through the corresponding value.

The image processing system can access the IFD area for each tag through the tag list and offset information and extract metadata through the accessed IFD area. The image processing system can infer the offset, size, and format of the child tag through the index information in the tag information table of the Exiftool forum.

Figure 5 is an example showing matching a thermal image and a real image in one embodiment.

The image processing system can match thermal images and real images. In FLIR models, thermal images and real images are captured simultaneously by the same camera, but while real images support high resolution, thermal images cannot support the same resolution as real images. Therefore, in order to compare the two images, it is necessary to use high-resolution images. You need to find the part captured in the thermal image in the real image.

Figure 5 is an example showing matching a thermal image and a real image. The images in Figure 5 are, from the left, the original image (640 480).

The resolution of the middle image (1402 x 1502) can be determined by the size of the thermal image and the ratio value between the real image and the thermal image unique to each lens.

Figure 6 is an example showing the original, an image in which a color map is applied to sensor values, and an image in which the original is reproduced by modifying the color map, according to an embodiment.

The image processing system can adjust the color map of the thermal image. Ultimately, when using thermal imaging images, the most important information is the temperature value, which is not expressed in thermal imaging images. At this time, the temperature value is stored as a sensor value in the image file. In fact, software such as FlirTool operates based on temperature values. In the embodiment, a function of reading sensor values from a thermal image and converting them into temperature values is supported, and a function of visualizing sensor values as a thermal image can be implemented using a color map.

If the color map included in the Flir thermal image is applied to the sensor value as is, an image different from the original thermal image may be generated, as shown in the center image of FIG. 6.

Figure 7 is a diagram showing the color map and pixel distribution analysis results of an image, according to one embodiment.

The image processing system can compare and analyze the color map applied to the original image and the color map recorded as metadata to reproduce the original image. As a result, it can be expressed as shown in FIG. 7.

If you look at the two color maps shown on the left of FIG. 7, you can see that they appear different from each other. The image processing system can analyze the pixel distribution for each temperature range, as shown in the graph on the right of FIG. 7, to analyze the principle of color map differences. At this time, each temperature range can be set by the user or computer.

The image processing system can adjust the length of the pixel distribution as it analyzes the pixel distribution. The image processing system can adjust the pixel distribution for the length of the area above a preset standard and the length of the area below the preset standard.

For example, as a result of the analysis, it can be seen that in order to improve the visibility of the image, the length of the area with high pixel distribution was reduced and the length of the area with low pixel distribution was increased so that the ratio of each color appearing in the image was similar.

To reproduce this function, the following algorithm can be developed and used, and can be reproduced as shown in FIG. 6.

(1) The image processing system may divide the color map into a plurality of sections (for example, 6) of constant length. As the color map is divided into certain lengths, the length ratio of each section is as follows.

R ₁ : R ₂ : R ₃ : R ₄ : R ₅ : R ₆ = 1: 1: 1: 1: 1: 1

_{( 2 ) The image} processing _{system can} derive For example, X ₁ can be calculated as 1/6.

(3) The image processing system can modify the ratio of section lengths by applying Equation 1 to R _n and X _n .

Equation 1: ( Rn = Rn + 10 / ( W × Xn + 1)

Here, in Equation 1, Rn is Xn is the length ratio of each section, Xn is the ratio value of each section, and W is a constant for reducing the time taken to converge.

(4) The image processing system may repeat the process of modifying the ratio value of each acquired section and the ratio of the section length until the minimum ratio among the ratio values of each obtained section exceeds a preset ratio or converges. For example, the image processing system may repeat processes (2) and (3) until the minimum ratio of X _n exceeds 10% or converges.

FIG. 8 is a block diagram for explaining the configuration of an image processing system according to an embodiment, and FIG. 9 is a flowchart for explaining an image processing method in an image processing system according to an embodiment.

The processor of the image processing system 100 may include a data extraction unit 810 and an image processing unit 820. These processor components may be expressions of different functions performed by the processor according to control instructions provided by program codes stored in the image processing system. The processor and its components may control the image processing system to perform steps 910 to 920 included in the image processing method of FIG. 9 . At this time, the processor and its components may be implemented to execute instructions according to the code of an operating system included in the memory and the code of at least one program.

The processor may load program code stored in a program file for an image processing method into memory. For example, when a program is executed in an image processing system, the processor can control the image processing system to load program code from the program file into memory under the control of the operating system. At this time, the processor and each of the data extraction unit 810 and the image processing unit 820 included in the processor are processors for executing the subsequent steps 910 to 920 by executing instructions of the corresponding portion of the program code loaded in the memory. may be different functional expressions of .

In step 910, the data extraction unit 810 may extract metadata through reverse engineering of the thermal image. The data extraction unit 810 analyzes header information for metadata extraction among the headers included in the thermal image based on the file information of the thermal image, and uses the offset obtained through the analyzed header information to file the image file. You can access the Image File Directory (IFD) area and extract metadata about the thermal image from the tag information contained in the accessed image file directory area. In order to access the first tag indicating the file path value, the data extraction unit 810 acquires the offset of the image file directory area of the second tag obtained through the analyzed header information and the offset of the first tag indicating the file path value. can do. At this time, the first tag means a sub-tag of the second tag. The data extraction unit 810 generates metadata through information about the first tag indicating the file path value through address data plus the offset of the image file directory area of the acquired second tag and the offset of the first tag indicating the file path value. can be extracted.

In step 920, the image processing unit 820 may perform image processing on the thermal image using the extracted metadata. The image processing unit 820 can adjust the color map of the thermal image through an image processing tool. The image processing unit 820 reads the sensor value stored in the file information of the thermal image image, converts the read sensor value into a temperature value, and visualizes the sensor value stored in the image file as a thermal image using a color map. . The image processing unit 820 divides the color map into a plurality of sections, sets the length ratio of each section, and each section represents the ratio of pixels included in each section among all pixels according to the set length ratio of each section. The ratio value can be obtained. The image processing unit 820 may divide the color map into a plurality of sections of constant length and set the length ratio of each section to be the same. The image processing unit 820 can modify the ratio of section lengths through Equation 1 ( Rn = Rn + 10 / ( W × At this time, in Equation 1, Rn is Xn is the length ratio of each section, Xn is the ratio value of each section, and W is a constant for reducing the time taken to converge. The image processing unit 820 may repeat the process of modifying the ratio of the obtained ratio value of each section and the section length until the minimum ratio among the obtained ratio values of each section exceeds a preset ratio or converges.

The device described above may be implemented with hardware components, software components, and/or a combination of hardware components and software components. For example, devices and components described in embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), etc. , may be implemented using one or more general-purpose or special-purpose computers, such as a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. A processing device may execute an operating system (OS) and one or more software applications that run on the operating system. Additionally, a processing device may access, store, manipulate, process, and generate data in response to the execution of software. For ease of understanding, a single processing device may be described as being used; however, those skilled in the art will understand that a processing device includes multiple processing elements and/or multiple types of processing elements. It can be seen that it may include. For example, a processing device may include a plurality of processors or one processor and one controller. Additionally, other processing configurations, such as parallel processors, are possible.

Software may include a computer program, code, instructions, or a combination of one or more of these, which may configure a processing unit to operate as desired, or may be processed independently or collectively. You can command the device. Software and/or data may be used on any type of machine, component, physical device, virtual equipment, computer storage medium or device to be interpreted by or to provide instructions or data to a processing device. It can be embodied in . Software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc., singly or in combination. Program instructions recorded on the medium may be specially designed and configured for the embodiment or may be known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. -Includes optical media (magneto-optical media) and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, flash memory, etc. Examples of program instructions include machine language code, such as that produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc.

As described above, although the embodiments have been described with limited examples and drawings, various modifications and variations can be made by those skilled in the art from the above description. For example, the described techniques are performed in a different order than the described method, and/or components of the described system, structure, device, circuit, etc. are combined or combined in a different form than the described method, or other components are used. Alternatively, appropriate results may be achieved even if substituted or substituted by an equivalent.

Therefore, other implementations, other embodiments, and equivalents of the claims also fall within the scope of the claims described below.

Claims (11)

In an image processing method performed by an image processing system,
Extracting metadata by performing reverse engineering-based file analysis executed through an API for image processing of thermal images; and
Performing image processing on the thermal image using the extracted metadata
Including,
The extracting step is,
Based on the file information of the thermal image, header information for metadata extraction among the headers included in the thermal image is analyzed, and the offset obtained through the analyzed header information is used to create an image file directory (Image File). Directory (IFD) area, extract metadata for the thermal image from tag information included in the accessed image file directory area, and access the first tag indicating the file path value. Obtaining an offset of the first tag indicating an offset and a file path value of an image file directory area of a second tag obtained through information, and indicating an offset of an image file directory area of the obtained second tag and the file path value. Extracting metadata through information about the first tag indicating the file path value through address data plus the offset of the first tag
Including,
The first tag is a sub-tag of the second tag,
The step of performing the image processing is,
Adjust the color map of the thermal image through the image processing tool, read the sensor value stored in the file information of the thermal image, convert the read sensor value into a temperature value, and convert the color map Visualizing sensor values stored in the file information of the thermal image as a thermal image using
Including,
The color map is one in which the length of the pixel distribution is adjusted through analysis of the pixel distribution for each temperature range by comparing the color map applied to the original image and the color map recorded in the metadata,
Image processing method. delete delete delete delete delete According to paragraph 1,
The step of performing the image processing is,
By dividing the color map into a plurality of sections, the length ratio of each section is set, and the ratio value of each section is obtained, which represents the ratio of pixels included in each section among all pixels according to the set length ratio of each section. steps to do
An image processing method comprising: According to paragraph 1,
The step of performing the image processing is,
Setting the length ratio of each section to be the same by dividing the color map into a plurality of sections of constant length
An image processing method comprising: In clause 7,
The step of performing the image processing is,
A step of modifying the ratio of section lengths through Equation 1 ( Rn = Rn + 10 / ( W ×
Including,
In Equation 1 above, Rn is The length ratio of each section is the ratio,
An image processing method characterized by: According to clause 9,
The step of performing the image processing is,
Repeating the process of modifying the ratio of the ratio value of each obtained section and the section length until the minimum ratio among the obtained ratio values of each section exceeds a preset ratio or converges.
An image processing method comprising: In an image processing system,
a data extraction unit that extracts metadata by performing reverse engineering-based file analysis executed through an API for image processing of thermal images; and
An image processing unit that performs image processing on the thermal image using the extracted metadata.
Including,
The data extraction unit,
Based on the file information of the thermal image, header information for metadata extraction among the headers included in the thermal image is analyzed, and the offset obtained through the analyzed header information is used to create an image file directory (Image File). Directory (IFD) area, extract metadata for the thermal image from tag information included in the accessed image file directory area, and access the first tag indicating the file path value. Obtaining an offset of the first tag indicating an offset and a file path value of an image file directory area of a second tag obtained through information, and indicating an offset of an image file directory area of the obtained second tag and the file path value. It includes extracting metadata through information about the first tag indicating the file path value through address data plus the offset of the first tag,
The first tag is a sub-tag of the second tag,
The image processing unit,
Adjust the color map of the thermal image through the image processing tool, read the sensor value stored in the file information of the thermal image, convert the read sensor value into a temperature value, and convert the color map It includes visualizing the sensor value stored in the file information of the thermal image as a thermal image using,
The color map is one in which the length of the pixel distribution is adjusted through analysis of the pixel distribution for each temperature range by comparing the color map applied to the original image and the color map recorded in the metadata,
Image processing system.