CN113256525B - Infrared thermal imaging image enhancement method, device, equipment and readable storage medium - Google Patents
Infrared thermal imaging image enhancement method, device, equipment and readable storage medium Download PDFInfo
- Publication number
- CN113256525B CN113256525B CN202110572941.2A CN202110572941A CN113256525B CN 113256525 B CN113256525 B CN 113256525B CN 202110572941 A CN202110572941 A CN 202110572941A CN 113256525 B CN113256525 B CN 113256525B
- Authority
- CN
- China
- Prior art keywords
- network
- enhancement
- image
- pixel
- infrared
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000001931 thermography Methods 0.000 title claims abstract description 54
- 238000000034 method Methods 0.000 title claims abstract description 49
- 238000000605 extraction Methods 0.000 claims abstract description 57
- 238000013528 artificial neural network Methods 0.000 claims abstract description 43
- 238000012545 processing Methods 0.000 claims abstract description 30
- 230000002708 enhancing effect Effects 0.000 claims description 10
- 238000004590 computer program Methods 0.000 claims description 9
- 230000004927 fusion Effects 0.000 claims description 9
- 230000004913 activation Effects 0.000 claims description 7
- 238000013507 mapping Methods 0.000 claims description 7
- 230000003213 activating effect Effects 0.000 claims description 2
- 230000000694 effects Effects 0.000 abstract description 22
- 230000008569 process Effects 0.000 abstract description 13
- 230000004044 response Effects 0.000 abstract description 8
- 238000010586 diagram Methods 0.000 description 13
- 238000012549 training Methods 0.000 description 12
- 238000004422 calculation algorithm Methods 0.000 description 9
- 238000004364 calculation method Methods 0.000 description 4
- 238000013135 deep learning Methods 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 238000012935 Averaging Methods 0.000 description 2
- 230000008859 change Effects 0.000 description 2
- 238000010276 construction Methods 0.000 description 2
- 230000007547 defect Effects 0.000 description 2
- 239000000284 extract Substances 0.000 description 2
- 239000011159 matrix material Substances 0.000 description 2
- 238000012795 verification Methods 0.000 description 2
- 230000000007 visual effect Effects 0.000 description 2
- ORILYTVJVMAKLC-UHFFFAOYSA-N Adamantane Natural products C1C(C2)CC3CC1CC2C3 ORILYTVJVMAKLC-UHFFFAOYSA-N 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 239000011449 brick Substances 0.000 description 1
- 238000004883 computer application Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000002474 experimental method Methods 0.000 description 1
- 238000001914 filtration Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 238000011478 gradient descent method Methods 0.000 description 1
- 230000002401 inhibitory effect Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000007500 overflow downdraw method Methods 0.000 description 1
- 230000002085 persistent effect Effects 0.000 description 1
- 238000007781 pre-processing Methods 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 230000035945 sensitivity Effects 0.000 description 1
- 239000013589 supplement Substances 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
- 230000001052 transient effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/70—Denoising; Smoothing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/50—Image enhancement or restoration using two or more images, e.g. averaging or subtraction
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10048—Infrared image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20212—Image combination
- G06T2207/20221—Image fusion; Image merging
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Molecular Biology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Artificial Intelligence (AREA)
- Evolutionary Computation (AREA)
- General Health & Medical Sciences (AREA)
- Biomedical Technology (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Life Sciences & Earth Sciences (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Health & Medical Sciences (AREA)
- Image Processing (AREA)
Abstract
The invention discloses an infrared thermal imaging image enhancement method, which applies a deep neural network to carry out image enhancement on acquired infrared image data, and carries out enhancement processing on each pixel point in the network through a pixel enhancement network, thereby solving the problem of difference of each pixel response of an infrared thermal imaging detector; meanwhile, the global feature extraction network is called to extract the overall features of the image, the overall features act on the output of the pixel enhancement network to ensure the local and overall consistency of enhancement, noise can be inhibited, and the image enhancement effect is improved; in addition, the process is a parameter-free image enhancement process, parameters do not need to be set manually, user side operation can be simplified, and user experience during application can be improved. The invention also discloses an infrared thermal imaging image enhancement device, equipment and a readable storage medium, and has corresponding technical effects.
Description
Technical Field
The invention relates to the technical field of image preprocessing, in particular to an infrared thermal imaging image enhancement method, device and equipment and a readable storage medium.
Background
Digital image processing technology is now commonly used in various industries, and image enhancement processing technology is one of the main components thereof. The image enhancement process is to suppress the disturbance factors on the original image, thereby improving the quality of the original image. In other words, important contents are enhanced and unimportant contents are suppressed, and then a clear image display effect is obtained. Sometimes, the visual effect of the captured image is not good due to the influence of the scene conditions, so that the visual effect seen by people needs to be improved by using an image enhancement technology, for example, some characteristics of the target object in the image are highlighted, characteristic parameters of the target are extracted from the digital image, and the like, so as to facilitate the identification, tracking and understanding of the target object in the image.
Although the visible light image enhancement algorithm has made a certain progress based on means such as deep learning, in the scenes with insufficient visible light, such as at night and in foggy days, a reliable visible light image cannot be acquired, and in these cases, infrared thermal imaging is a beneficial supplement.
However, at present, the visible light image enhancement algorithm is not suitable for infrared image enhancement, and the image enhancement effect is very poor when the visible light image enhancement algorithm is directly applied to infrared thermal imaging, so that the requirement on the accuracy of subsequent image processing cannot be met; in addition, the existing enhancement algorithm aiming at the infrared thermal imaging image has poor enhancement effect and poor user experience due to the defects that the existing enhancement algorithm cannot be suitable for local brightness characteristic enhancement and has the defects of large limitation of application environment, large noise, numerous parameters, large adjustment difficulty and the like; at present, some deep learning-based infrared image enhancement algorithms also exist, but because the effect influence factors of the infrared enhancement process are more, the enhancement effect difference problem caused by the influence factors is not considered in the deep learning-based infrared image enhancement algorithms at present, and the feature information in the infrared image cannot be completely extracted, particularly, the image details cannot be enhanced, so that the enhancement effect is poor.
In summary, how to improve the infrared image enhancement effect is a technical problem that needs to be solved urgently by those skilled in the art at present.
Disclosure of Invention
The invention aims to provide an infrared thermal imaging image enhancement method, device and equipment and a readable storage medium, so as to improve the infrared image enhancement effect.
In order to solve the technical problems, the invention provides the following technical scheme:
an infrared thermal imaging image enhancement method, comprising:
acquiring infrared image data;
calling a pre-trained infrared image enhancement depth neural network to carry out image enhancement forward reasoning on the infrared image data; the infrared image enhancement deep neural network comprises a plurality of cascaded network layers; the network layer comprises: the pixel enhancement network is used for enhancing each pixel point and the global feature extraction network is used for extracting the overall features of the image;
and acquiring an image output by the infrared image enhanced depth neural network as an enhanced image corresponding to the infrared image data.
Optionally, the image-enhanced forward inference processing of the network layer includes:
determining image data input by the network layer as target image data;
the pixel enhancement network carries out enhancement processing on each pixel point in the target image data to obtain an output result of the pixel enhancement network;
the global feature extraction network carries out overall feature extraction on the target image data to obtain an output result of the global feature extraction network;
and performing data fusion on the output result of the pixel enhancement network and the output result of the global feature extraction network to obtain network layer output image data.
Optionally, the pixel enhancement network performs enhancement processing on each pixel point in the received image data, including:
the convolution layer in the pixel enhancement network performs feature extraction on each pixel point in the received image data to obtain pixel features;
an activation layer in the pixel enhancement network performs nonlinear mapping capability enhancement processing on the pixel characteristics to obtain enhanced data;
multiplying the enhancement data by the signal value of the corresponding pixel point to obtain an enhancement result corresponding to the pixel point;
and counting the enhancement results corresponding to all the pixel points, and taking the enhancement results as the output results of the pixel enhancement network.
Optionally, the active layer specifically includes: and (5) sigmoid activating a layer.
Optionally, the global feature extraction network performs overall feature extraction on the received image data, and includes:
the convolution layer in the global feature extraction network carries out feature extraction on each part of image in the received image data to obtain grouping features;
and counting the average value of all the grouping features in the image data, and taking the average value as an output result of the global feature extraction network.
Optionally, the performing data fusion on the output result of the pixel enhancement network and the output result of the global feature extraction network includes:
and multiplying the output result of the pixel enhancement network with the output result of the global feature extraction network.
Optionally, the infrared image enhanced deep neural network comprises 3 cascaded network layers.
An infrared thermal imaging image enhancement device, comprising:
the image acquisition unit is used for acquiring infrared image data;
the network reasoning unit is used for calling a pre-trained infrared image enhanced depth neural network to carry out image enhanced forward reasoning on the infrared image data; the infrared image enhancement deep neural network comprises a plurality of cascaded network layers; the network layer comprises: the pixel enhancement network is used for enhancing each pixel point and the global feature extraction network is used for extracting the overall features of the image;
and the output acquisition unit is used for acquiring the image output by the infrared image enhanced depth neural network as an enhanced image corresponding to the infrared image data.
An infrared thermal imaging image enhancing device, comprising:
a memory for storing a computer program;
a processor for implementing the steps of the above-mentioned infrared thermographic image enhancement method when executing said computer program.
A readable storage medium having stored thereon a computer program which, when being executed by a processor, carries out the steps of the above-mentioned infrared thermal imaging image enhancement method.
According to the method provided by the embodiment of the invention, the infrared image enhancement depth neural network is called to carry out image enhancement forward reasoning on the acquired infrared image data, and each pixel point in the network is enhanced through the pixel enhancement network, so that the problem of difference of pixel response of the infrared thermal imaging detector is solved; meanwhile, a global feature extraction network is called to extract the overall features of the image, the overall features act on the output of a pixel enhancement network to ensure the local and overall consistency of enhancement, noise can be inhibited, and the image enhancement effect is improved; in addition, the process is a parameter-free image enhancement process, manual parameter setting is not needed, user side operation can be simplified, and user experience in application can be improved.
Accordingly, embodiments of the present invention further provide an infrared thermal imaging image enhancement apparatus, a device, and a readable storage medium corresponding to the infrared thermal imaging image enhancement method, which have the above technical effects and are not described herein again.
Drawings
In order to more clearly illustrate the embodiments of the present invention or technical solutions in related arts, the drawings used in the description of the embodiments or related arts will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.
FIG. 1 is a flowchart illustrating an implementation of a method for enhancing an infrared thermal imaging image according to an embodiment of the present invention;
FIG. 2 is a schematic structural diagram of a pixel enhancement network according to an embodiment of the present invention;
FIG. 3 is a schematic structural diagram of a global feature extraction network according to an embodiment of the present invention;
FIG. 4 is a schematic structural diagram of a three-level infrared image enhanced deep neural network according to an embodiment of the present invention;
FIG. 5 is a schematic diagram of an infrared image enhanced deep neural network training in an embodiment of the present disclosure;
FIG. 6a is a schematic diagram of an infrared image to be enhanced input to an infrared image enhancement deep neural network according to an embodiment of the present invention;
FIG. 6b is a schematic diagram of an enhanced IR image according to an embodiment of the present invention;
FIG. 7a is a schematic diagram of a visible light image to be enhanced inputted to an infrared image enhancement deep neural network according to an embodiment of the present invention;
FIG. 7b is a schematic view of an enhanced visible light image according to an embodiment of the present invention;
FIG. 8 is a schematic structural diagram of an IR thermal imaging image intensifier device according to an embodiment of the present invention;
fig. 9 is a schematic structural diagram of an infrared thermal imaging image enhancement device in an embodiment of the present invention.
Detailed Description
The core of the invention is to provide an infrared thermal imaging image enhancement method which can improve the infrared image enhancement effect.
In order that those skilled in the art will better understand the disclosure, the invention will be described in further detail with reference to the accompanying drawings and specific embodiments. It is to be understood that the described embodiments are merely exemplary of the invention, and not restrictive of the full scope of the invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Referring to fig. 1, fig. 1 is a flowchart illustrating an infrared thermal imaging image enhancement method according to an embodiment of the present invention, the method including the following steps:
s101, acquiring infrared image data;
the infrared image data is image data generated by acquiring through an infrared thermal imager, and the shooting scene and the temperature of a shooting object of the infrared image data are not limited in the embodiment, so that the acquired infrared image data can be acquired according to actual use requirements.
S102, calling a pre-trained infrared image enhancement depth neural network to carry out image enhancement forward reasoning on infrared image data;
the method for image enhancement forward reasoning by calling the infrared image enhancement deep neural network is a parameter-free image enhancement method, manual parameter setting is not needed, user side operation can be simplified, and user experience during practice is improved.
The infrared image enhancement deep neural network is a pre-trained pixel-sensitivity-based lightweight infrared thermal imaging image enhancement network, the network is composed of a plurality of cascaded network layers, the structure of each network layer is the same, and internal parameters and configurations such as selected convolution kernels can be adaptively adjusted according to different levels.
Each network layer includes two networks: a pixel enhancement network and a global feature extraction network. Since the two network layers respectively perform different processing on the input image data, the two networks may be parallel in the network layer, or may be serial in some cases, which is not limited in this embodiment.
The pixel enhancement network is mainly used for enhancing each pixel point in image data input in the network layer. Because the pixel points of the same detector have different sensitivity degrees to different temperatures, the response curve difference of each pixel point of the infrared thermal imaging detector is large, and the response curve difference changes along with the change of the temperature, and the traditional convolution kernels such as 3 x 3, 5 x 5 and the like cannot describe the independent characteristics of each pixel of the infrared thermal imaging detector, therefore, the enhancement processing is carried out on each independent pixel in the embodiment, the description of the temperature response curve is carried out on each pixel point, the problem of the difference of the response of each pixel of the infrared thermal imaging detector is solved, and the image enhancement capability is improved.
The global feature extraction network is mainly used for carrying out global feature extraction on image data input in a network layer and acting on an enhanced image output by the pixel enhancement network so as to ensure local and global consistency in the enhanced image; and the different noise interferences of different pixel points in the enhanced image are reduced through the global characteristic effect, and the effect of inhibiting the noise is realized.
Each network layer generates a hierarchical result according to the network output results of the pixel enhancement network and the global feature extraction network, the two networks can be connected in series or in parallel, in order to improve the calculation speed, the two networks can respectively carry out corresponding processing (enhancement processing and overall feature extraction) on the output image data in parallel, then the results output by the two networks are subjected to data fusion, and the fused image data is used as the output result of the network layer. In this embodiment, an implementation manner of generating the layer-level result after the image processing results are generated by the pixel enhancement network and the global feature extraction network is not limited, that is, a data fusion manner is not limited, and data fusion may be performed in the forms of subtraction, multiplication, weighted sum, and the like, and optionally, an output result of the pixel enhancement network may be multiplied by an output result of the global feature extraction network to serve as network-level output image data. Experiments prove that the implementation mode is simple and easy when the multiplication is taken as a result fusion mode, and the generated image effect is good. For other data fusion methods, reference may be made to the description of this embodiment, and further description is omitted here.
In this embodiment, the network structures of the pixel enhancement network and the global feature extraction network are not limited, and the corresponding processing units may be selected to configure the network and the parameters according to the actual use requirements. Since the network structure is not limited, the data processing process of the two networks is not limited in this embodiment accordingly. For the sake of better understanding, a network structure of a pixel enhancement network and a global feature extraction network and a corresponding data processing procedure are introduced.
Alternatively, a pixel enhancement network is constructed as shown in fig. 2, and is mainly composed of a 1 × 1 convolution layer and an active layer connected in series. Wherein x is image data input to a network layer, if the image data is the first network layer, x is the received infrared image data, and if the image data is not the first network layer, x is the image data output by the previous network layer; and y is the image data output by the network layer. The characteristics of each independent pixel are extracted through 1-by-1 convolution in the pixel enhancement network, the nonlinear mapping capability of the pixel enhancement network is enhanced by adopting an active layer, and the nonlinear mapping capability is multiplied by an original signal to be used as the output of each pixel, and the mathematical form of the pixel enhancement network is as follows:
wherein x is the input of each pixel, y is the output of each pixel, and w is the learned enhanced feature weight of each pixel.
Correspondingly, the process of performing enhancement processing on each pixel point in the received image data by the pixel enhancement network specifically includes the following steps:
(1) Performing feature extraction on each pixel point in received image data by a convolution layer in a pixel enhancement network to obtain pixel features;
(2) Carrying out nonlinear mapping capability enhancement processing on pixel characteristics by an active layer in a pixel enhancement network to obtain enhanced data;
(3) Multiplying the enhancement data by the signal value of the corresponding pixel point to obtain an enhancement result corresponding to the pixel point;
(4) And counting the enhancement results corresponding to all the pixel points, and taking the enhancement results as the output results of the pixel enhancement network.
The final generated output result can be output in a matrix form, and then according to the above (equation 1), the generated output result can be expressed as:
wherein X ∈ R m*n ，W i R m*n ，Y i ∈R m*n M is the number of rows of the input image, n is the number of columns of the input image, X is the original un-enhanced infrared image, Y i For the prediction output of each layer, g i Extracting a gain factor, W, for network learning for global features i A weight matrix learned for each pixel, i ∈ {1,2,3}, Y 0 ＝X。
The sigmoid activation layer is specifically selected as the activation layer in the pixel enhancement network, the activation mode is simple to realize, the resource occupation is small, and the pixel enhancement network is suitable for light-weight equipment. Of course, other types of active layers may be used, which is not limited in this embodiment.
Optionally, a global feature extraction network is configured as shown in fig. 3, and mainly includes a convolution layer (Conv) and an averaging layer (mean), where the convolution layer extracts local and global features of an image, and outputs the number of images to the convolution layerAnd performing averaging processing as an output result of the global feature extraction network, and fusing the output result with an output result of the pixel enhancement network (for example, multiplying the output result of the global feature extraction network by the output result of the pixel enhancement network) to control the overall gain of the pixel enhancement network and suppress the enhancement noise. Where x is image data input to the network layer and Y c And g is a gain factor extracted by the convolutional layer in the global feature extraction network.
Under the network structure shown in fig. 3, the process of extracting the overall features of the received image data by the global feature extraction network specifically includes the following steps:
(1) The convolution layer in the global feature extraction network carries out feature extraction on each part of image in the received image data to obtain grouping features;
(2) And counting the average value of all the grouping features in the image data, and taking the average value as an output result of the global feature extraction network.
In addition, it should be noted that, in this embodiment, the number of network layers is not limited, and 2,3, 4 or more network layers may be set, the more the number of network layers is, the better the image enhancement effect is, but the more complex the algorithm is, the lower the processing efficiency is, in order to ensure the image enhancement effect and simultaneously improve the calculation efficiency, a network model of a 3-layer network layer may be selected for construction. Fig. 4 is a schematic structural diagram of an infrared image enhanced deep neural network composed of three network layers based on the network structure configuration shown in fig. 2 and 3.
In fig. 4, the entire network is formed by cascading and stacking three identical network layer structures, and each network layer is composed of a pixel enhancement network and a global feature extraction network. Wherein X is the acquired original infrared image data, Y 1 、Y 2 、Y 3 The outputs of the pixel enhancement networks in each network layer are respectively, the convolution kernels of the convolution layers in each network layer are respectively 7 × 7, 3 × 3 and 3 × 3, and stride (step size) is 2.Y is c1 、Y c2 Extracting convolutional layer outputs of the network for global features in each network layer, g 1 、g 2 、g 3 Are respectively eachThe global feature in the network layer extracts the gain factor extracted by the network convolutional layer, and the gain factor is automatically obtained through learning. The size of the model is about 146k, 3ms is needed when the model is applied to RTX6000 processing of 720p images, and the structure occupies small space and is high in calculation speed, can be transplanted to a low-end AI chip, and is small in limitation on application equipment.
In addition, it should be noted that, in this embodiment, the training process of the infrared image enhanced deep neural network is not limited, and a network training process in the related art may be referred to. For the sake of understanding, a network training mode is introduced by using the network structure shown in fig. 4, and fig. 5 is a schematic diagram of infrared image enhanced deep neural network training.
The computer with an operating system of Ubuntu 18.04 is used for training, a training and testing model of the infrared image enhanced deep neural network is built under a Pythroch 1.4 framework, and the used server is RTX 6000. The data set was acquired using an HD1280 infrared camera.
Firstly, 10000 original infrared images with different temperatures and different scenes are obtained through an infrared thermal imager, and a high-quality training image set (original infrared image data correspond to processed high-quality images one by one, and the high-quality images refer to enhanced images) is obtained by utilizing a traditional image enhancement method (such as high dynamic enhancement of guided filtering, tone mapping, stripe removal, pot cover removal, noise reduction and the like). 10000 images were collected, 9000 of which were used as training samples and 1000 of which were used as verification samples.
In FIG. 5, Y is the training sample and the verification sample, loss is the Loss function, and the calculation formula is
Where N is the number of samples trained, Y 3 Is the output of the third network layer.
The infrared image enhanced depth neural network was trained using Adam gradient descent method for 200 rounds with a batch size of 12 and an input/output resolution of 1280 × 720. For the front 150 rounds, the learning rate is 10- 4 The remaining 50 rounds reduce the learning rate to 10- 5 And then outputting the infrared image to enhance the deep neural network.
It should be noted that, in this embodiment, only the structure and the training mode of the infrared image enhanced deep neural network are described as an example, and the construction and the training mode of the infrared image enhanced deep neural network based on other structures can refer to the above description, which is not described herein again.
And S103, acquiring an image output by the infrared image enhanced depth neural network, and taking the image as an enhanced image corresponding to the infrared image data.
An image output by the infrared image enhanced deep neural network is obtained and used as an enhanced image corresponding to infrared image data, as shown in fig. 6a, an infrared image to be enhanced is input to the infrared image enhanced deep neural network, and as shown in fig. 6b, an enhanced infrared image corresponding to fig. 6a and output by the infrared image enhanced deep neural network is obtained. As can be seen from comparison between FIG. 6a and FIG. 6b, the details of the enhanced infrared image are significantly better than those of the non-enhanced image, such as the keyboard, the joint of wall bricks, etc., so that the infrared image enhanced deep neural network provided by the embodiment can be invoked to achieve a better image enhancement effect.
Although the infrared thermal imaging detector is originally designed for the situation that the difference of response curves of each pixel point of the infrared thermal imaging detector is large and changes along with the change of temperature, the infrared image enhancement depth neural network provided by the embodiment is also effective in enhancing a visible light image, as shown in fig. 7a, the visible light image to be enhanced (obtained by limited acquisition of light rays) input into the infrared image enhancement depth neural network is shown, as shown in fig. 7b, the enhanced visible light image corresponding to fig. 7a and output by the infrared image enhancement depth neural network is shown, as can be seen by comparing fig. 7a with fig. 7b, details in the enhanced visible light image are clear and obvious, and the infrared thermal imaging detector has good generalization capability, not only has a good enhancement effect on the infrared thermal imaging image, but also has an enhancement effect on the visible light image without modifying any model.
Based on the introduction, the technical scheme provided by the embodiment of the invention calls the infrared image enhancement deep neural network to carry out image enhancement forward reasoning on the acquired infrared image data, and the network carries out enhancement processing on each pixel point through the pixel enhancement network, so that the problem of difference of response of each pixel of the infrared thermal imaging detector is solved; meanwhile, the global feature extraction network is called to extract the overall features of the image, the overall features act on the output of the pixel enhancement network to ensure the local and overall consistency of enhancement, noise can be inhibited, and the image enhancement effect is improved; in addition, the process is a parameter-free image enhancement process, manual parameter setting is not needed, user side operation can be simplified, and user experience in application can be improved.
Corresponding to the above method embodiments, the embodiments of the present invention further provide an infrared thermal imaging image enhancement apparatus, and the infrared thermal imaging image enhancement apparatus described below and the infrared thermal imaging image enhancement method described above may be referred to correspondingly.
Referring to fig. 8, the apparatus includes the following modules:
the image acquiring unit 110 is mainly used for acquiring infrared image data;
the network reasoning unit 120 is mainly used for calling a pre-trained infrared image enhanced depth neural network to perform image enhanced forward reasoning on the infrared image data; the infrared image enhancement deep neural network comprises a plurality of cascaded network layers; the network layer comprises: the pixel enhancement network is used for enhancing each pixel point and the global feature extraction network is used for extracting the overall features of the image;
the output obtaining unit 130 is mainly configured to obtain an image output by the infrared image enhanced depth neural network, as an enhanced image corresponding to the infrared image data.
Corresponding to the above method embodiment, an embodiment of the present invention further provides an infrared thermal imaging image enhancement device, and an infrared thermal imaging image enhancement device described below and an infrared thermal imaging image enhancement method described above may be referred to in a mutually corresponding manner.
The infrared thermal imaging image enhancement device includes:
a memory for storing a computer program;
a processor for implementing the steps of the infrared thermal imaging image enhancement method of the above method embodiments when executing the computer program.
Specifically, referring to fig. 9, a schematic structural diagram of an infrared thermal imaging image enhancement apparatus provided in this embodiment is a schematic structural diagram, where the infrared thermal imaging image enhancement apparatus may generate relatively large differences due to different configurations or performances, and may include one or more processors (CPUs) 322 (for example, one or more processors) and a memory 332, where the memory 332 stores one or more computer applications 342 or data 344. Memory 332 may be, among other things, transient or persistent storage. The program stored in memory 332 may include one or more modules (not shown), each of which may include a sequence of instructions operating on a data processing device. Still further, processor 322 may be configured to communicate with memory 332 to execute a series of instruction operations in memory 332 on infrared thermal imaging image intensifier device 301.
The infrared thermal imaging image enhancement apparatus 301 may also include one or more power supplies 326, one or more wired or wireless network interfaces 350, one or more input-output interfaces 358, and/or one or more operating systems 341.
The steps in the above-described infrared thermal imaging image enhancement method may be implemented by the structure of an infrared thermal imaging image enhancement device.
Corresponding to the above method embodiment, the embodiment of the present invention further provides a readable storage medium, and a readable storage medium described below and an infrared thermal imaging image enhancement method described above may be correspondingly referred to with each other.
A readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the method for enhancing an infrared thermal imaging image of the above-mentioned method embodiment.
The readable storage medium may be a usb disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and various other readable storage media capable of storing program codes.
Those of skill would further appreciate that the various illustrative components and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or combinations of both, and that the various illustrative components and steps have been described above generally in terms of their functionality in order to clearly illustrate this interchangeability of hardware and software. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.
Claims (8)
1. An infrared thermal imaging image enhancement method is characterized by comprising the following steps:
acquiring infrared image data;
calling a pre-trained infrared image enhancement depth neural network to carry out image enhancement forward reasoning on the infrared image data; the infrared image enhancement deep neural network comprises a plurality of cascaded network layers; the network layer comprises: the pixel enhancement network is used for enhancing each pixel point and the global feature extraction network is used for extracting the overall features of the image;
acquiring an image output by the infrared image enhanced depth neural network, and taking the image as an enhanced image corresponding to the infrared image data;
the image enhancement forward reasoning processing of the network layer comprises the following steps:
determining image data input by the network layer as target image data; the pixel enhancement network carries out enhancement processing on each pixel point in the target image data to obtain an output result of the pixel enhancement network; the global feature extraction network carries out overall feature extraction on the target image data to obtain an output result of the global feature extraction network; performing data fusion on the output result of the pixel enhancement network and the output result of the global feature extraction network to obtain network layer output image data;
the pixel enhancement network performs enhancement processing on each pixel point in the target image data, and the enhancement processing comprises the following steps:
the convolution layer in the pixel enhancement network performs feature extraction on each pixel point in the target image data to obtain pixel features; an activation layer in the pixel enhancement network performs nonlinear mapping capability enhancement processing on the pixel characteristics to obtain enhanced data; multiplying the enhancement data by the signal value of the corresponding pixel point to obtain an enhancement result corresponding to the pixel point; and counting the enhancement results corresponding to all the pixel points, and taking the enhancement results as the output results of the pixel enhancement network.
2. The infrared thermal imaging image enhancement method according to claim 1, wherein the activation layer is specifically: and (5) sigmoid activating a layer.
3. The infrared thermal imaging image enhancement method of claim 1, wherein the global feature extraction network performs global feature extraction on the target image data, and comprises:
the convolution layer in the global feature extraction network carries out feature extraction on each part of image in the target image data to obtain grouping features;
and counting the average value of all the grouping features in the image data, and taking the average value as an output result of the global feature extraction network.
4. The infrared thermal imaging image enhancement method according to claim 1, wherein the performing data fusion on the output result of the pixel enhancement network and the output result of the global feature extraction network includes:
and multiplying the output result of the pixel enhancement network with the output result of the global feature extraction network.
5. The infrared thermal imaging image enhancement method of claim 1, characterized in that the infrared image enhancement deep neural network comprises 3 cascaded network layers.
6. An infrared thermal imaging image intensifier, comprising:
the image acquisition unit is used for acquiring infrared image data;
the network reasoning unit is used for calling a pre-trained infrared image enhanced depth neural network to carry out image enhanced forward reasoning on the infrared image data; the infrared image enhancement deep neural network comprises a plurality of cascaded network layers; the network layer comprises: the pixel enhancement network is used for enhancing each pixel point and the global feature extraction network is used for extracting the overall features of the image;
the output acquisition unit is used for acquiring an image output by the infrared image enhanced depth neural network as an enhanced image corresponding to the infrared image data;
wherein, the network inference unit is specifically configured to:
determining image data input by the network layer as target image data; the pixel enhancement network carries out enhancement processing on each pixel point in the target image data to obtain an output result of the pixel enhancement network; the global feature extraction network carries out overall feature extraction on the target image data to obtain an output result of the global feature extraction network; performing data fusion on the output result of the pixel enhancement network and the output result of the global feature extraction network to obtain network layer output image data;
the network inference unit is specifically further configured to:
the convolution layer in the pixel enhancement network performs feature extraction on each pixel point in the target image data to obtain pixel features; an activation layer in the pixel enhancement network carries out nonlinear mapping capability enhancement processing on the pixel characteristics to obtain enhanced data; multiplying the enhancement data by the signal value of the corresponding pixel point to obtain an enhancement result corresponding to the pixel point; and counting the enhancement results corresponding to all the pixel points, and taking the enhancement results as the output results of the pixel enhancement network.
7. An infrared thermal imaging image intensifier device, comprising:
a memory for storing a computer program;
a processor for implementing the steps of the infrared thermographic image enhancement method according to any of claims 1 to 5 when executing said computer program.
8. A readable storage medium, having stored thereon a computer program which, when being executed by a processor, carries out the steps of the infrared thermographic image enhancement method according to any of the claims 1 to 5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110572941.2A CN113256525B (en) | 2021-05-25 | 2021-05-25 | Infrared thermal imaging image enhancement method, device, equipment and readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110572941.2A CN113256525B (en) | 2021-05-25 | 2021-05-25 | Infrared thermal imaging image enhancement method, device, equipment and readable storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113256525A CN113256525A (en) | 2021-08-13 |
| CN113256525B true CN113256525B (en) | 2023-01-10 |
Family
ID=77184371
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110572941.2A Active CN113256525B (en) | 2021-05-25 | 2021-05-25 | Infrared thermal imaging image enhancement method, device, equipment and readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113256525B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116797471A (en) * | 2022-12-20 | 2023-09-22 | 慧之安信息技术股份有限公司 | Underwater target image detection method and system based on deep learning |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109102483A (en) * | 2018-07-24 | 2018-12-28 | 厦门美图之家科技有限公司 | Image enhancement model training method, device, electronic equipment and readable storage medium storing program for executing |
| CN109558840A (en) * | 2018-11-29 | 2019-04-02 | 中国科学院重庆绿色智能技术研究院 | A kind of biopsy method of Fusion Features |
| CN110097617A (en) * | 2019-04-25 | 2019-08-06 | 北京理工大学 | Image interfusion method based on convolutional neural networks Yu conspicuousness weight |
| CN110717864A (en) * | 2019-08-29 | 2020-01-21 | 苏州千视通视觉科技股份有限公司 | Image enhancement method and device, terminal equipment and computer readable medium |
Family Cites Families (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108564025A (en) * | 2018-04-10 | 2018-09-21 | 广东电网有限责任公司 | A kind of infrared image object identification method based on deformable convolutional neural networks |
| CN108961180B (en) * | 2018-06-22 | 2020-09-25 | 理光软件研究所（北京）有限公司 | Infrared image enhancement method and system |
| CN109544487A (en) * | 2018-09-30 | 2019-03-29 | 西安电子科技大学 | A kind of infrared image enhancing method based on convolutional neural networks |
| CN109784358B (en) * | 2018-11-23 | 2023-07-11 | 南京航空航天大学 | No-reference image quality evaluation method integrating artificial features and depth features |
| CN110084757B (en) * | 2019-04-15 | 2023-03-07 | 南京信息工程大学 | Infrared depth image enhancement method based on generation countermeasure network |
| CN111861914B (en) * | 2020-07-07 | 2023-02-03 | 天津大学 | Low-illumination color polarization image enhancement method based on deep neural network |
| CN112580558A (en) * | 2020-12-25 | 2021-03-30 | 烟台艾睿光电科技有限公司 | Infrared image target detection model construction method, detection method, device and system |
-
2021
- 2021-05-25 CN CN202110572941.2A patent/CN113256525B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109102483A (en) * | 2018-07-24 | 2018-12-28 | 厦门美图之家科技有限公司 | Image enhancement model training method, device, electronic equipment and readable storage medium storing program for executing |
| CN109558840A (en) * | 2018-11-29 | 2019-04-02 | 中国科学院重庆绿色智能技术研究院 | A kind of biopsy method of Fusion Features |
| CN110097617A (en) * | 2019-04-25 | 2019-08-06 | 北京理工大学 | Image interfusion method based on convolutional neural networks Yu conspicuousness weight |
| CN110717864A (en) * | 2019-08-29 | 2020-01-21 | 苏州千视通视觉科技股份有限公司 | Image enhancement method and device, terminal equipment and computer readable medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113256525A (en) | 2021-08-13 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11882357B2 (en) | Image display method and device | |
| CN111402130B (en) | Data processing method and data processing device | |
| EP3716198A1 (en) | Image reconstruction method and device | |
| CN111402146B (en) | Image processing method and image processing apparatus | |
| JP6798183B2 (en) | Image analyzer, image analysis method and program | |
| CN108898567A (en) | Image denoising method, apparatus and system | |
| CN112541877B (en) | Defuzzification method, system, equipment and medium for generating countermeasure network based on condition | |
| CN111914997B (en) | Method for training neural network, image processing method and device | |
| CN112446834A (en) | Image enhancement method and device | |
| WO2022134971A1 (en) | Noise reduction model training method and related apparatus | |
| WO2019019086A1 (en) | Image contrast enhancement method and device, and storage medium | |
| CN109993707A (en) | Image de-noising method and device | |
| CN113011562A (en) | Model training method and device | |
| CN110148088B (en) | Image processing method, image rain removing method, device, terminal and medium | |
| CN114730456A (en) | Training method of neural network model, image processing method and device thereof | |
| US20240311976A1 (en) | Image correction model training method and apparatus, image correction method and apparatus, and computer device | |
| CN114627034A (en) | Image enhancement method, training method of image enhancement model and related equipment | |
| CN116547694A (en) | Method and system for deblurring blurred images | |
| CN113256525B (en) | Infrared thermal imaging image enhancement method, device, equipment and readable storage medium | |
| CN113962859A (en) | Panorama generation method, device, equipment and medium | |
| CN116797504A (en) | Image fusion method, electronic device and storage medium | |
| CN114372931A (en) | Target object blurring method and device, storage medium and electronic equipment | |
| CN114708172A (en) | Image fusion method, computer program product, storage medium, and electronic device | |
| CN114708173A (en) | Image fusion method, computer program product, storage medium, and electronic device | |
| CN108734712B (en) | Background segmentation method and device and computer storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| EE01 | Entry into force of recordation of patent licensing contract | ||
| EE01 | Entry into force of recordation of patent licensing contract |
Application publication date: 20210813 Assignee: INFIRAY TECHNOLOGIES CO.,LTD. Assignor: Yantai Airui Photo-Electric Technology Co.,Ltd. Contract record no.: X2024980006380 Denomination of invention: Infrared thermal imaging image enhancement method, device, equipment, and readable storage medium Granted publication date: 20230110 License type: Common License Record date: 20240530 |