CN112991200B - Method and device for adaptively enhancing infrared image - Google Patents
Method and device for adaptively enhancing infrared image Download PDFInfo
- Publication number
- CN112991200B CN112991200B CN202110183673.5A CN202110183673A CN112991200B CN 112991200 B CN112991200 B CN 112991200B CN 202110183673 A CN202110183673 A CN 202110183673A CN 112991200 B CN112991200 B CN 112991200B
- Authority
- CN
- China
- Prior art keywords
- image
- infrared
- images
- network
- visible light
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 35
- 230000002708 enhancing effect Effects 0.000 title description 6
- 238000012545 processing Methods 0.000 claims abstract description 49
- 238000012549 training Methods 0.000 claims abstract description 24
- 238000013528 artificial neural network Methods 0.000 claims abstract description 16
- 230000008569 process Effects 0.000 claims abstract description 8
- 238000013135 deep learning Methods 0.000 claims description 28
- 230000006870 function Effects 0.000 claims description 27
- 238000006243 chemical reaction Methods 0.000 claims description 18
- 238000010606 normalization Methods 0.000 claims description 11
- 230000005540 biological transmission Effects 0.000 claims description 10
- 230000003042 antagnostic effect Effects 0.000 claims description 9
- 230000004913 activation Effects 0.000 claims description 7
- 238000005516 engineering process Methods 0.000 claims description 6
- 239000013598 vector Substances 0.000 claims description 5
- 230000003044 adaptive effect Effects 0.000 claims description 4
- 238000013459 approach Methods 0.000 claims description 4
- 230000003014 reinforcing effect Effects 0.000 claims 1
- 238000013461 design Methods 0.000 abstract description 11
- 230000007547 defect Effects 0.000 abstract description 3
- 230000000694 effects Effects 0.000 description 9
- 238000012544 monitoring process Methods 0.000 description 6
- 238000003331 infrared imaging Methods 0.000 description 5
- 230000015572 biosynthetic process Effects 0.000 description 2
- 238000001514 detection method Methods 0.000 description 2
- 230000004927 fusion Effects 0.000 description 2
- 238000012360 testing method Methods 0.000 description 2
- 230000001960 triggered effect Effects 0.000 description 2
- 230000000007 visual effect Effects 0.000 description 2
- 230000001133 acceleration Effects 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000013527 convolutional neural network Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000018109 developmental process Effects 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 238000011156 evaluation Methods 0.000 description 1
- 238000002474 experimental method Methods 0.000 description 1
- 238000003384 imaging method Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 238000007500 overflow downdraw method Methods 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 230000011218 segmentation Effects 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
- 238000010415 tidying Methods 0.000 description 1
- 238000012795 verification Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/90—Dynamic range modification of images or parts thereof
- G06T5/92—Dynamic range modification of images or parts thereof based on global image properties
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10048—Infrared image
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- Artificial Intelligence (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Health & Medical Sciences (AREA)
- Life Sciences & Earth Sciences (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Image Processing (AREA)
- Image Analysis (AREA)
Abstract
The invention provides an infrared image self-adaptive enhancement method and device, which aims to solve the technical problems that the existing infrared image enhancement method has the defects of image local detail loss, insufficient image detail and incapability of forming an end-to-end solution, capability of exploring and enriching image details and capability of automatically repairing the image details. The invention designs an infrared image enhancement model based on an countermeasure generation network, fully utilizes the characteristics of the countermeasure neural network, realizes the dynamic balance of the generation sub-network and the identification sub-network through the neural network design and the loss function design, and can realize the end-to-end infrared image enhancement processing; in the model training process, the real visible light gray level image is used as a condition, the identification sub-network D is used for supervising and generating the sub-network G, and the mutual game between the sub-network and the identification sub-network is generated, so that the network model has good infrared image enhancement capability.
Description
Technical Field
The invention relates to a method and a device for adaptively enhancing an infrared image.
Background
With development and maturity of infrared imaging technology, various infrared imaging devices suitable for civil use are continuously emerging, and the infrared imaging devices play an increasingly important role in various fields of national economy. The infrared imaging has important applications of day and night observation and thermal target detection, and is widely applied to the fields of security monitoring, auxiliary driving, fire-fighting police, industrial monitoring, electric power monitoring and the like. However, due to the limitation of the infrared imaging principle, the infrared image has the characteristics of dark whole body, low contrast, blurred edge, large noise, unobvious detailed information, poor visual effect and the like.
To overcome the above-mentioned drawbacks of the infrared image, a histogram equalization method of the infrared image is disclosed in patent document publication No. CN104252700B, comprising: reading an infrared image, dividing the infrared image into at least two sub-images, carrying out histogram equalization processing on each sub-image to obtain an equalized sub-image of the sub-image, and then splicing the histogram equalized sub-images; the method comprises the core ideas that an infrared image is subjected to segmentation processing to obtain a plurality of sub-images, then each sub-image is subjected to histogram equalization enhancement processing in parallel, and the sub-images after histogram equalization are spliced and fused into a final infrared image after histogram equalization; the method has the defects that: (1) the contrast of the tidying image depends on the original infrared image, the local contrast of the image is insufficient, and the phenomena of detail loss and the like exist; (2) the image features of the response visible light are not combined, the image detail is not abundant enough, and the subsequent application such as target detection is difficult.
The patent document with publication number of CN110009569A discloses an infrared and visible light image fusion method based on a lightweight convolutional neural network. The key point of the scheme is that the neural network is utilized to extract the infrared image characteristics and the visible light image characteristics respectively, then a characteristic weight map is constructed, and then the fusion strategy is utilized to realize the fusion of the infrared image and the visible light image. Although the infrared image and the visible light image are fused, an end-to-end solution cannot be formed in engineering application, and each extracted feature is independent, and the method does not have the capability of exploring and enriching the image details and the capability of automatically repairing the image details.
Disclosure of Invention
The invention provides an infrared image self-adaptive enhancement method and device, which aims to solve the technical problems that the existing infrared image enhancement method has the defects of image local detail loss, insufficient image detail and incapability of forming an end-to-end solution, capability of exploring and enriching image details and capability of automatically repairing the image details.
The technical scheme of the invention is as follows:
the method for adaptively enhancing the infrared image is characterized by comprising the following steps of:
step 1: constructing an image dataset
Selecting an existing ImageNet data set as a basis for constructing an image data set, converting an image in the ImageNet data set into a visible light gray image by using gray level conversion, converting the gray level image into an infrared image by using contrast conversion or histogram conversion, and taking each obtained infrared image and a visible light gray level image corresponding to the infrared image as one sample of the image data set;
step 2: model training
Constructing an antagonistic neural network, inputting a sample in an image data set into the constructed antagonistic neural network to perform deep learning training on the antagonistic neural network to obtain an infrared image enhancement model;
step 3: downloading the infrared image enhancement model obtained in the step 2 into a parallel processing module;
step 4: the infrared and visible light image acquisition device acquires infrared images and visible light images of a target scene in real time, when the number of groups of the acquired infrared images and visible light gray images is 10% greater than the number of samples in the current image data set, randomly replacing the corresponding number of samples in the image data set with the acquired infrared images and visible light gray images, and then returning to the step 2; simultaneously, copying one part of infrared image acquired in real time and sending the copied part of infrared image to a parallel processing module;
step 5: and simultaneously carrying out parallel enhancement processing on a plurality of infrared images by utilizing an infrared image enhancement model arranged in a parallel processing module, and outputting corresponding enhanced images.
Further, the countermeasure neural network constructed in the step 2 includes generating a sub-network G and an authentication sub-network D;
the generation sub-network is used for generating an enhanced infrared image; the identification sub-network D is used for judging the authenticity of the visible light gray level image and the enhanced infrared image to obtain a probability value of 0-1, wherein 0 represents that the enhanced infrared image quality is very low and has no credibility, and 1 represents that the enhanced infrared image quality is very high and approaches to the real image.
Further, the network structure of the generated sub-network is as follows:
CR-CBR-CBR-D-Tanh
wherein, C is a convolution layer, R is a ReLu activation function, B is a batch normalization layer, D is a deconvolution layer, and Tanh is:z is a feature image obtained after deconvolution; e is the bottom of natural logarithm;
the loss function of the generated subnetwork is:
L G ＝E[λ×L cout +log(D(G(X),X)]
wherein:
e is Euclidean distance;
L cont a quality loss function representing an enhanced infrared image and a visible gray scale image, defined by an L2 distance,
y is a visible light gray scale image corresponding to the input infrared image;
g (X) is an enhanced infrared image obtained by utilizing the generation sub-network; d is the output of the authentication sub-network;
the network structure of the authentication sub-network D is:
CL-CBL-CBL-CBL-C-Sigmoid；
wherein: sigmoid:c is a convolution layer, B is a batch normalization layer, and L is a leakage ReLU activation function;
the loss function of the authentication subnetwork D is:
wherein:
x is an infrared image, Y is a visible light gray scale image, and G (X) is an enhanced infrared image obtained by utilizing a generation sub-network.
The invention also provides an infrared image self-adaptive enhancement device, which is characterized in that: the system comprises an image data set, an infrared image enhancement module based on deep learning, an infrared image enhancement model, a model downloading module, an infrared and visible light image acquisition device, an image transmission module and a parallel processing module;
the image dataset is used for providing image samples for model training for the infrared image enhancement module based on the deep learning;
the infrared image enhancement module based on the deep learning is used for the deep learning training to obtain the infrared image enhancement model;
the model lower loading module is used for downloading the infrared image enhancement model into the parallel processing module in a remote service bus mode;
the infrared and visible light image acquisition device is used for acquiring infrared images and corresponding visible light gray images in real time and transmitting the infrared images and the corresponding visible light gray images to the image transmission module and the parallel processing module;
the parallel processing module is used for carrying out parallel enhancement processing on the received multiple infrared images;
the image transmission module is used for transmitting the visible light gray level image and the infrared image acquired by the infrared and visible light image acquisition device to the image data set.
Further, the image dataset is constructed according to the following method:
the visible light image is converted into a visible light gray image by gray level conversion, then the gray level image is converted into an infrared image by contrast conversion or histogram conversion, and each obtained infrared image and the corresponding visible light gray level image are taken as one sample of an image data set.
Further, the infrared image enhancement module based on deep learning comprises a generation sub-network G and an identification sub-network D;
the generation sub-network is used for generating an enhanced infrared image; the identification sub-network D is used for judging the authenticity of the visible light gray level image and the enhanced infrared image to obtain a probability value of 0-1, wherein 0 represents that the enhanced infrared image quality is very low and has no credibility, and 1 represents that the enhanced infrared image quality is very high and approaches to the real image.
Further, the network structure of the generated sub-network is as follows:
CR-CBR-CBR-D-Tanh
wherein, C is a convolution layer, R is a ReLu activation function, B is a batch normalization layer, D is a deconvolution layer, and Tanh is:z is a feature image obtained after deconvolution; e is the bottom of natural logarithm;
the loss function of the generated subnetwork is:
L G ＝E[λ×L cout +log(D(G(X),X)]
wherein:
e is Euclidean distance;
L cont a quality loss function representing an enhanced infrared image and a visible gray scale image, defined by an L2 distance,
y is a visible light gray scale image corresponding to the input infrared image;
g (X) is an enhanced infrared image obtained by utilizing the generation sub-network; d is the output of the authentication sub-network;
the network structure of the authentication sub-network D is:
CL-CBL-CBL-CBL-C-Sigmoid；
wherein: sigmoid:c is a convolution layer, B is a batch normalization layer, and L is a Leaky ReLU laserA living function;
the loss function of the authentication subnetwork D is:
wherein:
x is an infrared image, Y is a visible light gray scale image, and G (X) is an enhanced infrared image obtained by utilizing a generation sub-network.
Compared with the prior art, the invention has the advantages that:
1. in the model training process, the invention uses the real visible light gray level image as the condition, and monitors the generation sub-network G by means of the identification sub-network D, and generates the mutual game between the sub-network and the identification sub-network, so that the network model has good infrared image enhancement capability.
2. The invention designs an infrared image enhancement model based on an countermeasure generation network, fully utilizes the characteristics of the countermeasure neural network, realizes the dynamic balance of the generation sub-network and the identification sub-network through the neural network design and the loss function design, and can realize the end-to-end infrared image enhancement processing.
3. The invention provides a synchronous update mechanism for infrared image enhancement training and dynamic acquisition of an infrared and visible light image acquisition device, which realizes automatic optimization of an infrared image enhancement model, further improves the infrared image enhancement processing effect and finally realizes the infrared image enhancement application of off-line modeling-on-line application-sustainable update.
4. The invention has been proved by system verification under the monitoring scene of the electric power system equipment, the result shows that the invention can effectively promote the detail characteristics of the infrared image, and the image contrast is improved by a self-adaptive mode, and the detail level of the image is effectively enhanced by introducing the visible light characteristics, thereby providing a good image foundation for the image recognition and state monitoring of the subsequent equipment monitoring.
Drawings
Fig. 1 is a schematic diagram of an infrared image adaptive enhancement device of the present invention.
FIG. 2 is an infrared image enhancement model of the present invention based on an countermeasure generation network.
FIG. 3 is a graph of the contrast of the effect of an original image before and after enhancement with an enhanced image according to the present invention, (a) is an original infrared image; (b) is an enhanced infrared image.
Detailed Description
The invention is further described below with reference to the accompanying drawings.
As shown in fig. 1, the method for adaptively enhancing an infrared image provided by the invention comprises the following steps:
step 1: constructing an image dataset
Selecting an existing ImageNet data set as a basis for constructing an image data set, converting an image in the ImageNet data set into a visible light gray image by using gray level conversion, converting the gray level image into an infrared image by using contrast conversion or histogram conversion, and taking each obtained infrared image and a visible light gray level image corresponding to the infrared image as one sample of the image data set;
step 2: model training
Constructing an antagonistic neural network, inputting a sample in an image data set into the constructed antagonistic neural network to perform deep learning training on the antagonistic neural network to obtain an infrared image enhancement model;
step 3: downloading the infrared image enhancement model obtained in the step 2 into a parallel processing module;
step 4: the infrared and visible light image acquisition device acquires infrared images and visible light images of a target scene in real time, when the number of groups of the acquired infrared images and visible light gray images is 10% greater than the number of samples in the current image data set, randomly replacing the corresponding number of samples in the image data set with the acquired infrared images and visible light gray images, and then returning to the step 2; simultaneously, copying one part of infrared image acquired in real time and sending the copied part of infrared image to a parallel processing module;
step 5: and simultaneously carrying out parallel enhancement processing on a plurality of infrared images by utilizing an infrared image enhancement model arranged in a parallel processing module, and outputting the enhanced images.
As shown in FIG. 1, the invention also provides an infrared image self-adaptive enhancement device for realizing the infrared image self-adaptive enhancement method, which comprises an image data set, an infrared image enhancement module based on deep learning, an infrared image enhancement model, a model downloading module, an infrared and visible light image acquisition device, an image transmission module and a parallel processing module.
1. Image data set
The image data set is an image set used for training an infrared image enhancement module based on deep learning, and in view of the fact that infrared images are generally lacking in the infrared image enhancement based on a neural network at present, the infrared image is one of key factors capable of achieving a better image processing effect based on the deep learning infrared image enhancement technology. The present invention therefore primarily builds an image dataset with a sufficient number of samples.
The construction method of the image dataset comprises the following steps:
selecting an existing ImageNet data set as a basis for constructing an image data set, converting an image in the ImageNet data set into a visible light gray image by using gray level conversion, obtaining infrared images by using contrast conversion or histogram conversion technology on the gray level image, and taking each obtained infrared image and the corresponding visible light gray level image as one sample of the image data set; in this embodiment, the basic data set includes 10000 samples, where the resolution of the visible light gray scale image is: 512x512, infrared image resolution is: 512x512.
The method has the advantages that the problem of infrared image shortage can be solved, a training set with enough sample numbers is provided for the infrared image enhancement module based on deep learning, but because the infrared images in the basic data set constructed by the method are not real infrared images, if the infrared image enhancement module based on deep learning is trained by using only the basic data set, the infrared image enhancement model obtained by training is insufficient in infrared image coverage characteristics, and the infrared image enhancement effect is not ideal. To solve this problem, the present invention dynamically updates the image dataset based on the application scenario.
A method for dynamically updating an image dataset:
the infrared and visible light image acquisition device acquires a group of visible light gray level images and infrared images every set time (for example, 15 minutes), the acquisition time per day is 09:00-17:00, and the infrared and visible light gray level images of 32 groups are acquired by a single infrared and visible light image acquisition device per day. The acquisition time period is selected to be 09:00-17:00, and under the condition of better imaging conditions, infrared images and corresponding visible light gray level images are acquired at the same time, when the whole system is applied, not less than 50 sets of infrared and visible light image acquisition devices are deployed, and the acquisition time period is distributed in different scenes such as roads, forests, schools and the like, so that application pictures covering different scenes are ensured, namely: 1600 additional images are generated daily.
When the number of the acquired images is 10% greater than the number of the images in the current image dataset, randomly extracting 10% of the images from the current image dataset, carrying out replacement processing on the images by the acquired images, marking the images which are used for replacement and are acquired by the infrared and visible light image acquisition device in the replacement process, and when the images which are marked are updated and replaced next time, not carrying out replacement on the marked images, finally, when all the images in the whole image dataset are marked, indicating that the whole image dataset is updated, deleting the marks marked by the current sample images in the image dataset after the whole image dataset is updated, and then entering the next round of dynamic updating according to the same method.
Table 1 update threshold selection for image dataset
| Dynamically updating thresholds | Infrared image enhanced PSNR |
| 3％ | 25.36 |
| 6％ | 25.78 |
| 9％ | 26.02 |
| 10％ | 26.39 |
Table 1 is an experiment made by adopting different thresholds when the image dataset is dynamically updated, and after each dynamic update, an infrared image enhancement module based on deep learning is triggered to train to obtain a new infrared image enhancement model, and then the infrared image enhancement is performed by using the new infrared image enhancement model; by PSNR analysis of 512 infrared image enhancement training, it was found that as the threshold increases, the quality of the infrared image after enhancement gradually increases, and in view of quality and speed, 10% is finally selected as the threshold at the time of dynamic update.
2. Infrared image enhancement module based on deep learning
The infrared image enhancement module based on the deep learning is mainly used for the deep learning training, the input of the infrared image enhancement module is an image data set, and the output of the infrared image enhancement module is a trained infrared image enhancement model.
As shown in fig. 2, the network architecture of the deep learning-based infrared image enhancement module is based on an countermeasure generation network, and includes 2 sub-networks in total: a subnetwork G and an authentication subnetwork D are generated.
2.1 Generation of subnetworks
2.1.1 network formation
The generating sub-network is composed of 3 convolution layers conv and 1 deconvolution layer Deconv, and a batch of normalization networks are added after the second convolution layer and the third convolution layer, namely: in deep learning training, the Batch NN is a data normalization method proved by practice, and the effect of the Batch NN can accelerate the convergence rate of model training, so that the model training process is more stable, and a certain regularization effect is achieved.
2.1.2 generating a subnetwork design:
(1) the resolution of the input image of the generation sub-network is 256x256, so that the image in the image dataset needs to be scaled before being input into the generation sub-network;
(2) the network structure is as follows:
CR-CBR-CBR-D-Tanh
wherein, C is a convolution layer, R is a ReLu activation function, B is a batch normalization layer, D is a deconvolution layer, and Tanh is:z is a feature image obtained after deconvolution; e is the bottom of natural logarithm;
the size of the CR input image of the first convolution layer is 256x256, and the better characteristics are reserved through convolution operation and ReLu processing, so that 64 characteristic vectors are finally obtained, and then 2 convolution layer operations are followed to further strengthen the extracted characteristics.
(3) In order to reduce loss of image details, as shown in fig. 2, residual Connection Skip Connection is added after a convolution layer 1 and after a convolution layer 2, and residual Connection is performed again between the feature vector obtained after weighting processing and a convolution layer 3, so that the image feature vectors with different dimensions obtained by each convolution layer flow in each layer in a generation sub-network, and image detail features are reserved;
(4) generating a sub-network for generating an enhanced image (enhanced infrared image);
(5) and (3) loss function design: l (L) G ＝E[λ×L cout +log(D(G(X),X)]The method comprises the steps of carrying out a first treatment on the surface of the Wherein: l (L) cont A quality loss function representing an enhanced infrared image and a visible gray scale image, defined by an L2 distance,e is Euclidean distance; y is a visible light gray scale image corresponding to the input infrared image; g (X) is an enhanced infrared image obtained by utilizing the generation sub-network; d is the output of the authentication sub-network; the authentication sub-network is used for judging the authenticity of the visible light gray level image and the enhanced infrared image to obtainProbability value of 0-1, 0 represents that the enhanced infrared image quality is very low and has no credibility, 1 represents that the enhanced infrared image quality is very high and approaches to a real image, L G Using mathematical expectations to measure characteristics of generating and discriminating sub-networks in infrared image enhancement, where λ×L cout The generation of the sub-network is encouraged to generate more realistic infrared enhanced images, and the log (G (X), X) in the formula encourages the identification of the sub-network to distinguish the infrared enhanced images, and the two are finally balanced dynamically.
2.1.3 related network parameter settings:
| type(s) | Convolution kernel | Convolution step length | Filling |
| conv | 4×4 | 2×2 | 1×1 |
| conv | 3×3 | 1×1 | 1×1 |
| conv | 3×3 | 1×1 | 1×1 |
| deconv | 4×4 | 2×2 | 1×1 |
2.2 authentication subnetworks
2.2.1 network formation
The authentication sub-network is composed of 5 convolutional layers conv.
2.2.2 authentication subnetwork design:
(1) the network input image is: generating an enhanced image output by a sub-network and a visible light gray image corresponding to the enhanced image
(2) The network structure is as follows: CL-CBL-CBL-CBL-C-Sigmoid, wherein: sigmoid:c is a convolution layer, B is a batch normalization layer, and L is a leakage ReLU activation function;
(3) the network adopts 5 convolution layers to obtain characteristic images, and uses a Sigmoid function to perform normalization processing to judge the similarity degree [0-1] of the infrared enhanced image and the visible light gray image;
(4) and (3) loss function design:wherein: l (L) D For identifying the loss function of the sub-network, X is an infrared image, Y is a visible light gray scale image, G (X) is an enhanced infrared image obtained by generating the sub-network; the loss function is designed by adopting a cross entropy loss function, and aims to maximize the expected value of the loss function when the identification sub-network encounters real data such as a visible light gray level image, and judge the image as a generated image when the identification sub-network encounters generated data such as an infrared enhanced image, wherein the aim of the identification sub-network is that the generated image is identified in the whole training process, L D Maximum, but for the generation of a sub-network the goal is for the generated image not to be identified, L D Tends to be minimal, and adopts a maximumThe loss function design of the small game mode can enable the network model to have good infrared image enhancement capability.
2.2.3 related network parameter settings:
| type(s) | Convolution kernel | Convolution step length | Filling |
| conv | 4×4 | 2×2 | 1×1 |
| conv | 4×4 | 2×2 | 1×1 |
| conv | 4×4 | 2×2 | 1×1 |
| conv | 4×4 | 1×1 | 1×1 |
| conv | 4×4 | 1×1 | 1×1 |
The image quality data obtained after the infrared image enhancement module based on deep learning is used for enhancing 512 infrared test images are shown in the following table, 512 images are randomly tested, and after the infrared image processing is carried out by the method, the image evaluation indexes PSNT and SSIM are better represented, so that the method has a good infrared image enhancement effect.
| PSNR | SSIM | |
| 512 infrared test images | 26.67 | 0.974 |
3. Infrared image enhancement model
The infrared image enhancement model is a result obtained after the image dataset is trained by the infrared image enhancement module based on deep learning.
4. Lower die assembly module of die
The model underfilling module is used for underfilling the infrared image enhancement model which is output after the training of the infrared image enhancement module based on the deep learning into the parallel processing module through a remote service bus mode.
5. Infrared and visible light image acquisition device
The infrared and visible light image acquisition device is used for acquiring infrared images and corresponding visible light gray images in real time, the acquired infrared images are used for being sent into the parallel processing module to be subjected to image enhancement processing, and the infrared images and the visible light gray images are also sent to the image data set through the image transmission module and used for dynamically updating the image data set.
6. Parallel processing module
The input of the parallel processing module is an infrared image acquired by an infrared and visible light image acquisition device in real time, and the received infrared image is enhanced by using an infrared image enhancement model arranged in the parallel processing module; in order to improve the real-time processing speed of infrared image enhancement, the parallel processing module adopts a Cuda-based acceleration technology, fully utilizes the GPU computing capability, and simultaneously adopts a multithread parallel technology to improve the processing speed of image enhancement.
7. Image transmission module
The image transmission module is responsible for transmitting visible light gray level images and infrared images acquired by the infrared and visible light image acquisition device to an image data set, when the number of incremental image groups is greater than or equal to a dynamic update threshold value of the image data set, the infrared image enhancement module based on deep learning is triggered to automatically train, automatic update of an infrared image enhancement model installed in a parallel processing module is realized, continuous update of the infrared image enhancement model is realized in a subsequent image enhancement process, the infrared image enhancement effect is continuously optimized, and finally, the infrared image enhancement application of offline modeling-online application-sustainable update is realized.
Fig. 3 is a graph comparing the effects of the original image before and after enhancement with the enhanced image after enhancement, wherein the left side is the original infrared image, the right side is the enhanced infrared image, and the picture content of the right side image is richer, the texture is obvious, the detail is prominent, and the visual effect is better.
Claims (3)
1. An infrared image self-adaptation reinforcing device is characterized in that: the system comprises an image data set, an infrared image enhancement module based on deep learning, an infrared image enhancement model, a model downloading module, an infrared and visible light image acquisition device, an image transmission module and a parallel processing module;
the image dataset is used for providing image samples for model training for the infrared image enhancement module based on the deep learning; the updating mode of the image data set is as follows: when the number of groups of the collected infrared images and the visible light gray images is 10% greater than the number of samples in the current image data set, randomly replacing the corresponding number of samples in the image data set by the collected infrared images and the visible light gray images, marking the images collected by the infrared and visible light image collecting devices for replacement in the replacement process, and when the images are updated and replaced next time, not replacing the marked images; after the whole image data set is updated, deleting marks marked by the current sample image in the image data set, and then entering the next round of dynamic updating according to the same method;
the infrared image enhancement module based on the deep learning is used for the deep learning training to obtain the infrared image enhancement model;
the infrared image enhancement module based on deep learning comprises a generation sub-network G and an identification sub-network D;
the generation sub-network is used for generating an enhanced infrared image; the identification sub-network D is used for judging the authenticity of the visible light gray level image and the enhanced infrared image to obtain a probability value of 0-1, wherein 0 represents that the enhanced infrared image quality is very low and has no credibility, and 1 represents that the enhanced infrared image quality is very high and approaches to the real image;
the network structure of the generation sub-network is as follows:
CR-CBR-CBR-D-Tanh
wherein, C is a convolution layer, R is a ReLu activation function, B is a batch normalization layer, D is a deconvolution layer, and Tanh is:z is a feature image obtained after deconvolution; e is the bottom of natural logarithm;
the loss function of the generated subnetwork is:
L G ＝E[λ×L cout +log(D(G(X),X)]
wherein:
e is Euclidean distance;
L cont a quality loss function representing an enhanced infrared image and a visible gray scale image, defined by an L2 distance,
y is a visible light gray scale image corresponding to the input infrared image;
g (X) is an enhanced infrared image obtained by utilizing the generation sub-network; d is the output of the authentication sub-network;
the network structure of the authentication sub-network D is:
CL-CBL-CBL-CBL-C-Sigmoid；
wherein: sigmoid:c is a convolution layer, B is a batch normalization layer, and L is a leakage ReLU activation function;
the loss function of the authentication subnetwork D is:
wherein:
x is an infrared image, Y is a visible light gray scale image, and G (X) is an enhanced infrared image obtained by utilizing a generation sub-network;
the model lower loading module is used for downloading the infrared image enhancement model into the parallel processing module in a remote service bus mode;
the infrared and visible light image acquisition device is used for acquiring infrared images and corresponding visible light gray images in real time and transmitting the infrared images and the corresponding visible light gray images to the image transmission module and the parallel processing module;
the parallel processing module is used for carrying out parallel enhancement processing on a plurality of received infrared images, and simultaneously processing a plurality of images by utilizing a multithreading parallel technology, so that the processing speed of image enhancement is improved;
the image transmission module is used for transmitting the visible light gray level images and the infrared images acquired by the infrared and visible light image acquisition device to the image data set, and triggering the automatic training of the infrared image enhancement module when the number of the incremental image groups is greater than or equal to the dynamic update threshold value of the image data set, so as to realize the automatic update of the infrared image enhancement model which is downloaded in the parallel processing module.
2. The infrared image adaptive enhancement device of claim 1, wherein:
the image dataset is constructed according to the following method:
the visible light image is converted into a visible light gray image by gray level conversion, then the gray level image is converted into an infrared image by contrast conversion or histogram conversion, and each obtained infrared image and the corresponding visible light gray level image are taken as one sample of an image data set.
3. A method of adaptive enhancement of an infrared image based on the infrared image adaptive enhancement device according to any one of claims 1 or 2, comprising the steps of:
step 1: constructing an image dataset
Selecting an existing ImageNet data set as a basis for constructing an image data set, converting an image in the ImageNet data set into a visible light gray image by using gray level conversion, converting the gray level image into an infrared image by using contrast conversion or histogram conversion, and taking each obtained infrared image and a visible light gray level image corresponding to the infrared image as one sample of the image data set;
step 2: model training
Constructing an antagonistic neural network, inputting a sample in an image data set into the constructed antagonistic neural network to perform deep learning training on the antagonistic neural network to obtain an infrared image enhancement model;
the generation sub-network of the infrared image enhancement model consists of 3 convolution layers and 1 deconvolution layer, wherein a residual error processing module is used between the convolution layers, and the method specifically comprises the following steps: adding residual connection after the first convolution layer and after the second convolution layer, carrying out residual connection again on the feature vector obtained after weighting treatment and the third convolution layer, enabling the image feature vectors with different dimensions obtained by all convolution layers to flow in all layers in the generation sub-network, and reserving image detail features;
step 3: downloading the infrared image enhancement model obtained in the step 2 into a parallel processing module;
step 4: the infrared and visible light image acquisition device acquires infrared images and visible light images of a target scene in real time, when the number of groups of the acquired infrared images and visible light gray images is 10% greater than the number of samples in the current image data set, randomly replacing the corresponding number of samples in the image data set by the acquired infrared images and visible light gray images, marking the images acquired by the infrared and visible light image acquisition device for replacement in the replacement process, and when the images marked for the next updating and replacement are replaced, not replacing the marked images, and then returning to the step 2; after the whole image data set is updated, deleting marks marked by the current sample image in the image data set, and then entering the next round of dynamic updating according to the same method; simultaneously, copying one part of infrared image acquired in real time and sending the copied part of infrared image to a parallel processing module;
step 5: and simultaneously carrying out parallel enhancement processing on a plurality of infrared images by utilizing an infrared image enhancement model arranged in a parallel processing module, and outputting corresponding enhanced images.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110183673.5A CN112991200B (en) | 2021-02-08 | 2021-02-08 | Method and device for adaptively enhancing infrared image |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110183673.5A CN112991200B (en) | 2021-02-08 | 2021-02-08 | Method and device for adaptively enhancing infrared image |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112991200A CN112991200A (en) | 2021-06-18 |
| CN112991200B true CN112991200B (en) | 2024-02-02 |
Family
ID=76393138
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110183673.5A Active CN112991200B (en) | 2021-02-08 | 2021-02-08 | Method and device for adaptively enhancing infrared image |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112991200B (en) |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108154523A (en) * | 2017-12-25 | 2018-06-12 | 北京航空航天大学 | A kind of real-time modeling method system and method in airborne photoelectric platform |
| CN109544487A (en) * | 2018-09-30 | 2019-03-29 | 西安电子科技大学 | A kind of infrared image enhancing method based on convolutional neural networks |
| CN110084757A (en) * | 2019-04-15 | 2019-08-02 | 南京信息工程大学 | A kind of infrared depth image enhancement method based on generation confrontation network |
| CN110889813A (en) * | 2019-11-15 | 2020-03-17 | 安徽大学 | Low-light image enhancement method based on infrared information |
| CN111160307A (en) * | 2019-12-31 | 2020-05-15 | 帷幄匠心科技（杭州）有限公司 | Face recognition method and face recognition card punching system |
| CN111612725A (en) * | 2020-04-29 | 2020-09-01 | 中国科学院西安光学精密机械研究所 | Image fusion method based on contrast enhancement of visible light image |
| CN112116573A (en) * | 2020-09-16 | 2020-12-22 | 四川嘉能佳网创新能源科技有限责任公司 | High-precision infrared image anomaly detection method and system |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10475174B2 (en) * | 2017-04-06 | 2019-11-12 | General Electric Company | Visual anomaly detection system |
| CN108446617B (en) * | 2018-03-09 | 2022-04-22 | 华南理工大学 | Side face interference resistant rapid human face detection method |
-
2021
- 2021-02-08 CN CN202110183673.5A patent/CN112991200B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108154523A (en) * | 2017-12-25 | 2018-06-12 | 北京航空航天大学 | A kind of real-time modeling method system and method in airborne photoelectric platform |
| CN109544487A (en) * | 2018-09-30 | 2019-03-29 | 西安电子科技大学 | A kind of infrared image enhancing method based on convolutional neural networks |
| CN110084757A (en) * | 2019-04-15 | 2019-08-02 | 南京信息工程大学 | A kind of infrared depth image enhancement method based on generation confrontation network |
| CN110889813A (en) * | 2019-11-15 | 2020-03-17 | 安徽大学 | Low-light image enhancement method based on infrared information |
| CN111160307A (en) * | 2019-12-31 | 2020-05-15 | 帷幄匠心科技（杭州）有限公司 | Face recognition method and face recognition card punching system |
| CN111612725A (en) * | 2020-04-29 | 2020-09-01 | 中国科学院西安光学精密机械研究所 | Image fusion method based on contrast enhancement of visible light image |
| CN112116573A (en) * | 2020-09-16 | 2020-12-22 | 四川嘉能佳网创新能源科技有限责任公司 | High-precision infrared image anomaly detection method and system |
Non-Patent Citations (2)
| Title |
|---|
| Single infrared image enhancement using a deep convolutional neural network;Xiaodong Kuang, et.al;《Neurocomputing》;20190307;摘要，第1-10页 * |
| 基于生成对抗网络的红外图像数据增强;陈佛计等;《计算机应用》;20200331(第07期);第228-232页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112991200A (en) | 2021-06-18 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN113178255B (en) | GAN-based medical diagnosis model attack resistance method | |
| CN113592736B (en) | Semi-supervised image deblurring method based on fused attention mechanism | |
| CN111292264A (en) | Image high dynamic range reconstruction method based on deep learning | |
| CN104809443A (en) | Convolutional neural network-based license plate detection method and system | |
| CN111539343B (en) | Black smoke vehicle detection method based on convolution attention network | |
| CN111242868B (en) | Image enhancement method based on convolutional neural network in scotopic vision environment | |
| CN111696021B (en) | Image self-adaptive steganalysis system and method based on significance detection | |
| CN109685772B (en) | No-reference stereo image quality evaluation method based on registration distortion representation | |
| CN116342894B (en) | GIS infrared feature recognition system and method based on improved YOLOv5 | |
| CN110751591A (en) | Self-adaptive color image steganography method based on fuzzy inference system | |
| CN113689382A (en) | Tumor postoperative life prediction method and system based on medical images and pathological images | |
| Zhou et al. | Deep multi-scale features learning for distorted image quality assessment | |
| CN116798070A (en) | Cross-mode pedestrian re-recognition method based on spectrum sensing and attention mechanism | |
| CN106355210A (en) | Method for expressing infrared image features of insulators on basis of depth neuron response modes | |
| CN113537110A (en) | False video detection method fusing intra-frame and inter-frame differences | |
| CN112149526A (en) | Lane line detection method and system based on long-distance information fusion | |
| CN112102186A (en) | Real-time enhancement method for underwater video image | |
| CN112330562B (en) | Heterogeneous remote sensing image transformation method and system | |
| CN110503049B (en) | Satellite video vehicle number estimation method based on generation countermeasure network | |
| CN112991200B (en) | Method and device for adaptively enhancing infrared image | |
| CN112528788A (en) | Re-recognition method based on domain invariant features and space-time features | |
| CN116612355A (en) | Training method and device for face fake recognition model, face recognition method and device | |
| CN115170985B (en) | Remote sensing image semantic segmentation network and segmentation method based on threshold attention | |
| CN115630361A (en) | Attention distillation-based federal learning backdoor defense method | |
| CN112991257B (en) | Heterogeneous remote sensing image change rapid detection method based on semi-supervised twin network |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |