CN114170094A

CN114170094A - Airborne infrared image super-resolution and noise removal algorithm based on twin network

Info

Publication number: CN114170094A
Application number: CN202111361000.0A
Authority: CN
Inventors: 李伟; 陶然; 杨小冬; 黄展超; 马鹏阁; 揭斐然
Original assignee: Beijing Institute of Technology BIT; Luoyang Institute of Electro Optical Equipment AVIC; Zhengzhou University of Aeronautics
Current assignee: Beijing Institute of Technology BIT; Luoyang Institute of Electro Optical Equipment AVIC; Zhengzhou University of Aeronautics
Priority date: 2021-11-17
Filing date: 2021-11-17
Publication date: 2022-03-11
Anticipated expiration: 2041-11-17
Also published as: CN114170094B

Abstract

The invention provides an airborne infrared image super-resolution and noise removal algorithm based on a twin network, which comprises the following steps: step 1: constructing a training set and a testing set; step 2: constructing a basic twin network framework; and step 3: and (5) training and testing. The invention utilizes the characteristic that two branches of the twin network share weight, can achieve the purpose of removing noise and simultaneously improve the resolution of the image. And local jump connection and global jump connection are used simultaneously, so that the network convergence is accelerated while low-frequency information is fully used. In addition, a spatial attention mechanism is added, spatial domain weight is adjusted in a self-adaptive mode, the contrast ratio of the target and the background is highlighted, and the target tracking, the target detection and the like are better served.

Description

Airborne infrared image super-resolution and noise removal algorithm based on twin network

Technical Field

The invention belongs to the technical field of image processing, relates to an infrared image super-resolution and noise removal algorithm based on a twin network, and is particularly suitable for improving the quality of a low-resolution infrared image containing noise.

Background

The aerial remote sensing utilizes an aerial vehicle as a sensor carrier, is a multifunctional comprehensive detection technology for acquiring ground information in the air, and is widely applied to various scenes such as traffic early warning, road assistance, terrain mapping, military reconnaissance and the like. The improvement of the aerial image quality is a high and new technology integrating a plurality of fields of optical imaging, machine vision, artificial intelligence, image processing and the like, and is a basic and key technology for realizing an intelligent aerial monitoring system and intelligent weapon reconnaissance and aiming. The aerial camera has the advantages of rapid deployment and wide movement range by depending on an aircraft, and can carry out all-around, multi-angle and active image acquisition aiming at a ground target. Therefore, aerial images have been widely used both in military and civilian applications. With the popularization of various digital instruments and digital products, images and videos become the most common information carriers in human life, and the images and videos contain a large amount of object information and become a main way for people to acquire external rock information. However, the image quality is often degraded by various noises during the acquisition, transmission and storage of the image. Meanwhile, during the imaging process, the image may be distorted by the optical lens. The influence of a series of interference factors such as atmospheric blur, sensor blur, optical blur and motion blur can only obtain images with relatively low resolution. Various factors degrade the quality of the obtained image, and the quality of the image preprocessing algorithm is related to the effect of subsequent image processing, such as image segmentation, target detection, edge extraction, and the like. Therefore, in order to better serve the subsequent work, it is necessary to improve the quality of the image and obtain a high-quality digital image.

The final purpose of image quality improvement is to improve a given image and solve the problem that the image quality is reduced due to low resolution and noise interference of an actual image. The super-resolution and denoising technology can effectively improve the image quality, increase the signal-to-noise ratio and better embody the information carried by the original image, and as an important preprocessing means, people respectively and widely research the image super-resolution algorithm and the image denoising algorithm. In the existing image super-resolution method, an interpolation-based method usually estimates an unknown pixel value on a high-resolution image grid by using a known pixel value on a low-resolution image grid, and the common interpolation method includes: polynomial interpolation and edge-driven based interpolation. Although the methods have small computational complexity, the reconstructed high-resolution images are easy to generate distortion phenomena such as edge smoothing, blurring and aliasing. In order to break through the defects of the traditional image interpolation, a reconstruction-based method is combined with an image degradation process, and a super-resolution model is established by utilizing image prior to estimate a high-resolution image. The learning-based method is to learn the mapping relation between high-resolution images and low-resolution images on sample data by using a machine, and use the learned mapping relation for the reconstruction of the high-resolution images, so that high-frequency information of the images can be well restored. Common denoising algorithms are mainly divided into spatial domain filtering, transform domain filtering, morphological noise filter and the like. Spatial domain filtering is to directly perform data operation on an original image and process the gray value of a pixel. Common space domain image denoising algorithms include a neighborhood averaging method, median filtering, low-pass filtering and the like. The image transformation domain denoising method is to convert an image from a space domain to a transformation domain, process a transformation coefficient in the transformation domain, and perform inverse transformation to convert the image from the transformation domain to the space domain, so as to achieve the purpose of removing image noise. The morphological noise filter first performs an opening operation on the noisy image, and the optional structural element matrix is larger than the noise size, so that the opening operation results in removing the background noise. And performing closed operation on the image obtained in the previous step to remove noise in the image.

Summarizing the technology, it can be seen that a single image super-resolution and noise removal algorithm has achieved a certain effect, but the realization of image super-resolution and noise removal at the same time still needs to be improved greatly, so as to better complete image quality improvement. In addition, the combined research in the fields of deep learning and aerial remote sensing can promote the development of an aerial target detection system towards the intelligent direction, realize accurate detection, identification and analysis, effectively serve for detecting and identifying target information, improve the capability of practical application in military reconnaissance, daily monitoring, safe environment early warning and the like, and have important significance for promoting the development of the aerial science and technology.

Disclosure of Invention

The invention provides an infrared image super-resolution and noise removal algorithm based on a twin network, aiming at the singleness and the deficiency of the existing image quality improvement technology. The algorithm can improve the resolution of the image of the airborne embedded platform, can effectively remove noise in the image, and further improves the image quality, so that the algorithm can better serve the work such as target detection, image segmentation, edge extraction and the like.

The technical scheme of the invention is as follows:

the airborne infrared image super-resolution and noise removal algorithm based on the twin network comprises the following steps:

step 1: constructing training set and testing set

1200 high-resolution infrared images were selected as high-resolution (HR) tag images. The image is down-sampled and denoised respectively to construct a Low Resolution (LR) image and noiseAnd (4) an image. And upsampling the low-resolution image to restore the low-resolution image to the original size. Taking a noisy image as input I of a first channel₁Using the low resolution image as input I of the second channel₂In this way, a training data set is constructed. Selecting 60 images, down-sampling the images, adding noise, up-sampling to restore the original size, and constructing a test image (I)_T) A data set.

Step 2: building a basic twin network framework

The number of network convolution layers and the size of convolution kernels are designed for a target data set and an experimental purpose. The local jump connection is used to accelerate network convergence, and the global jump connection is used to make full use of low-frequency information. And a space attention mechanism is added to highlight the contrast of the target and the background, so that the subsequent items such as target detection, target tracking and the like are better served.

And step 3: training and testing

Low resolution image I₁And a noisy image I₂Inputting the images into a twin network for training, circularly iterating, distributing different loss function weights according to the requirements of the target images, optimizing the network, and storing a training model. Then calling the training model to test the image I_TAnd testing to obtain a high-quality airborne infrared image.

Specifically, step 1 comprises the following substeps:

step 1.1: 1200 high-resolution infrared images with the size of 1024 x 1024 were selected. The image is partitioned into 256 × 256 image blocks, and 19200 high-resolution image blocks are obtained as label images (HR).

Step 1.2: the HR is interpolated 2 times bicubically to obtain low resolution image blocks of size 128 x 128, which are then interpolated back to size 256 x 256 as the low resolution input image I1.

Step 1.3: the HR image is subjected to a noise addition process, resulting in a noise image of 256 × 256 in size as a noise input image I2.

Step 1.4: and selecting 60 high-resolution infrared images with the size of 1024 × 1024, and performing 2-time bicubic interpolation on the images to obtain low-resolution images with the size of 512 × 512. The image is then interpolated back to a size of 1024 x 1024. And finally, carrying out noise processing on the image to obtain a test image IT.

Step 2 comprises the following substeps:

step 2.1: and a twin network framework is built for the target image, so that the network can remove image noise and improve the image resolution.

Step 2.2: the number of convolutional layers is designed. Two layers of the first convolution layer and the activation function, three spatial attention blocks and one layer of the third convolution layer are used for forming a network framework.

Step 2.3: and designing the size of a convolution kernel. For the first convolution layer, the convolution kernel size is 64 x 1 x 9. For the second convolution layer, the convolution kernel size was 64 × 3. For the third convolution layer, the convolution kernel size is 1 × 3. Wherein the activation functions are all RELU functions.

Step 2.4: a space attention mechanism is added for the target image, the contrast between the target and the background is highlighted, and the subsequent work of target detection, target tracking and the like is better served.

Step 3 comprises the following substeps:

step 3.1: the low resolution image I1 and the noise image I2 are input into the network as inputs to two branches of the twin network, respectively.

Step 3.2: and respectively calculating loss functions of two branches of the network, then weighting to obtain the loss function of the whole network, circularly iterating and optimizing the network. Saving training parameters and weights, and saving training models

Step 3.3: and calling the training model to test the test image IT to obtain the high-quality airborne infrared image.

The deep learning convolutional neural network in the invention uses a twin network based on a spatial attention mechanism. The method has a simple structure, and by utilizing the characteristic that two branches of the twin network share weight, the purpose of removing noise can be achieved, the resolution of the image is improved, and the quality of the airborne infrared image is greatly improved. And local jump connection and global jump connection are used simultaneously, so that the network convergence is accelerated while low-frequency information is fully used. In addition, a spatial attention mechanism is added, spatial domain weight is adjusted in a self-adaptive mode, the contrast ratio of the target and the background is highlighted, and the target tracking, the target detection and the like are better served. Meanwhile, aerial image target detection and identification tasks of the airborne platform are limited by conditions of the airborne platform, and large-scale graphic computing equipment cannot be deployed, so that the deployment of various deep neural networks on a mobile embedded platform is a main trend of the development of future artificial intelligence technology in the aviation industry. Whether mobile embedded devices or edge computing devices, which have very limited computing and memory resources, it is impractical to deploy a high-memory, high-power-consumption deep network model directly on such devices. Therefore, in order to meet the task requirement of deploying a target detection and identification algorithm on an embedded platform, on the basis of a high-performance deep neural network algorithm, the quality of an airborne infrared image is improved to obtain a high-quality airborne infrared image, the research value is huge, and the method is a key point for breaking through artificial intelligence technology represented by deep learning to really land on the ground, closely combining with practical application and promoting the intelligent development of the aviation industry technology.

Drawings

The invention is further illustrated with reference to the following figures and examples.

FIG. 1 is a flow chart of step 1 data set construction;

FIG. 2 is a flow chart of step 2 twin network construction;

FIG. 3 is a flow chart of step 3 training and testing;

FIG. 4 is a diagram of a twin network structure based on a spatial attention mechanism according to an embodiment;

FIG. 5 is a diagram of an embodiment spatial attention residual block;

FIG. 6 is an example of an embodiment low resolution input image;

FIG. 7 is an example of a noisy input image according to an embodiment;

FIG. 8 is an example of an embodiment test image;

FIG. 9 shows the results of the experiment in the examples.

Detailed Description

The present invention is further described with reference to the following embodiments and the accompanying drawings, which are not intended to limit the scope of the invention as claimed.

step 1: a training set and a test set are constructed, and a flow chart is shown in FIG. 1.

1260 high resolution airborne infrared images, 1024 x 1024 in size, were selected to construct a data set, as shown in fig. 1. 1200 of the high-resolution (HR) airborne infrared images are selected to construct a training data set. The HR image is first interpolated 2 times bicubically into a low resolution image of size 512 x 512 and then interpolated back to a size 1024 x 1024 as the low resolution input image I1 in the twin network, as shown in fig. 6. The HR image is subjected to the addition of the stripe noise conforming to the gaussian distribution as the noise input image I2 of the twin network, as shown in fig. 7.

And selecting the rest 60 high-resolution airborne infrared images, performing 2-time bicubic interpolation on the high-resolution airborne infrared images, interpolating the high-resolution airborne infrared images into low-resolution images with the size of 512 x 512, and then interpolating the low-resolution images back to 1024 x 1024. Finally, strip noise conforming to Gaussian distribution is added to construct a test data set.

Step 2: and constructing a basic twin network framework, wherein the flow chart is shown in figure 2.

As in fig. 4, the entire network consists of three parts. Wherein the first part comprises two convolutional layers, mainly completing the coarse removal of the stripe noise. Where the convolution kernels are all 64 x 1 x 9 in size. And (3) sequentially passing the input image through two first convolution layers and an activation function, and performing 0 complementing operation to obtain 64 × 1024 airborne infrared images.

F_D1＝Conv₁(I_in)

F_{D_R1}＝Relu(F_D1)

Wherein, I_inTo input an image, Conv₁(. for a first convolution operation, F_D1To pass through the feature map of the first convolution layer, Relu (-) operates as an activation function, F_{D_R1}Is a characteristic diagram after passing through the activation function.

F_D2＝Conv₁(F_{D_R1})

F_{D_R2}＝Relu(F_D2)

Wherein, F_D2Is the output characteristic diagram of the second first convolution layer, F_{D_R2}Is the output of the first part of the network.

Referring to fig. 5, the second part includes three spatial attention residual blocks, which mainly complete further noise removal and image resolution improvement. Each spatial attention residual block comprises two second convolution layers and a spatial attention block, the convolution kernel size of each second convolution layer is 64 x 3, and a 0 complementing operation is also used to obtain 64 x 1024 infrared airborne images. And finally, connecting the input with the output of the spatial attention block by using local jump connection, wherein the local jump connection is used as the output of the whole spatial attention residual block, and the contrast of the target and the background is enhanced while the noise removal and the image resolution are improved.

F_{S1_1}＝Conv₂(F_{D_R2})

F_{S_R1}＝Relu(F_{S1_1})

Wherein, Conv₂(. is a second convolution operation, F_{S1_1}And F_{S_R1}The feature map after the first convolution operation in the first spatial attention residual block and the feature map after the activation function are respectively.

F_{S1_2}＝Conv₂(F_{S_R1})

F_{Att_1}＝Att(F_{S1_2})

Wherein, F_{S1_2}For the feature map after the second convolution operation in the first block of spatial attention residues, F_{Att_1}Is a feature map after passing spatial attention in the first spatial attention residual block.

F_{out_1}＝F_{Att_1}+F_{D_R2}

Wherein, F_{out_1}Is the output of the first spatial attention residual block.

And then sequentially passes through two spatial attention residual blocks.

F_{S2_1}＝Conv₂(F_{out_1})

F_{S_R2}＝Relu(F_{S2_1})

Wherein, F_{S2_1}And F_{S_R2}The feature map after the first convolution operation in the second spatial attention residual block and the feature map after the activation function are respectively.

F_{S2_2}＝Conv₂(F_{S_R2})

F_{Att_2}＝Att(F_{S2_2})

Wherein, F_{S2_2}For the feature map after the second convolution operation in the second block of spatial attention residues, F_{Att_2}Is a feature map after spatial attention in the second block of spatial attention residues.

F_{out_2}＝F_{Att_2}+F_{out_1}

Wherein, F_{out_2}Is the output of the second spatial attention residual block.

F_{S3_1}＝Conv₂(F_{out_2})

F_{S_R3}＝Relu(F_{S3_1})

Wherein, F_{S3_1}And F_{S_R3}The feature map after the first convolution operation in the third spatial attention residual block and the feature map after the activation function are respectively.

F_{S3_2}＝Conv₂(F_{S_R3})

F_{Att_3}＝Att(F_{S3_2})

Wherein, F_{S3_2}Is a feature map of the third block of spatial attention residual after the second convolution operation, F_{Att_3}Is a feature map after passing spatial attention in the third spatial attention residual block.

F_{out_3}＝F_{Att_3}+F_{out_2}

Wherein, F_{out_3}The output of the third spatial residual of attention block.

The third part consists of a third convolution layer, where the convolution kernel size is 1 x 1, resulting in an airborne infrared image of 1 x 1024.

F₃＝Conv₃(F_{out_3}+F_{D_R2})

Wherein, Conv₃(. is a third convolution operation, F₃Is an output characteristic diagram.

And step 3: the flow chart of training and testing is shown in fig. 3.

And respectively inputting the constructed input images I1 and I2 as branches of a twin network, setting cycle times, calculating a loss function, and performing iterative optimization to train parameters and weights of the model.

loss＝αloss₁+βloss₂

Wherein, loss is the total output loss function, loss₁And loss₂Which are the loss functions of the two branches of the network, respectively. loss₁For low resolution input of the branch loss function, loss₂The branch loss function is input for noise. Alpha and beta are the weights of the two branches, respectively.

Wherein n is the total number of the training data set, and the value of this embodiment is 19200.

For the ith label image, the label is printed,

for the ith low resolution branch output image,

an image is output for the ith noise branch.

Parameters and weights of the model are saved, and the training model is saved.

And calling a training model, directly distributing parameters and weights to the test image IT shown in the figure 8, and completing the test of the airborne infrared image to obtain a high-quality airborne infrared image, as shown in the figure 9.

The above examples are merely preferred embodiments of the present invention, and the scope of the present invention is not limited to the above examples. All technical schemes belonging to the idea of the invention belong to the protection scope of the invention. It should be noted that modifications and embellishments within the scope of the invention may be made by those skilled in the art without departing from the principle of the invention, and such modifications and embellishments should also be considered as within the scope of the invention.

Claims

1. The airborne infrared image super-resolution and noise removal algorithm based on the twin network is characterized by comprising the following steps of:

step 1: constructing training set and testing set

Selecting 1200 high-resolution infrared images as high-resolution HR label images; respectively carrying out down-sampling and noise adding processing on the image so as to construct a low-resolution LR image and a noise image; up-sampling the low-resolution image to restore the low-resolution image to the original size; taking a noisy image as input I of a first channel₁Using the low resolution image as input I of the second channel₂Constructing a training data set; selecting 60 images, down-sampling the images, adding noise, up-sampling to restore the original size, and constructing a test image I_TA data set;

step 2: building a basic twin network framework

Designing the number of network convolution layers and the size of convolution kernels according to a target data set and an experimental purpose; the local jump connection is used for accelerating the network convergence, and the global jump connection is used for fully utilizing the low-frequency information; adding a space attention mechanism to highlight the contrast ratio of the target and the background;

and step 3: training and testing

Low resolution image I₁And a noisy image I₂Inputting into twin network for training, performing iteration circularly, and distributing according to the requirement of target imageOptimizing the network by the same loss function weight, and storing a training model; then calling the training model to test the image I_TAnd testing to obtain a high-quality airborne infrared image.

2. The airborne infrared image super-resolution and noise removal algorithm based on the twin network according to claim 1, wherein the step 1 comprises the following sub-steps:

step 1.1: selecting 1200 high-resolution infrared images with the size of 1024 × 1024; partitioning the image into 256 × 256 image blocks to obtain 19200 high-resolution image blocks as label images HR;

step 1.2: carrying out 2-time bicubic interpolation on the HR to obtain a low-resolution image block with the size of 128 x 128, and then interpolating the image block back to the size of 256 x 256 to obtain a low-resolution input image I1;

step 1.3: performing noise processing on the HR image to obtain a noise image with the size of 256 × 256, and using the noise image as a noise input image I2;

step 1.4: selecting 60 high-resolution infrared images with the size of 1024 × 1024, and performing 2-time bicubic interpolation on the images to obtain low-resolution images with the size of 512 × 512; then the image is interpolated back to a size of 1024 x 1024; and finally, carrying out noise processing on the image to obtain a test image IT.

3. The twin network based airborne infrared image super-resolution and noise removal algorithm according to claim 1, wherein step 2 comprises the following sub-steps:

step 2.1: a twin network framework is built for a target image, so that the network can remove image noise and improve the image resolution;

step 2.2: designing the number of the convolution layers; forming a network framework by using two layers of first convolution layers and activation functions, three spatial attention blocks and one layer of third convolution layer;

step 2.3: designing the size of a convolution kernel; for the first convolution layer, its convolution kernel size is 64 x 1 x 9; for the second convolution layer, its convolution kernel size is 64 x 3; for the third convolution layer, its convolution kernel size is 1 x 3; wherein the activation functions are RELU functions;

step 2.4: and adding a space attention mechanism aiming at the target image to highlight the contrast ratio of the target and the background.

4. The airborne infrared image super-resolution and noise removal algorithm based on the twin network according to claim 1, wherein the step 3 comprises the following sub-steps:

step 3.1: inputting the low-resolution image I1 and the noise image I2 into the network as the input of two branches of the twin network respectively;

step 3.2: respectively calculating loss functions of two branches of the network, then weighting to obtain the loss function of the whole network, circularly iterating and optimizing the network; saving the training parameters and the weights, and saving the training model;