CN111951177A - Infrared image detail enhancement method based on image super-resolution loss function - Google Patents
Infrared image detail enhancement method based on image super-resolution loss function Download PDFInfo
- Publication number
- CN111951177A CN111951177A CN202010645997.1A CN202010645997A CN111951177A CN 111951177 A CN111951177 A CN 111951177A CN 202010645997 A CN202010645997 A CN 202010645997A CN 111951177 A CN111951177 A CN 111951177A
- Authority
- CN
- China
- Prior art keywords
- resolution
- infrared
- image
- super
- layer
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 26
- 238000012549 training Methods 0.000 claims description 23
- 238000012545 processing Methods 0.000 claims description 17
- 238000013527 convolutional neural network Methods 0.000 claims description 11
- 238000010586 diagram Methods 0.000 claims description 7
- 238000012360 testing method Methods 0.000 claims description 7
- 238000000605 extraction Methods 0.000 claims description 6
- 239000011159 matrix material Substances 0.000 claims description 6
- 125000004122 cyclic group Chemical group 0.000 claims description 3
- 230000004927 fusion Effects 0.000 claims description 3
- 239000000126 substance Substances 0.000 claims description 2
- 230000006870 function Effects 0.000 abstract description 26
- 238000013135 deep learning Methods 0.000 abstract description 12
- 230000000694 effects Effects 0.000 abstract description 2
- 238000005516 engineering process Methods 0.000 description 4
- 238000013528 artificial neural network Methods 0.000 description 2
- 238000011084 recovery Methods 0.000 description 2
- 230000004913 activation Effects 0.000 description 1
- 238000013473 artificial intelligence Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000003745 diagnosis Methods 0.000 description 1
- 230000002708 enhancing effect Effects 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 238000002076 thermal analysis method Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/70—Denoising; Smoothing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T3/4053—Scaling of whole images or parts thereof, e.g. expanding or contracting based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10048—Infrared image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20172—Image enhancement details
- G06T2207/20182—Noise reduction or smoothing in the temporal domain; Spatio-temporal filtering
Landscapes
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Image Processing (AREA)
Abstract
The invention discloses an infrared image detail enhancement method based on an image super-resolution loss function. The method comprises the steps of extracting the features of an image by adopting an infrared image super-resolution network based on deep learning, using the extracted features to describe main information of the infrared image, constructing a loss function to train the infrared image super-resolution network based on the deep learning, and then extracting the image features by utilizing the trained infrared image super-resolution network to obtain the infrared super-resolution image. The invention can reduce the difference between the super-resolution image and the real infrared image high-level characteristics, thereby improving the super-resolution effect of the image.
Description
Technical Field
The invention belongs to an image enhancement method in the field of deep learning for extracting image features by a deep neural network, and particularly relates to an infrared image detail enhanced image super-resolution loss function fused with a neural network, an image super-resolution function and the like.
Background
With the development of deep learning technology, more and more deep learning algorithms are applied to various fields, and two image super-resolution technologies based on deep learning become research hotspots in the field of artificial intelligence at present.
Image super-resolution reconstruction aims at restoring a corresponding higher resolution image from one or more low resolution images. The image super-resolution technology has wide application, such as thermal analysis, video monitoring, medical diagnosis, remote sensing and the like, and due to the limitation of infrared image hardware, it is difficult to obtain high-quality and high-resolution infrared images, so that the application of the infrared images is limited. Therefore, the method for improving the resolution and the image quality of the image through the image super-resolution algorithm is a very concise and efficient way. Then, the existing infrared image super-resolution technology has some difficulties, and the problem of poor image recovery quality in the infrared image super-resolution network training process based on deep learning is caused because the signal-to-noise ratio of the infrared image is low and very much noise is easy to appear in the shooting process.
The key point for solving the problems is to design an image super-resolution loss function for enhancing the infrared image details, and the details of the infrared image can be better optimized in the infrared image super-resolution network training process based on deep learning, so that the image quality generated by the image super-resolution is improved.
Disclosure of Invention
The invention overcomes the defects of the prior art, provides an image super-resolution loss function with enhanced infrared image details, aims to train an image super-resolution model by using the image super-resolution loss function with enhanced infrared image details, and improves the detail recovery of super-resolution generated images of summarized images in real scenes.
In order to achieve the purpose, the invention adopts the following technical scheme:
(1) preparing an infrared image super-resolution training data set and an infrared image super-resolution testing data set;
(2) constructing an infrared image super-resolution network structure based on a convolutional neural network,
(3) inputting an infrared image super-resolution training data set into the infrared image super-resolution network based on the convolutional neural network constructed in the step (2) for training, constructing a loss function in the training process, continuously performing cyclic iterative training on the infrared image super-resolution network to continuously reduce the loss function until the set iteration times Q are finished, and storing the infrared image super-resolution network;
(4) and (4) carrying out image feature extraction on the infrared super-resolution test data set by using the infrared image super-resolution network obtained in the step (3) to obtain a final infrared super-resolution image.
The loss function in the training process is obtained by processing in the following way:
(1) convolving the real infrared picture by a Sobel operator to obtain a convolved picture:
Is＝convs(Ihr) (1)
wherein, IsFor the convolved picture, convsFor the convolution operation with the Sobel operator, IhrThe real infrared picture is obtained;
the real infrared picture refers to an image shot by an infrared camera, and the infrared super-resolution image refers to an image of integral multiple of the resolution of the original image. The two images are constructed together as a training data set. The infrared super-resolution test data set only consists of infrared pictures to be tested.
(2) Respectively inputting an infrared super-resolution image, a real infrared picture and a convolved picture which are generated in each training process of the infrared image super-resolution network and are used as prediction results into a pre-trained VGG16 network for feature extraction, wherein the pre-trained VGG16 network is obtained by training the VGG16 network through an existing known data set:
the VGG16 network comprises a plurality of characteristic layers which are sequentially composed of a convolutional layer and a relu layer, wherein one convolutional layer and one relu layer connected to the rear of the convolutional layer form one characteristic layer; for an input image, when the current k-th relu layer of the VGG16 network is equal to 1,2,3,4, the relu layer is a linear activation function, k represents the sequence number of the relu layer, and the image features processed by the k-th relu layer are obtained by the following processing:
Fk＝relu(conv(Fk-1)) (2)
wherein, Fk-1The feature map obtained by processing the k-1 relu layer of the previous layer, conv is the convolution operation with the convolution layer, FkProcessing the obtained characteristic diagram for the kth relu layer;
(3) calculating the characteristic loss of each layer of characteristic layer according to the infrared super-resolution image generated by the infrared image super-resolution network and the convolved image by the following formula:
wherein, CkHkWkThe number of channels in three dimensions of the input image,
and
respectively processing the convolved picture and the infrared super-resolution image generated by the infrared image super-resolution network on a kth relu layer to obtain characteristic graphs;
(4) calculating an infrared super-resolution image and a real infrared picture generated by an infrared image super-resolution network, and calculating a Gram matrix under each layer of characteristic layer by the following formula:
wherein the content of the first and second substances,
extracting a characteristic diagram of the real infrared picture in the kth relu layer, wherein h and w respectively represent ordinal numbers of the width and height of the image;
and then calculating style loss according to the Gram matrix of each layer of feature layer:
wherein F is a Frobenius norm;
representing the square of the frobenius norm,
and respectively representing the gray matrixes calculated by the characteristics of the real infrared picture and the infrared super-resolution image under the kth relu layer.
(5) And finally, performing weighted fusion on the calculated characteristic loss of the third layer of characteristic layer and the style loss of the first four layers of characteristic layers to obtain a final loss function:
wherein, w1And w2Weighted values for style loss and feature loss, respectively.
The infrared image super-resolution network based on the convolutional neural network comprises N convolutional layers connected in series and a loss function layer.
The method adopts the infrared image super-resolution network based on deep learning to extract the characteristics of the image, uses the extracted characteristics to describe the main information of the infrared image, and then adopts a specially designed loss function to train the infrared image super-resolution network based on deep learning.
Compared with the prior art, the invention has the beneficial effects that:
the method adopts an infrared image super-resolution network based on deep learning to extract the characteristics of the image, the extracted characteristics are used for describing the main information of the image, and the infrared image super-resolution network based on deep learning is trained through a specially designed image super-resolution loss function.
The loss functions of the conventional infrared image super-resolution network can only ensure that the image is consistent with a real image as much as possible, but due to the fact that the infrared image has low signal-to-noise ratio and high noise, the loss functions cannot ensure that the network generates a high-quality super-resolution image. The method can optimize the detail generation of the infrared image in the super-resolution process, overcomes the characteristic of low image quality of the infrared image, and improves the super-resolution effect of the image and the quality of the super-resolution generated image by reducing the difference between the super-resolution image and the high-level characteristics of the real infrared image through the loss function.
The method is suitable for image super-resolution processing, can overcome the problem that other super-resolution loss functions cannot sufficiently recover image details generated by image super-resolution trained on the basis of a convolutional neural network, and improves the quality of generated images.
Drawings
Fig. 1 is a schematic diagram of the structure of a VDSR model.
FIG. 2 is a schematic diagram of the logical structure of the method of the present invention.
FIG. 3 is a graph showing the results of 2-fold super resolution.
Detailed Description
The invention is further described below with reference to the accompanying drawings:
the method of the embodiment of the invention comprises the following steps:
step 1: preparing an infrared image super-resolution training data set and a test data set;
step 2: an infrared image super-resolution network structure based on a convolutional neural network is constructed, as shown in fig. 2, the infrared image super-resolution network structure based on the convolutional neural network is compared by adopting a VDSR model, and the infrared image super-resolution network based on the convolutional neural network comprises convolution layers and loss function layers, and specifically comprises N convolution layers connected in series and a loss function layer;
and step 3: inputting an infrared image super-resolution training data set into the infrared image super-resolution network based on the convolutional neural network constructed in the step (2) for training, wherein the loss function in the training process adopts the loss function of the invention, the loss function is continuously reduced by continuously carrying out cyclic iterative training on the network until the set iteration times Q are finished, and the infrared image super-resolution network is stored.
And 4, step 4: and (4) carrying out image feature extraction on the infrared super-resolution test data set by using the infrared image super-resolution model obtained in the step (3) to obtain a final infrared super-resolution image, wherein the example result is shown in fig. 3.
As shown in fig. 1, the processing steps are as follows:
(1) convolving the real infrared picture by a Sobel operator to obtain a convolved picture:
Is＝convs(Ihr) (1)
wherein, IsFor the convolved picture, convsFor the convolution operation with the Sobel operator, IhrThe real infrared picture is obtained;
(2) respectively inputting the infrared super-resolution image, the real infrared picture and the convolved picture which are generated in the training process of the infrared image super-resolution network in each time and are used as prediction results into a pre-trained VGG16 network for feature extraction:
the VGG16 network comprises a plurality of characteristic layers which are sequentially composed of a convolutional layer and a relu layer, wherein one convolutional layer and one relu layer connected to the rear of the convolutional layer form one characteristic layer; for an input image, when k is 1,2,3,4 and the current k-th relu layer of the VGG16 network, k represents the sequence number of the relu layer, and the image features obtained by processing the k-th relu layer are obtained by the following processing:
Fk＝relu(conv(Fk-1)) (2)
wherein, Fk-1The feature map obtained by processing the k-1 relu layer of the previous layer, conv is the convolution operation with the convolution layer, FkProcessing the obtained characteristic diagram for the kth relu layer;
(3) calculating the characteristic loss of each layer of characteristic layer according to the infrared super-resolution image generated by the infrared image super-resolution network and the convolved image by the following formula:
(4) calculating an infrared super-resolution image and a real infrared picture generated by an infrared image super-resolution network, and calculating a Gram matrix under each layer of characteristic layer by the following formula:
and then calculating style loss according to the Gram matrix of each layer of feature layer:
(5) and performing weighted fusion on the calculated characteristic loss of the third layer of characteristic layer and the style loss of the first four layers of characteristic layers to obtain a final loss function:
wherein, w1And w2Weighted values for style loss and feature loss, respectively.
Claims (3)
1. An infrared image detail enhancement method based on an image super-resolution loss function is characterized by comprising the following steps:
(1) preparing an infrared image super-resolution training data set and an infrared image super-resolution testing data set;
(2) constructing an infrared image super-resolution network structure based on a convolutional neural network,
(3) inputting an infrared image super-resolution training data set into the infrared image super-resolution network based on the convolutional neural network constructed in the step (2) for training, constructing a loss function in the training process, continuously performing cyclic iterative training on the infrared image super-resolution network to continuously reduce the loss function until the set iteration times Q are finished, and storing the infrared image super-resolution network;
(4) and (4) carrying out image feature extraction on the infrared super-resolution test data set by using the infrared image super-resolution network obtained in the step (3) to obtain a final infrared super-resolution image.
2. The infrared image detail enhancement method based on the image super-resolution loss function as claimed in claim 1, characterized in that: the loss function in the training process is obtained by processing in the following way:
(1) convolving the real infrared picture by a Sobel operator to obtain a convolved picture:
Is＝convs(Ihr) (1)
wherein, IsFor the convolved picture, convsFor the convolution operation with the Sobel operator, IhrThe real infrared picture is obtained;
(2) respectively inputting the infrared super-resolution image generated by the infrared image super-resolution network, the real infrared picture and the convolved picture into a pre-trained VGG16 network for feature extraction:
the VGG16 network comprises a plurality of characteristic layers which are sequentially composed of a convolutional layer and a relu layer, wherein one convolutional layer and one relu layer connected to the rear of the convolutional layer form one characteristic layer; for an input image, when k is 1,2,3,4 and the current k-th relu layer of the VGG16 network, k represents the sequence number of the relu layer, and the image features obtained by processing the k-th relu layer are obtained by the following processing:
Fk＝relu(conv(Fk-1)) (2)
wherein, Fk-1For the feature map obtained by processing the k-1 relu layer, conv is the convolution operation with convolutional layer, FkProcessing the obtained characteristic diagram for the kth relu layer;
(3) calculating the characteristic loss of each layer of characteristic layer according to the infrared super-resolution image generated by the infrared image super-resolution network and the convolved image by the following formula:
wherein, CkHkWkNumber of channels in three dimensions of the input image, Fs kAnd
respectively processing the convolved picture and the infrared super-resolution image generated by the infrared image super-resolution network on a kth relu layer to obtain characteristic graphs;
(4) calculating an infrared super-resolution image and a real infrared picture generated by an infrared image super-resolution network, and calculating a Gram matrix under each layer of characteristic layer by the following formula:
wherein the content of the first and second substances,
extracting a characteristic diagram of the real infrared picture in the kth relu layer, wherein h and w respectively represent ordinal numbers of the width and height of the image;
and then calculating style loss according to the Gram matrix of each layer of feature layer:
wherein F is a Frobenius norm;
representing the square of the frobenius norm,
and respectively representing the gray matrixes calculated by the characteristics of the real infrared picture and the infrared super-resolution image under the kth relu layer.
(5) And finally, performing weighted fusion on the calculated characteristic loss of the third layer of characteristic layer and the style loss of the first four layers of characteristic layers to obtain a final loss function:
wherein, w1And w2Weighted values for style loss and feature loss, respectively.
3. The infrared image detail enhancement method based on the image super-resolution loss function as claimed in claim 1, characterized in that: the infrared image super-resolution network based on the convolutional neural network comprises N convolutional layers connected in series and a loss function layer.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010645997.1A CN111951177B (en) | 2020-07-07 | 2020-07-07 | Infrared image detail enhancement method based on image super-resolution loss function |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010645997.1A CN111951177B (en) | 2020-07-07 | 2020-07-07 | Infrared image detail enhancement method based on image super-resolution loss function |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111951177A true CN111951177A (en) | 2020-11-17 |
| CN111951177B CN111951177B (en) | 2022-10-11 |
Family
ID=73340175
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010645997.1A Active CN111951177B (en) | 2020-07-07 | 2020-07-07 | Infrared image detail enhancement method based on image super-resolution loss function |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111951177B (en) |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20180075581A1 (en) * | 2016-09-15 | 2018-03-15 | Twitter, Inc. | Super resolution using a generative adversarial network |
| CN109671022A (en) * | 2019-01-22 | 2019-04-23 | 北京理工大学 | A kind of picture texture enhancing super-resolution method based on depth characteristic translation network |
| US20190138838A1 (en) * | 2017-11-09 | 2019-05-09 | Boe Technology Group Co., Ltd. | Image processing method and processing device |
| CN110473142A (en) * | 2019-05-22 | 2019-11-19 | 南京理工大学 | Single image super resolution ratio reconstruction method based on deep learning |
| CN110889813A (en) * | 2019-11-15 | 2020-03-17 | 安徽大学 | Low-light image enhancement method based on infrared information |
| US20200126205A1 (en) * | 2018-10-18 | 2020-04-23 | Boe Technology Group Co., Ltd. | Image processing method, image processing apparatus, computing device and computer-readable storage medium |
| CN111062872A (en) * | 2019-12-17 | 2020-04-24 | 暨南大学 | Image super-resolution reconstruction method and system based on edge detection |
-
2020
- 2020-07-07 CN CN202010645997.1A patent/CN111951177B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20180075581A1 (en) * | 2016-09-15 | 2018-03-15 | Twitter, Inc. | Super resolution using a generative adversarial network |
| US20190138838A1 (en) * | 2017-11-09 | 2019-05-09 | Boe Technology Group Co., Ltd. | Image processing method and processing device |
| US20200126205A1 (en) * | 2018-10-18 | 2020-04-23 | Boe Technology Group Co., Ltd. | Image processing method, image processing apparatus, computing device and computer-readable storage medium |
| CN109671022A (en) * | 2019-01-22 | 2019-04-23 | 北京理工大学 | A kind of picture texture enhancing super-resolution method based on depth characteristic translation network |
| CN110473142A (en) * | 2019-05-22 | 2019-11-19 | 南京理工大学 | Single image super resolution ratio reconstruction method based on deep learning |
| CN110889813A (en) * | 2019-11-15 | 2020-03-17 | 安徽大学 | Low-light image enhancement method based on infrared information |
| CN111062872A (en) * | 2019-12-17 | 2020-04-24 | 暨南大学 | Image super-resolution reconstruction method and system based on edge detection |
Non-Patent Citations (1)
| Title |
|---|
| 钱文华等: "粉笔画艺术风格模拟", 《中国图象图形学报》 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111951177B (en) | 2022-10-11 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110119780B (en) | Hyper-spectral image super-resolution reconstruction method based on generation countermeasure network | |
| CN109741256B (en) | Image super-resolution reconstruction method based on sparse representation and deep learning | |
| CN110120011B (en) | Video super-resolution method based on convolutional neural network and mixed resolution | |
| CN113362223B (en) | Image super-resolution reconstruction method based on attention mechanism and two-channel network | |
| CN114092330B (en) | Light-weight multi-scale infrared image super-resolution reconstruction method | |
| CN115222601A (en) | Image super-resolution reconstruction model and method based on residual mixed attention network | |
| CN110363068B (en) | High-resolution pedestrian image generation method based on multiscale circulation generation type countermeasure network | |
| CN112699844A (en) | Image super-resolution method based on multi-scale residual error level dense connection network | |
| CN104408697B (en) | Image Super-resolution Reconstruction method based on genetic algorithm and canonical prior model | |
| CN114219719A (en) | CNN medical CT image denoising method based on dual attention and multi-scale features | |
| Liu et al. | Single image super resolution techniques based on deep learning: Status, applications and future directions | |
| Shi et al. | Multi-scale deep networks for image compressed sensing | |
| Wang et al. | Image super-resolution using a improved generative adversarial network | |
| CN112950480A (en) | Super-resolution reconstruction method integrating multiple receptive fields and dense residual attention | |
| CN113379597A (en) | Face super-resolution reconstruction method | |
| CN113379606B (en) | Face super-resolution method based on pre-training generation model | |
| CN109615576B (en) | Single-frame image super-resolution reconstruction method based on cascade regression basis learning | |
| Wen et al. | The power of complementary regularizers: Image recovery via transform learning and low-rank modeling | |
| Chen et al. | Guided dual networks for single image super-resolution | |
| CN111951177B (en) | Infrared image detail enhancement method based on image super-resolution loss function | |
| CN113191947B (en) | Image super-resolution method and system | |
| CN114549361A (en) | Improved U-Net model-based image motion blur removing method | |
| CN110648291B (en) | Unmanned aerial vehicle motion blurred image restoration method based on deep learning | |
| CN113205005A (en) | Low-illumination low-resolution face image reconstruction method | |
| Nie et al. | Image restoration from patch-based compressed sensing measurement |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |