CN110852947A - Infrared image super-resolution method based on edge sharpening - Google Patents

Infrared image super-resolution method based on edge sharpening Download PDF

Info

Publication number
CN110852947A
CN110852947A CN201911046112.XA CN201911046112A CN110852947A CN 110852947 A CN110852947 A CN 110852947A CN 201911046112 A CN201911046112 A CN 201911046112A CN 110852947 A CN110852947 A CN 110852947A
Authority
CN
China
Prior art keywords
image
infrared image
infrared
super
resolution
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN201911046112.XA
Other languages
Chinese (zh)
Other versions
CN110852947B (en
Inventor
冯华君
杨一帆
徐之海
李奇
陈跃庭
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Zhejiang University ZJU
Original Assignee
Zhejiang University ZJU
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Zhejiang University ZJU filed Critical Zhejiang University ZJU
Priority to CN201911046112.XA priority Critical patent/CN110852947B/en
Publication of CN110852947A publication Critical patent/CN110852947A/en
Application granted granted Critical
Publication of CN110852947B publication Critical patent/CN110852947B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
    • G06T3/4053—Scaling of whole images or parts thereof, e.g. expanding or contracting based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
    • G06T3/4007—Scaling of whole images or parts thereof, e.g. expanding or contracting based on interpolation, e.g. bilinear interpolation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
    • G06T3/4046—Scaling of whole images or parts thereof, e.g. expanding or contracting using neural networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/70—Denoising; Smoothing
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/73—Deblurring; Sharpening
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10048—Infrared image
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20172—Image enhancement details
    • G06T2207/20192—Edge enhancement; Edge preservation

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Artificial Intelligence (AREA)
  • Evolutionary Computation (AREA)
  • Image Processing (AREA)
  • Studio Devices (AREA)

Abstract

The invention discloses an infrared image super-resolution method based on edge sharpening. Acquiring an infrared image by using an infrared camera, and establishing an infrared super-resolution neural network structure, wherein the network comprises two sub-networks of image processing and image edge processing, the image processing network is mainly used for recovering the structure information of the image aiming at the infrared image input into the network, and the edge processing network is used for recovering the detail edge information of the image; the image processing network is divided into two stages, wherein the first stage realizes the denoising of the infrared image and the recovery of the structural information of the image, and the second stage realizes the super resolution of the image and the recovery of more detailed structural information of the image. The invention realizes the high-magnification infrared image super-resolution by respectively processing the image structure and the edge information based on the requirement of the digital infrared image super-resolution.

Description

Infrared image super-resolution method based on edge sharpening
Technical Field
The invention belongs to an infrared image super-resolution algorithm in the technical field of digital imaging, and particularly relates to an infrared image super-resolution method based on edge sharpening.
Technical Field
With the development of the infrared detection technology, the infrared imaging technology is adopted for target identification, the intelligent detection and detection identification capabilities of the target are improved, the infrared image is influenced by the edge contour feature interference of the target in the detection process of the infrared image, the output quality of the infrared image is poor, and the identification and detection capabilities of the target single-frame infrared image are reduced.
A high resolution image may provide more detail than its corresponding low resolution image. These details should be crucial in all areas. Due to the limitation of hardware devices, super resolution has been widely applied to many imaging devices. Super-resolution is the digital magnification of an image without changing the focal length of the lens, thus resulting in a degradation of image quality: however, image processing algorithms (e.g., image interpolation) do not produce high quality pictures except for aliasing and blurring artifacts. To address this problem, many improved algorithms have been proposed over the past few decades. For example, interpolation is used to increase the spatial resolution of the input image, interpolation-based restoration methods aim at searching for connections between neighboring pixels and filling missing pixel functions or interpolation kernels, one by one, and so on. Although it has a fast processing time at low computational complexity, the method of stepwise operation does not guarantee the accuracy of the estimation, especially in the presence of noise. Some documents propose to super-resolve the infrared image by using a neural network method, but the output image quality is not high due to the large noise of the infrared image.
Disclosure of Invention
In order to solve the problems in the background art, the invention provides an infrared image super-resolution method based on edge sharpening, which improves the imaging quality of the infrared image super-resolution, designs images with different magnifications and provides a new method in a mode of repairing by using the edge of the infrared image and sharpening.
The method is based on the neural network, the image is denoised and the image details are recovered in the network, the detail information of the image is enhanced by extracting the edge of the image, and the image is sharpened, so that the details of the output image are richer.
The technical scheme adopted by the invention comprises the following steps:
step 1: selecting an infrared image obtained by shooting a scene or an object by an infrared camera and a training target image corresponding to the infrared image as a training set;
step 2: establishing an infrared image super-resolution neural network, wherein the infrared image super-resolution neural network structure comprises an image processing sub-network and an image edge processing sub-network;
and step 3: inputting the training set into an infrared image super-resolution neural network for training;
and 4, step 4: and (3) inputting the infrared image to be restored, which is obtained by shooting a scene or an object by using an infrared camera, into the infrared image super-resolution neural network trained in the step (3) to obtain the super-resolution image of the infrared image to be restored.
The step 2 specifically comprises the following steps:
2.1) the image processing sub-network comprises two stages, stage one and stage two;
adding the extracted features after carrying out a plurality of groups of convolution operations on the infrared image input stage I and the input infrared image pixel by pixel to obtain a blurred image subjected to Gaussian denoising and recover low-frequency information of the infrared image;
inputting the image obtained in the first stage after the Gaussian denoising into a second stage to continue carrying out multiple groups of convolution operations, and recovering high-frequency information of the infrared image;
2.2) inputting the infrared image into an image edge processing sub-network for convolution operation, extracting edge detail characteristics of the infrared image and generating an edge image;
2.3) adding the edge image generated in the step 2.2) and the processing result of the stage two in the step 2.1) pixel by pixel to generate a final output image, namely obtaining a super-resolution image of the infrared image.
The training target images in the step 1 are as follows:
training a target image I: performing Gaussian blur processing with a blur kernel of 7 and a variance of 3 on the infrared image to realize image denoising, wherein the image is used as a training target image of the image processing sub-network stage one in the step 2.1);
training a target image II: carrying out image edge extraction on the infrared image by using a sobel operator to serve as a training target image of the image edge processing sub-network in the step 2.2);
training a target image three: and (3) carrying out sharpening processing on the infrared image to be used as the training target image in the step 2.3).
The super-resolution image of the infrared image obtained in the step 2.3) is 2 times or 4 times of the size of the input infrared image, and the resolution of the super-resolution image is higher than that of the input infrared image.
The invention has the following beneficial effects:
(1) the invention realizes the super resolution of digital images with any multiplying power through the structure of an image convolution network based on the requirement of the super resolution of infrared images.
(2) The method carries out denoising processing on the infrared image in the super-resolution process of the infrared image for the first time, improves the details of the image by utilizing the sharpened image and the edge information of the enhanced image, and obviously improves the visual effect of the image compared with the prior art.
Drawings
FIG. 1 is a schematic diagram of a super-resolution neural network structure of an infrared image;
FIG. 2 is an input diagram of example 1 in which 2-fold super resolution is performed;
FIG. 3 is a comparison of the input plot of example 1 using bicubic interpolation, VDSR, and the method of the present invention;
FIG. 4 is a graph of the input graph of example 1 using bicubic interpolation, VDSR, and a comparison of the details of the method of the present invention;
FIG. 5 is an input diagram of exemplary embodiment 2 with 4-fold super resolution;
FIG. 6 is a graph of comparative results of the input graph of example 2 using bicubic interpolation, VDSR, and the method of the present invention;
fig. 7 is a graph of the input graph of example 2 using bicubic interpolation, VDSR, and a comparison of the details of the method of the present invention.
Detailed Description
The invention is further described below with reference to the figures and examples.
As shown in fig. 1, the infrared image super-resolution neural network structure includes an image processing sub-network and an image edge processing sub-network;
1) the image processing sub-network comprises a first stage and a second stage;
in stage one, the input infrared image is subjected to image feature extraction through a multilayer convolution layer, and the convolution can be expressed by formula (1):
Fgi＝Convi(I) (1)
wherein, Convi(I) Denotes the convolution of step size i, FgiRepresenting features extracted by the ith convolution in the first stage, wherein I represents an input image or features;
adding the input infrared image and the characteristics extracted by the stage convolution layer pixel by pixel to obtain a Gaussian denoised image; and inputting the Gaussian denoised image generated in the stage one into a plurality of convolution layers in the stage two to further extract the characteristics of the graph.
2) The infrared image is input into an image edge processing sub-network, and the convolution layer is directly extracted to generate an edge image.
3) And finally, adding the edge image and the processing result of the second stage pixel by pixel to generate a final output image.
And in the training stage, comparing the Gaussian denoised image generated in the first stage with the image of the infrared image after Gaussian blur processing, comparing the edge image generated by the image edge processing sub-network with the image edge target image extracted by the sobel operator, and respectively calculating an L1 norm as a loss function. And comparing the finally generated output image with the sharpened infrared image, and calculating an L1 norm as a loss function. Weighted summation is carried out on the three loss functions to serve as the loss function of the whole network, and an Adam optimizer is used for optimizing network parameters
The specific embodiment of the invention is as follows:
the specific implementation of the invention comprises two stages, namely a stage one and a stage two. In the first stage, inputting an infrared picture into a network, performing convolution operation on the image by nine convolution kernels with the step size of 1 and the size of 3 × 64 and one convolution kernel with the step size of 1 and the size of 3 × 1, and then performing pixel-by-pixel addition on the image and the input infrared image to generate a Gaussian denoised image; the edge image is generated by simultaneously convolving the image with five convolution kernels of size 3 x 64 with a step size 1 and one convolution kernel of size 3 x 1 with a step size 1. And then, performing convolution on the Gaussian denoised image generated in the first stage by using 5 convolution kernels with the step size of 1 and the size of 3 x 64 and one convolution kernel with the step size of 1 and the size of 3 x 1 to obtain a final output image.
And performing interpolation operation of corresponding multiplying power on the input image according to the resolution requirements of different scaling multiples, so as to obtain super-resolution images of different multiplying powers.
The invention uses the structure shown in fig. 1 to respectively carry out imaging with 2 times of resolution and 4 times of resolution on the infrared images shown in fig. 2 and 5, and compares the imaging with bicubic interpolation and VDSR algorithm, thereby illustrating the beneficial effect of the invention.
As shown in fig. 3 to 4 and fig. 6 to 7, comparing the super-resolution images generated by the method of the present invention and the bicubic interpolation and VDSR, it can be found that the texture of the image generated by the method of the present invention is richer and the details are more obvious, no matter the image is 2 times resolution or 4 times resolution.
In an infrared image super-resolution system, in order to reconstruct an infrared image with high resolution, a neural network super-resolution algorithm of a loss function structure by using edge extraction and picture sharpening is firstly proposed. Meanwhile, the image denoising purpose is achieved by combining the Gaussian denoised image loss, and the super-resolution imaging quality of the infrared image is obviously improved.

Claims (4)

1. An infrared image super-resolution method based on edge sharpening is characterized by comprising the following steps:
step 1: selecting an infrared image obtained by shooting a scene or an object by an infrared camera and a training target image corresponding to the infrared image as a training set;
step 2: establishing an infrared image super-resolution neural network, wherein the infrared image super-resolution neural network structure comprises an image processing sub-network and an image edge processing sub-network;
and step 3: inputting the training set into an infrared image super-resolution neural network for training;
and 4, step 4: and (3) inputting the infrared image to be restored, which is obtained by shooting a scene or an object by using an infrared camera, into the infrared image super-resolution neural network trained in the step (3) to obtain the super-resolution image of the infrared image to be restored.
2. The infrared image super-resolution method based on edge sharpening according to claim 1, wherein the step 2 specifically comprises:
2.1) the image processing sub-network comprises two stages, stage one and stage two;
adding the extracted features after carrying out a plurality of groups of convolution operations on the infrared image input stage I and the input infrared image pixel by pixel to obtain a blurred image subjected to Gaussian denoising and recover low-frequency information of the infrared image;
inputting the image obtained in the first stage after the Gaussian denoising into a second stage to continue carrying out multiple groups of convolution operations, and recovering high-frequency information of the infrared image;
2.2) inputting the infrared image into an image edge processing sub-network for convolution operation, extracting edge detail characteristics of the infrared image and generating an edge image;
2.3) adding the edge image generated in the step 2.2) and the processing result of the stage two in the step 2.1) pixel by pixel to generate a final output image, namely obtaining a super-resolution image of the infrared image.
3. The infrared image super-resolution method based on edge sharpening as claimed in claim 2, wherein the training target image in step 1 is:
training a target image I: performing Gaussian blur processing with a blur kernel of 7 and a variance of 3 on the infrared image to realize image denoising, wherein the image is used as a training target image of the image processing sub-network stage one in the step 2.1);
training a target image II: carrying out image edge extraction on the infrared image by using a sobel operator to serve as a training target image of the image edge processing sub-network in the step 2.2);
training a target image three: and (3) carrying out sharpening processing on the infrared image to be used as the training target image in the step 2.3).
4. The infrared image super-resolution method based on edge sharpening as claimed in claim 2, wherein the super-resolution image of the infrared image obtained in step 2.3) is 2 times or 4 times the size of the input infrared image, and the resolution of the super-resolution image is higher than that of the input infrared image.
CN201911046112.XA 2019-10-30 2019-10-30 Infrared image super-resolution method based on edge sharpening Active CN110852947B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201911046112.XA CN110852947B (en) 2019-10-30 2019-10-30 Infrared image super-resolution method based on edge sharpening

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201911046112.XA CN110852947B (en) 2019-10-30 2019-10-30 Infrared image super-resolution method based on edge sharpening

Publications (2)

Publication Number Publication Date
CN110852947A true CN110852947A (en) 2020-02-28
CN110852947B CN110852947B (en) 2021-07-20

Family

ID=69599307

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201911046112.XA Active CN110852947B (en) 2019-10-30 2019-10-30 Infrared image super-resolution method based on edge sharpening

Country Status (1)

Country Link
CN (1) CN110852947B (en)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111932460A (en) * 2020-08-10 2020-11-13 北京大学深圳医院 MR image super-resolution reconstruction method and device, computer equipment and storage medium
CN113674176A (en) * 2021-08-23 2021-11-19 北京市商汤科技开发有限公司 Image restoration method and device, electronic equipment and storage medium

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106897986A (en) * 2017-01-23 2017-06-27 浙江大学 A kind of visible images based on multiscale analysis and far infrared image interfusion method
CN107392852A (en) * 2017-07-10 2017-11-24 深圳大学 Super resolution ratio reconstruction method, device, equipment and the storage medium of depth image
WO2017219263A1 (en) * 2016-06-22 2017-12-28 中国科学院自动化研究所 Image super-resolution enhancement method based on bidirectional recursion convolution neural network
CN108961180A (en) * 2018-06-22 2018-12-07 理光软件研究所（北京）有限公司 infrared image enhancing method and system
CN109389552A (en) * 2017-08-02 2019-02-26 中山大学 A kind of Image Super-resolution based on context-sensitive multitask deep learning
CN109801215A (en) * 2018-12-12 2019-05-24 天津津航技术物理研究所 The infrared super-resolution imaging method of network is generated based on confrontation

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2017219263A1 (en) * 2016-06-22 2017-12-28 中国科学院自动化研究所 Image super-resolution enhancement method based on bidirectional recursion convolution neural network
CN106897986A (en) * 2017-01-23 2017-06-27 浙江大学 A kind of visible images based on multiscale analysis and far infrared image interfusion method
CN107392852A (en) * 2017-07-10 2017-11-24 深圳大学 Super resolution ratio reconstruction method, device, equipment and the storage medium of depth image
CN109389552A (en) * 2017-08-02 2019-02-26 中山大学 A kind of Image Super-resolution based on context-sensitive multitask deep learning
CN108961180A (en) * 2018-06-22 2018-12-07 理光软件研究所（北京）有限公司 infrared image enhancing method and system
CN109801215A (en) * 2018-12-12 2019-05-24 天津津航技术物理研究所 The infrared super-resolution imaging method of network is generated based on confrontation

Non-Patent Citations (3)

* Cited by examiner, † Cited by third party
Title
YI LIU等: "Low Resolution Cell Image Edge Segmentation Based on Convolutional Neural Network", 《IEEE XPLORE》 *
管志强等: "基于模糊集的自适应红外图像边缘锐化算法", 《光子学报》 *
邵保泰等: "基于生成对抗网络的单帧红外图像超分辨算法", 《红外与毫米波学报》 *

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111932460A (en) * 2020-08-10 2020-11-13 北京大学深圳医院 MR image super-resolution reconstruction method and device, computer equipment and storage medium
CN111932460B (en) * 2020-08-10 2023-09-22 北京大学深圳医院 MR image super-resolution reconstruction method, device, computer equipment and storage medium
CN113674176A (en) * 2021-08-23 2021-11-19 北京市商汤科技开发有限公司 Image restoration method and device, electronic equipment and storage medium
CN113674176B (en) * 2021-08-23 2024-04-16 北京市商汤科技开发有限公司 Image restoration method and device, electronic equipment and storage medium

Also Published As

Publication number Publication date
CN110852947B (en) 2021-07-20

Similar Documents

Publication Publication Date Title
Luo et al. Deep constrained least squares for blind image super-resolution
CN109102462B (en) Video super-resolution reconstruction method based on deep learning
Xu et al. Structure-texture aware network for low-light image enhancement
Dudhane et al. Burstormer: Burst image restoration and enhancement transformer
US20080025628A1 (en) Enhancement of Blurred Image Portions
CN103841298B (en) Video image stabilization method based on color constant and geometry invariant features
Sada et al. Image deblurring techniques—a detail review
CN110852947B (en) Infrared image super-resolution method based on edge sharpening
CN114331886A (en) Image deblurring method based on depth features
Zhang et al. Deep motion blur removal using noisy/blurry image pairs
CN115345791A (en) Infrared image deblurring algorithm based on attention mechanism residual error network model
Chang et al. Beyond camera motion blur removing: How to handle outliers in deblurring
Fazlali et al. Atmospheric turbulence removal in long-range imaging using a data-driven-based approach
Chi et al. Joint demosaicking and blind deblurring using deep convolutional neural network
Chen et al. Missing recovery: Single image reflection removal based on auxiliary prior learning
Tseng et al. Depth image super-resolution via multi-frame registration and deep learning
Holla et al. EFID: edge-focused image denoising using a convolutional neural network
Tao et al. LEGAN: A low-light image enhancement generative adversarial network for industrial internet of smart-cameras
Oh et al. Fpanet: Frequency-based video demoireing using frame-level post alignment
Sadaka et al. Efficient super-resolution driven by saliency selectivity
Ollion et al. Joint self-supervised blind denoising and noise estimation
Lal et al. A comparative study on CNN based low-light image enhancement
Suresha et al. Computer vision approach for motion blur image restoration system
Dudhane et al. Burst Image Restoration and Enhancement
Ahalya et al. Deep Learning for Single Image Deblurring

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant