CN112446859A - Satellite-borne thermal infrared camera image cloud detection method based on deep learning - Google Patents

Satellite-borne thermal infrared camera image cloud detection method based on deep learning Download PDF

Info

Publication number
CN112446859A
CN112446859A CN202011292436.4A CN202011292436A CN112446859A CN 112446859 A CN112446859 A CN 112446859A CN 202011292436 A CN202011292436 A CN 202011292436A CN 112446859 A CN112446859 A CN 112446859A
Authority
CN
China
Prior art keywords
thermal infrared
dice
image
convolution
model
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202011292436.4A
Other languages
Chinese (zh)
Inventor
陈凡胜
李丽圆
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Shanghai Institute of Technical Physics of CAS
Original Assignee
Shanghai Institute of Technical Physics of CAS
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Shanghai Institute of Technical Physics of CAS filed Critical Shanghai Institute of Technical Physics of CAS
Priority to CN202011292436.4A priority Critical patent/CN112446859A/en
Publication of CN112446859A publication Critical patent/CN112446859A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00Image analysis
    • G06T7/0002Inspection of images, e.g. flaw detection
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00Pattern recognition
    • G06F18/20Analysing
    • G06F18/21Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
    • G06F18/214Generating training patterns; Bootstrap methods, e.g. bagging or boosting
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06NCOMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00Computing arrangements based on biological models
    • G06N3/02Neural networks
    • G06N3/04Architecture, e.g. interconnection topology
    • G06N3/045Combinations of networks
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06NCOMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00Computing arrangements based on biological models
    • G06N3/02Neural networks
    • G06N3/08Learning methods
    • G06N3/084Backpropagation, e.g. using gradient descent
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00Image enhancement or restoration
    • G06T5/50Image enhancement or restoration using two or more images, e.g. averaging or subtraction
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00Indexing scheme for image analysis or image enhancement
    • G06T2207/10Image acquisition modality
    • G06T2207/10032Satellite or aerial image; Remote sensing
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00Indexing scheme for image analysis or image enhancement
    • G06T2207/10Image acquisition modality
    • G06T2207/10048Infrared image
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00Indexing scheme for image analysis or image enhancement
    • G06T2207/20Special algorithmic details
    • G06T2207/20021Dividing image into blocks, subimages or windows
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00Indexing scheme for image analysis or image enhancement
    • G06T2207/20Special algorithmic details
    • G06T2207/20212Image combination
    • G06T2207/20221Image fusion; Image merging

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Artificial Intelligence (AREA)
  • General Engineering & Computer Science (AREA)
  • Evolutionary Computation (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Computational Linguistics (AREA)
  • Software Systems (AREA)
  • Mathematical Physics (AREA)
  • Health & Medical Sciences (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • Computing Systems (AREA)
  • General Health & Medical Sciences (AREA)
  • Molecular Biology (AREA)
  • Evolutionary Biology (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Quality & Reliability (AREA)
  • Image Processing (AREA)

Abstract

The invention discloses a satellite-borne thermal infrared camera image cloud detection method based on deep learning, which comprises the following steps: (1) converting an uncalibrated thermal infrared image digital quantization value into an atmospheric top layer brightness temperature value, fusing all thermal infrared bands into a multi-channel image, cutting a remote sensing image, and constructing a data set; (2) the method comprises the steps that a full convolution neural network is combined with a residual error network to serve as a basic structure, cavity convolution is introduced to replace the ordinary convolution and pooling processes, and a new fusion neural network is constructed; (3) before training the model, setting a loss function distributed according to the weight; (4) training the constructed data set by a back propagation algorithm until a new model converges to obtain a deep learning model; (5) individual pixels are determined to be cloud or non-cloud targets using this model. The model has the advantages of light weight, strong generalization capability and suitability for various scenes and multi-channel images.

Description

Satellite-borne thermal infrared camera image cloud detection method based on deep learning
Technical Field
The invention belongs to the technical field of remote sensing image detection and deep learning, and particularly relates to a satellite-borne thermal infrared camera image cloud detection method based on deep learning.
Background
Estimated from global energy and water cycle experimental (GEWEX) cloud assessment databases, the global annual average cloud coverage is about 68%. When a satellite observes the earth surface, the existence of a cloud layer can block earth surface information, and extreme weather can be predicted by analyzing the movement of the cloud layer in meteorology, so remote sensing image cloud detection is an important step in various remote sensing applications. More importantly, when the thermal infrared band image is used for observation all day long, the satellite-ground transmission resources are occupied by excessive cloud layer information, so that the satellite-carried infrared camera image on-orbit cloud detection becomes a trend in the field of remote sensing.
The current cloud detection method comprises a traditional detection method and an intelligent detection method. However, the traditional cloud detection method is difficult to distinguish cloud layers with complex image backgrounds from the earth surface, and depends too much on human experience, so that many missed detections and false detections exist. In order to solve the above problem, an intelligent detection method is proposed for cloud detection. The deep learning network can eliminate the step of manually setting the characteristic parameters in the traditional cloud detection method, so that the artificial subjective influence is weakened, and the robustness of the detection algorithm is improved. The full convolution neural network and the U-type network evolve cloud detection into a classification problem at the pixel level, and end-to-end training is performed by using a remote sensing image through an encoder-decoder structure. However, the full convolution neural network has a network degradation phenomenon, and the pooling operation reduces the resolution, resulting in a reduction in the detection accuracy. The number of floating point operations required by the U-type network is high, the requirement on the computing capacity of hardware is high, and the on-orbit cloud detection is not facilitated.
In summary, according to the requirements of all-day, accurate, real-time and on-orbit cloud detection, there is an urgent need to develop a method for cloud detection of images of satellite-borne heat infrared cameras.
Disclosure of Invention
The invention aims to overcome the defects of the prior art, provides a satellite-borne thermal infrared camera image cloud detection method based on deep learning, and solves the problem of satellite-borne thermal infrared camera image cloud detection.
The technical problem to be solved by the invention is realized by adopting the following technical scheme:
1) converting an uncalibrated thermal infrared image digital quantization value into an atmospheric top layer brightness temperature value, fusing all thermal infrared bands into a multi-channel image, cutting a remote sensing image, and constructing a data set, wherein the method comprises the following specific steps:
1-1) converting the digital quantized value of the uncalibrated thermal infrared image into the value of the brightness temperature of the top layer of the atmosphere with physical significance, wherein the conversion formula is expressed by the following equation:
TOA_RA=ML·DN+AL (1)
Figure BDA0002784310850000021
wherein TOA _ RA is the atmospheric top level radiance value, DN is the digital quantization value, MLMultiplication rescaling factor for the thermal infrared band, ALIs the addition and readjustment factor of the thermal infrared band, TOA _ BT is the value of the top brightness temperature of the atmosphere, K1And K2Thermal conversion constants for the thermal infrared band;
1-2) fusing all thermal infrared bands into a multi-channel image by using a GDAL library in software;
1-3) cutting the remote sensing image into small blocks of images n.n, wherein n is between 200 and 600;
1-4) when the data volume is less, the proportion of the verification set and the test set is consistent to be 40-20%, and the proportion of the training set is 60-80%;
2) the method is characterized in that a full convolution neural network is combined with a residual error network to serve as a basic structure, a cavity convolution is introduced to replace the common convolution and pooling process, a new fusion neural network is constructed, and a residual error module and a cavity convolution formula are expressed by the following equation:
y=F(x)+x (3)
wherein x and y represent the input and output of the residual module, and f (x) is the residual function to be learned;
(F*lk)(s+lt)=∑s+ltF(s)k(t) (4)
wherein F is a discrete functionlFor discrete convolution, k is a discrete filter, s is a step size, l is a hole factor, t is an argument, F(s) is a discrete convolution with a step size s, k (t) is an argument of tA dispersive filter;
3) before training the model, setting a loss function distributed according to weight, wherein the loss function L is the sum of weight distribution of Dice loss suitable for extreme unevenness of the sample and binary cross entropy loss suitable for class equalization of the sample, and is represented by the following equation:
L=WBceLBce+WDiceLDice (5)
LBce=-∑iTiln(Pi)+(1-Ti)ln(1-Pi) (6)
Figure BDA0002784310850000031
WBce+WDice=1 (8)
wherein L is the loss function of the method, LBceIs a two-class cross entropy loss, LDiceIs the loss of Dice, P is the network prediction result, T is the true value, i is the independent variable, WDiceThe weight loss of the Dice is in the range of 0 to 0.2, WBceThe cross entropy loss weight for the binary class is in the range of 1 to 0.8;
4) training by using the constructed data set through a back propagation algorithm until a new model converges to obtain a deep learning model FCN _ ResNet _ k;
5) individual pixels are determined to be cloud or non-cloud targets using this model.
The invention has the advantages and positive effects that:
1. the invention takes the full convolution neural network and the residual error network as the basic structure, and effectively avoids the network degradation phenomenon caused by the deeper network. The cavity convolution is introduced to replace the common convolution and pooling processes, so that the influence of the reduction of the resolution due to the pooling operation in the original model is overcome, and the receptive field index is increased. And constructing a new converged neural network FCN _ ResNet _ k for cloud detection.
2. The invention has reasonable design and less floating point operation times required by the network structure, and can improve the reasoning speed and the network efficiency. The method has the advantages of light weight, strong generalization capability, suitability for various scenes and multi-channel images and the like, and can be applied and popularized in the field of satellite-borne heat infrared camera image cloud detection.
Drawings
Fig. 1 is a diagram illustrating an exemplary remote sensing image detection process according to an embodiment of the present invention;
fig. 2 is an exemplary diagram of a remote sensing image preprocessing process according to an embodiment of the present invention, where (a) and (b) are original images of bands of 10um to 11.19um and 11.5um to 12.51um, respectively, and (c) and (d) are images obtained by converting digital quantization values of corresponding bands into atmospheric top luminance temperature values, respectively, and (e) is a two-channel image obtained by fusing two thermal infrared bands.
Detailed Description
The following takes thermal infrared band cloud detection as an example, and the following detailed description is made on the embodiments of the present invention with reference to the accompanying drawings:
the invention realizes the high-precision infrared band cloud detection method by constructing an improved FCN model and introducing a voting principle to vote by using three different models. The method mainly comprises the following steps:
1) converting an uncalibrated thermal infrared image digital quantization value into an atmospheric top layer brightness temperature value, fusing all thermal infrared bands into a multi-channel image, cutting a remote sensing image, and constructing a data set, wherein the method comprises the following specific steps:
1-1) converting the digital quantized value of the uncalibrated thermal infrared image into the value of the brightness temperature of the top layer of the atmosphere with physical significance, wherein the conversion formula is expressed by the following equation:
TOA_RA=ML·DN+AL (1)
Figure BDA0002784310850000041
wherein TOA _ RA is the atmospheric top level radiance value, DN is the digital quantization value, ML=3.342×10-4Multiplication rescaling factor for the thermal infrared band, AL0.1 is the thermal infrared band addition and readjustment factor, TOA _ BT is the atmospheric top luminance temperature value, K1And K2In the thermal infrared bandThermal conversion constant of (d); selecting 774.89 for K1 with a wave band of 10um-11.19um, 1321.08 for K2, 480.89 for K1 with a wave band of 11.5um-12.51um and 1201.14 for K2;
1-2) fusing two thermal infrared bands of 10um-11.19um and 11.5um-12.51um into a two-channel image by using a GDAL library in software, and converting a 16-bit remote sensing image into an 8-bit remote sensing image;
1-3) cutting 96 two-channel images into small block images n.n, wherein n is 224, and 38400 small block images 224 are obtained after cutting;
1-4) the data volume is less, a training set is set: and (4) verification set: the ratio of test sets was 8: 1: 1, the data set comprises 32000 training sets, 3200 verification sets and 3200 test sets;
wherein F is a discrete functionlFor discrete convolution, k is a discrete filter, s is a step size, l is a hole factor, t is an argument, F(s) is discrete convolution with a step size s, and k (t) is a discrete filter with an argument t;
2) the method is characterized in that a full convolution neural network is combined with a residual error network to serve as a basic structure, void convolution is introduced to replace the common convolution and pooling process, a new model is constructed, and a residual error module and a void convolution formula are expressed by the following equation:
y=F(x)+x (3)
wherein x and y represent the input and output of the residual module, and f (x) is the residual function to be learned;
(F*lk)(s+lt)=∑s+ltF(s)k(t) (4)
wherein F is a discrete functionlFor discrete convolution, k is a discrete filter, s is a step size, l is a hole factor, t is an argument, F(s) is discrete convolution with a step size s, and k (t) is a discrete filter with an argument t;
3) before training the model, setting a loss function distributed according to weight, wherein the loss function L is the sum of weight distribution of Dice loss suitable for extreme unevenness of the sample and binary cross entropy loss suitable for class equalization of the sample, and is represented by the following equation:
L=WBceLBce+WDiceLDice (5)
LBce=-∑iTiln(Pi)+(1-Ti)ln(1-Pi) (6)
Figure BDA0002784310850000061
WBce+WDice=1 (8)
wherein P is the network prediction result, T is the truth value, and LDiceFor loss of Dice, LBceIs a binary cross entropy loss, i is an independent variable, WBceTake 0.2, W for the Dice loss weightDiceTaking 0.8 as the weight range of the binary cross entropy loss, wherein L is the loss function of the method;
4) training by using the constructed data set through a back propagation algorithm until a new model converges to obtain a deep learning model FCN _ ResNet _ k;
5) individual pixels are determined to be cloud or non-cloud targets using this model.
6) And calculating the floating point operation times of the model and measuring the calculation complexity of the model. The number of floating point operations of the new fusion neural network is 7445043712, the number of floating point operations of the U-type network is 85683945472, and the calculation complexity of the U-type network is about 11.51 times that of the new fusion neural network.

Claims (1)

1. A satellite-borne thermal infrared camera image cloud detection method based on deep learning is characterized by comprising the following steps:
1) converting an uncalibrated thermal infrared image digital quantization value into an atmospheric top layer brightness temperature value, fusing all thermal infrared bands into a multi-channel image, cutting a remote sensing image, and constructing a data set, wherein the method comprises the following specific steps:
1-1) converting the digital quantized value of the uncalibrated thermal infrared image into the value of the brightness temperature of the top layer of the atmosphere with physical significance, wherein the conversion formula is expressed by the following equation:
TOA_RA=ML·DN+AL (1)
Figure FDA0002784310840000011
wherein TOA _ RA is the atmospheric top level radiance value, DN is the digital quantization value, MLMultiplication rescaling factor for the thermal infrared band, ALIs the addition and readjustment factor of the thermal infrared band, TOA _ BT is the value of the top brightness temperature of the atmosphere, K1And K2Thermal conversion constants for the thermal infrared band;
1-2) fusing all thermal infrared bands into a multi-channel image by using a GDAL library in software;
1-3) cutting the remote sensing image into small blocks of images n.n, wherein n is between 200 and 600;
1-4) when the data volume is less, the proportion of the verification set and the test set is consistent to be 40-20%, and the proportion of the training set is 60-80%;
2) the method is characterized in that a full convolution neural network is combined with a residual error network to serve as a basic structure, a cavity convolution is introduced to replace the common convolution and pooling process, a new fusion neural network is constructed, and a residual error module and a cavity convolution formula are expressed by the following equation:
y=F(x)+x (3)
wherein x and y represent the input and output of the residual module, and f (x) is the residual function to be learned;
(F*lk)(s+lt)=∑s+ltF(s)k(t) (4)
wherein F is a discrete functionlFor discrete convolution, k is a discrete filter, s is a step size, l is a hole factor, t is an argument, F(s) is discrete convolution with a step size s, and k (t) is a discrete filter with an argument t;
3) before training the model, setting a loss function distributed according to weight, wherein the loss function L is the sum of weight distribution of Dice loss suitable for extreme unevenness of the sample and binary cross entropy loss suitable for class equalization of the sample, and is represented by the following equation:
L=WBceLBce+WDiceLDice (5)
LBce=-∑iTiln(Pi)+(1-Ti)ln(1-Pi) (6)
Figure FDA0002784310840000021
WBce+WDice=1 (8)
wherein P is the network prediction result, T is the truth value, and LDiceFor loss of Dice, LBceIs a binary cross entropy loss, i is an independent variable, WBceThe weight loss of the Dice is in the range of 0 to 0.2, WDiceThe weight loss range of the binary cross entropy is 1-0.8, and L is the loss function of the method;
4) training the constructed data set by a back propagation algorithm until a new model converges to obtain a deep learning model;
5) individual pixels are determined to be cloud or non-cloud targets using this model.
CN202011292436.4A 2020-11-18 2020-11-18 Satellite-borne thermal infrared camera image cloud detection method based on deep learning Pending CN112446859A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202011292436.4A CN112446859A (en) 2020-11-18 2020-11-18 Satellite-borne thermal infrared camera image cloud detection method based on deep learning

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202011292436.4A CN112446859A (en) 2020-11-18 2020-11-18 Satellite-borne thermal infrared camera image cloud detection method based on deep learning

Publications (1)

Publication Number Publication Date
CN112446859A true CN112446859A (en) 2021-03-05

Family

ID=74737473

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202011292436.4A Pending CN112446859A (en) 2020-11-18 2020-11-18 Satellite-borne thermal infrared camera image cloud detection method based on deep learning

Country Status (1)

Country Link
CN (1) CN112446859A (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115564778A (en) * 2022-12-06 2023-01-03 深圳思谋信息科技有限公司 Defect detection method and device, electronic equipment and computer readable storage medium

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20200058126A1 (en) * 2018-08-17 2020-02-20 12 Sigma Technologies Image segmentation and object detection using fully convolutional neural network
CN111274865A (en) * 2019-12-14 2020-06-12 深圳先进技术研究院 Remote sensing image cloud detection method and device based on full convolution neural network

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20200058126A1 (en) * 2018-08-17 2020-02-20 12 Sigma Technologies Image segmentation and object detection using fully convolutional neural network
CN111274865A (en) * 2019-12-14 2020-06-12 深圳先进技术研究院 Remote sensing image cloud detection method and device based on full convolution neural network

Non-Patent Citations (3)

* Cited by examiner, † Cited by third party
Title
张丹丹: "《基于深度神经网络技术的高分遥感图像处理及应用》", 31 August 2020, 中国宇航出版社 *
张家强 等: "基于深度残差全卷积网络的Landsat 8遥感影像云检测方法", 《激光与光电子学进展》 *
赵志刚: "《区域土地资源研究与农业规划实例 以宜春市袁州区为例》", 31 October 2017, 科学技术文献出版社 *

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115564778A (en) * 2022-12-06 2023-01-03 深圳思谋信息科技有限公司 Defect detection method and device, electronic equipment and computer readable storage medium
CN115564778B (en) * 2022-12-06 2023-03-14 深圳思谋信息科技有限公司 Defect detection method and device, electronic equipment and computer readable storage medium

Similar Documents

Publication Publication Date Title
CN112507793B (en) Ultra-short term photovoltaic power prediction method
CN108960333B (en) Hyperspectral image lossless compression method based on deep learning
CN112634292A (en) Asphalt pavement crack image segmentation method based on deep convolutional neural network
CN110210378B (en) Embedded video image analysis method and device based on edge calculation
CN111582555B (en) Photovoltaic power prediction method based on foundation cloud picture image characteristics
CN110826693A (en) Three-dimensional atmospheric temperature profile inversion method and system based on DenseNet convolutional neural network
CN113239830A (en) Remote sensing image cloud detection method based on full-scale feature fusion
CN112883887B (en) Building instance automatic extraction method based on high spatial resolution optical remote sensing image
CN112232543A (en) Multi-site prediction method based on graph convolution network
CN114444757A (en) Combined prediction method for plateau mountain multi-model multi-scale new energy power station output
CN114120036A (en) Lightweight remote sensing image cloud detection method
CN115935809A (en) Photovoltaic power generation power interval prediction method based on cloud cover characteristic of foundation cloud cover
CN115578280A (en) Construction method of double-branch remote sensing image defogging network
CN116205962A (en) Monocular depth estimation method and system based on complete context information
CN115393690A (en) Light neural network air-to-ground observation multi-target identification method
Zhou et al. MSAR‐DefogNet: Lightweight cloud removal network for high resolution remote sensing images based on multi scale convolution
CN112446859A (en) Satellite-borne thermal infrared camera image cloud detection method based on deep learning
CN113705340B (en) Deep learning change detection method based on radar remote sensing data
CN114373104A (en) Three-dimensional point cloud semantic segmentation method and system based on dynamic aggregation
CN114511787A (en) Neural network-based remote sensing image ground feature information generation method and system
CN117197632A (en) Transformer-based electron microscope pollen image target detection method
CN116720156A (en) Weather element forecasting method based on graph neural network multi-mode weather data fusion
CN116523875A (en) Insulator defect detection method based on FPGA pretreatment and improved YOLOv5
CN115561834A (en) Meteorological short-term and temporary forecasting all-in-one machine based on artificial intelligence
CN113034445B (en) Multi-scale connection image defogging algorithm based on UNet3+

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
WD01 Invention patent application deemed withdrawn after publication

Application publication date: 20210305

WD01 Invention patent application deemed withdrawn after publication