CN116563308A - SAR image end-to-end change detection method combining super-pixel segmentation and twin network - Google Patents
SAR image end-to-end change detection method combining super-pixel segmentation and twin network Download PDFInfo
- Publication number
- CN116563308A CN116563308A CN202310517307.8A CN202310517307A CN116563308A CN 116563308 A CN116563308 A CN 116563308A CN 202310517307 A CN202310517307 A CN 202310517307A CN 116563308 A CN116563308 A CN 116563308A
- Authority
- CN
- China
- Prior art keywords
- pixel
- super
- network
- sar image
- omega
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000001514 detection method Methods 0.000 title claims abstract description 72
- 230000008859 change Effects 0.000 title claims abstract description 63
- 230000011218 segmentation Effects 0.000 title claims abstract description 46
- 238000012549 training Methods 0.000 claims abstract description 19
- 238000005516 engineering process Methods 0.000 claims abstract description 7
- 238000000034 method Methods 0.000 claims description 27
- 230000014509 gene expression Effects 0.000 claims description 18
- 238000013507 mapping Methods 0.000 claims description 17
- 239000011159 matrix material Substances 0.000 claims description 15
- 238000004364 calculation method Methods 0.000 claims description 7
- 238000010606 normalization Methods 0.000 claims description 7
- 238000011160 research Methods 0.000 claims description 7
- 238000012937 correction Methods 0.000 claims description 6
- 238000007781 pre-processing Methods 0.000 claims description 6
- 230000008569 process Effects 0.000 claims description 4
- 230000005855 radiation Effects 0.000 claims description 3
- 230000002441 reversible effect Effects 0.000 claims description 3
- 230000002123 temporal effect Effects 0.000 claims description 3
- 230000000694 effects Effects 0.000 abstract description 7
- 238000005065 mining Methods 0.000 abstract 1
- 230000008447 perception Effects 0.000 abstract 1
- 230000006870 function Effects 0.000 description 9
- 230000008901 benefit Effects 0.000 description 5
- 238000010586 diagram Methods 0.000 description 5
- 238000013135 deep learning Methods 0.000 description 4
- 238000012545 processing Methods 0.000 description 3
- 238000004458 analytical method Methods 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 230000018109 developmental process Effects 0.000 description 2
- 238000011156 evaluation Methods 0.000 description 2
- 238000009499 grossing Methods 0.000 description 2
- 238000002372 labelling Methods 0.000 description 2
- 238000011158 quantitative evaluation Methods 0.000 description 2
- 238000005070 sampling Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 230000008034 disappearance Effects 0.000 description 1
- 230000009977 dual effect Effects 0.000 description 1
- 238000002474 experimental method Methods 0.000 description 1
- 238000001914 filtration Methods 0.000 description 1
- 238000005286 illumination Methods 0.000 description 1
- 238000003384 imaging method Methods 0.000 description 1
- 230000036039 immunity Effects 0.000 description 1
- 238000011835 investigation Methods 0.000 description 1
- 238000003064 k means clustering Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 230000000737 periodic effect Effects 0.000 description 1
- 238000002203 pretreatment Methods 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/11—Region-based segmentation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
- G06N3/084—Backpropagation, e.g. using gradient descent
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
- G06N3/088—Non-supervised learning, e.g. competitive learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/70—Denoising; Smoothing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/80—Geometric correction
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/30—Determination of transform parameters for the alignment of images, i.e. image registration
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/762—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using clustering, e.g. of similar faces in social networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
- G06V10/774—Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10032—Satellite or aerial image; Remote sensing
- G06T2207/10044—Radar image
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- General Health & Medical Sciences (AREA)
- Software Systems (AREA)
- Artificial Intelligence (AREA)
- Computing Systems (AREA)
- Health & Medical Sciences (AREA)
- Evolutionary Computation (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Multimedia (AREA)
- Medical Informatics (AREA)
- Databases & Information Systems (AREA)
- Life Sciences & Earth Sciences (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Molecular Biology (AREA)
- General Engineering & Computer Science (AREA)
- Mathematical Physics (AREA)
- Image Processing (AREA)
Abstract
The invention discloses a SAR image end-to-end change detection method combining super-pixel segmentation and a twin network, which comprises the steps of firstly generating a difference map by utilizing preprocessed SAR images in different time phases and selecting a pseudo sample; sample expansion is carried out by utilizing a data enhancement technology; further, a super-pixel segmentation network is constructed, and advanced features containing super-pixel segmentation information of different phases are obtained and input into a twin change detection network; the end-to-end training super-pixel segmentation network and the twin network obtain global optimal parameters, and the problem that the traditional super-pixel segmentation and the traditional change detection are mutually separated, and the detection precision is severely limited by the segmentation precision is solved. The invention utilizes the unsupervised idea, fully considers the independence and the cooperativity of multi-time phase information mining and utilization, generates high-quality task self-adaptive superpixels, enhances the semantic perception capability of a model, improves the noise robustness and the detail holding capability, and improves the segmentation precision of a change boundary and the detection effect of a small-area change region.
Description
Technical Field
The invention relates to a remote sensing image processing technology, in particular to a SAR image end-to-end change detection method combining super-pixel segmentation and a twin network.
Background
The remote sensing technology has the advantages of large-scale and periodic observation, is an important method for detecting the change of the earth surface based on the change detection of remote sensing data, and is widely applied to the aspects of urban planning, environment monitoring, agricultural investigation, disaster assessment and the like. The synthetic aperture radar (Synthetic Aperture Radar, SAR) is an advanced active remote sensing technology, has better penetrability, is not influenced by weather, illumination and atmospheric conditions, and can realize all-day, all-weather and large-scale observation. Therefore, the SAR image has unique advantages as an important data source for remote sensing change detection, and a change detection method based on the SAR image is attracting a great deal of attention of researchers. However, due to the special imaging mechanism of the SAR, such as speckle noise, special geometric features, boundary blurring and other complex factors, difficulties and challenges are brought to SAR image change detection.
At present, remote sensing change detection can be divided into a supervision method and an unsupervised method, wherein the supervision method needs a large amount of labeling samples as priori knowledge to train a model, SAR images are difficult to interpret, and a large amount of labeling samples are difficult and expensive to obtain. Therefore, an unsupervised SAR variation detection method is favored by researchers.
In early stages of SAR variation detection development, most of researches are pixel-based methods, namely taking pixels as research units, and generally comprise three steps of image preprocessing, difference map generation and difference map analysis, wherein the methods are simple to operate, but have poor robustness to speckle noise and difficult modeling of spatial context information, and can generate detection results accompanied by a plurality of noise, holes and saw tooth boundaries. In order to solve the problems, object-based change detection methods are vigorously developed, and the methods divide pixels into image objects, so that object-level change information can be provided, noise and holes can be well smoothed, and the boundary segmentation accuracy is improved. However, in object-based change detection methods, the selection of scale parameters is complex and mechanical, improper setting may result in some important minor changes being missed, while the detection accuracy is limited by the accuracy of the segmentation algorithm.
With the continuous development of computing resources and satellite sensors, a change detection method based on deep learning has attracted a great deal of attention. However, most of the existing methods use pixels as basic research units, and the ability of the model to perceive semantic information is limited. Thus, some researchers have proposed a method of combining deep learning with object-based change detection. However, the method completely cuts the object segmentation and the training of the depth network, the segmentation precision greatly limits the detection precision, and the global optimal solution cannot be obtained.
Disclosure of Invention
The invention aims to: aiming at the problems, the invention aims to provide an SAR image end-to-end change detection method combining super-pixel segmentation and a twin network, which solves the problems that the semantic information modeling is difficult, the noise robustness is poor and the change boundary segmentation precision is low by the pixel-based change detection method, and solves the problem that the global optimal solution is difficult to obtain by the change detection method combining deep learning and object segmentation so as to limit the detection precision.
The technical scheme is as follows: the invention discloses a SAR image end-to-end change detection method combining super-pixel segmentation and a twin network, which comprises the following steps:
step 1, selecting SAR image I of two different-scene time-phase research areas 1 And I 2 And performing pretreatment;
step 2, utilizing the preprocessed image I 1 ' and I 2 ' generating a difference map, and classifying the difference map into variation classes omega by adopting a clustering algorithm c Constant omega u And uncertainty class Ω i Will change omega c And invariant class Ω u As training samples of the variable class and the invariant class, a pseudo tag graph G is obtained, belonging to the uncertain class omega i To be predicted;
step 3, expanding training samples by utilizing a data enhancement technology;
step 4, building a super-pixel segmentation network, and preprocessing the preprocessed image I 1 ' and I 2 ' respectively inputting to two weight sharing super-pixel segmentation networks, outputting pixel-super-pixel soft correlation matrix Q corresponding to different time phases 1 And Q 2 And advanced features including superpixel segmentation informationAnd->
Step 5, constructing a twin network and setting up advanced featuresAnd->Inputting the data into different branches of a twin network, outputting a predictive probability map P, and recording an untrained network formed by the super-pixel segmentation network and the twin network as an initial SAR image change detection network;
and 6, calculating a total loss function according to the predictive probability map P and the pseudo-label map G, reversely transmitting and training an initial image change detection network, ending training until reaching a termination condition, taking the trained network as a target SAR image change detection network, and carrying out end-to-end change detection on SAR images by utilizing the target SAR image change detection network.
Further, the preprocessing in step 1 includes the SAR image I 1 And I 2 Geometric correction, radiation correction and high precision registration are performed.
Optionally, for SAR image I 1 And I 2 Filtering noise reduction processing may also be performed.
Further, in step 2, the preprocessed image I is used 1 ' and I 2 ' generate a difference map and divide the difference map into variation classes Ω c Constant omega u And uncertainty class Ω i Comprising:
generating a difference graph by using a logarithmic ratio operator, wherein the expression is as follows:
wherein I is 1 ' representing the preprocessed image I 1 ，I 2 ' representing the preprocessed image I 2 ；
Dividing the difference graph into three classes, namely a variation class omega by adopting an unsupervised clustering method c Marked 1, unchanged omega u Marked 0, uncertainty class Ω i Marked 255.
Further, the step 3 specifically includes:
double-time-phase SAR image I 1 ' and I 2 ' standardize, and implement data enhancement in synchronization with the pseudo tag map G,including random cropping and random rotation by a multiple of 90 degrees.
Further, the random rotation by a multiple of 90 degrees includes random rotations of 90 °, 180 °, and 270 °.
Further, in step 6, a total loss function is calculated according to the predictive probability map P and the pseudo tag map G, including:
calculating cross entropy loss L between predictive probability map P and pseudo tag map G CE And the Dice loss L Dice Belonging to omega i Is ignored and does not participate in the calculation, the expressions are respectively:
wherein N represents that the pseudo tag graph G belongs to Ω c And omega u I represents the pixel index, ω c And omega u Weights representing variant and invariant classes, G i One-hot encoding representing the ith pixel on pseudo tag map G, P i A softmax output representing the ith pixel on the predictive probability map P;
L Dice (P,G)＝1-Dice
in the formula, dice represents a Dice coefficient, and the expression is:
further, in step 6, a total loss function is calculated according to the predictive probability map P and the pseudo tag map G, and further includes:
computing task-specific reconstruction loss L rec And loss of compactness L cpt Belonging to omega i Is ignored and does not participate in the calculation, the expressions are respectively:
L rec (G,Q)＝L CE (G,G*)＝L CE (G,QQ T G)
wherein, Q represents the pixel-super pixel soft correlation matrix Q after column normalization, and the mapping from the pixel characteristic representation to the super pixel characteristic representation can be realized by using Q; q represents a row normalized pixel-a super-pixel soft-association matrix Q with which a reverse mapping of the super-pixel feature representation to the pixel feature representation can be achieved; mapping G to a superpixel feature representation using QI.e.T represents the matrix transpose, reuse Q will +.>Mapping back to pixel characteristic representation yields G, i.e +.>Calculating the cross entropy loss between G and G, namely reconstructing loss L rec ；
In the formula, I 2 Represents L 2 Norms, I xy The position characteristics of the input image are represented,obtained by the following steps: first, I is determined by Q xy Mapping to a super-pixel feature representation to obtain S xy The expression is:
S xy ＝Q T I xy
then, the absolute index of the super pixel where the pixel is located is given by hard associationThe expression is:
in the method, in the process of the invention,representation->The value of the ith pixel, j, represents the superpixel index, H i Representing the super-pixel index to which the i-th pixel hard-association belongs, expressed as:
the total loss function is calculated by combining different phases, and the expression is:
L Joint ＝L CE (P,G)+L Dice (P,G)+λ 1 (L rec (G,Q 1 )+L rec (G,Q 2 ))+λ 2 (L cpt (I xy ,Q 1 )+L cpt (I xy ,Q 2 ))
wherein lambda is 1 And lambda is 2 Representing weight factors, wherein the first term and the second term are main components of a total loss function after the equal sign, and punishing and training the whole model; the third term is used to penalize the super-pixel generation portion, calculates the sum of the different temporal reconstruction losses to facilitate the network to fully mine the multi-temporal information to generate super-pixels, and the fourth term facilitates the generation of compact super-pixels.
The beneficial effects are that: compared with the prior art, the invention has the remarkable advantages that:
1. the invention integrates the super-pixel segmentation network and the twin network under a unified framework to perform end-to-end training, thereby obtaining a global optimal solution; meanwhile, the addition of the super-pixel segmentation information is a successful practice of integrating priori knowledge into deep learning;
2. the super-pixel generated by the invention can be adaptively adjusted in the training process, and multi-time phase information is processed by utilizing different branches of the twin network, so that the consistency of super-pixel segmentation of a non-changed area of multi-time phase data is ensured, and the segmentation of a changed area is closer to a real change boundary;
3. the smooth effect brought by the super-pixel segmentation in the invention can better suppress the influence of speckle noise in SAR images, the high-quality super-pixel segmentation improves the segmentation precision of the change boundary and the detection effect of the small-area change region, and the acquisition of the global optimal solution better balances the noise smoothing and detail reservation;
4. the SAR change detection method is unsupervised, can accept SAR change detection images with different sizes to be input into a model, and finally obtains a high-precision change map; furthermore, the invention is easily extended to the task of variation detection of more complex data such as hyperspectral, fully polarized SAR data, and related tasks involving complex time series image processing.
Drawings
FIG. 1 is a flow chart of an unsupervised change detection design in the present embodiment;
fig. 2 is a diagram of a SAR variation detection network combining super-pixel segmentation and a twin network in the present embodiment;
FIG. 3 is a block diagram of a superpixel sampling network SSN in an embodiment;
FIG. 4 is a block diagram of a twinning change detection network in an embodiment;
FIG. 5 is a change detection result applied to a dual phase Ottawa SAR flood change detection dataset;
fig. 6 is a change detection result applied to a double-phase Sulzberger SAR glacier change detection dataset.
Detailed Description
In order to make the objects, technical solutions and advantages of the present application more apparent, the present application will be further described in detail with reference to the accompanying drawings and examples.
Fig. 1 is a flow chart of an unsupervised change detection in the present embodiment, and fig. 2 is a block diagram of a SAR image end-to-end change detection network of a combined super-pixel segmentation and twinning network in the present embodiment. The SAR image end-to-end variation detection method combining super-pixel segmentation and twin networks specifically comprises the following steps:
step 1, selecting SAR image I of two different simultaneous research areas 1 And I 2 And pre-treatment is performed.
Specifically, a research area SAR image I obtained by two different scenes at the same time is selected according to requirements 1 And I 2 Performing radiation correction and geometric correction by adopting processing software such as ENVI, gamma or NEST; performing high-precision registration on the double-phase SAR image by using an intensity cross-correlation method according to a coarse-to-fine two-step registration strategy to ensure that the registration precision reaches a sub-pixel level, and marking the preprocessed double-phase SAR image as I 1 ' and I 2 '。
Step 2, utilizing the preprocessed image I 1 ' and I 2 ' generating a difference map and classifying the difference map into variation classes omega by using a clustering algorithm c Constant omega u And uncertainty class Ω i Will change omega c And invariant class Ω u Respectively used as training samples of a variable class and a constant class to obtain a pseudo tag graph G which belongs to an uncertain class omega i To be predicted.
Specifically, a log-ratio (log-ratio) operator is used to generate a disparity map LR, expressed as:
wherein I is 1 ' representing the preprocessed image I 1 ，I 2 ' representing the preprocessed image I 2 ；
Dividing the difference map into three classes, namely a variation class omega by adopting hierarchical fuzzy C clustering c Marked 1, unchanged omega u Marked 0, uncertainty class Ω i Marked 255.
And 3, expanding the training sample by utilizing a data enhancement technology.
Specifically, the dual-temporal SAR image I 1 ' and I 2 ' normalization and data enhancement is performed in synchronization with the pseudo tag map G, including random clipping and random rotation by a multiple of 90 degrees, including random rotations of 90 °, 180 °, and 270 °.
Step 4, building a super-pixel segmentation network, and preprocessing the preprocessed image I 1 ' and I 2 ' respectively inputting to two weight sharing super-pixel segmentation networks, outputting pixel-super-pixel soft correlation matrix Q corresponding to different time phases 1 And Q 2 And advanced features including superpixel segmentation informationAnd->
Specifically, in this embodiment, a superpixel segmentation network (Superpixel Sampling Network, SSN) is introduced as a micro superpixel segmentation network, the network structure diagram is shown in fig. 3, where the feature extractor part may also select other networks to replace, the Soft clustering method may select Soft K-means clustering, and the iteration number is set to 10; double-time-phase SAR image I 1 ' and I 2 ' respectively input into two weight sharing SSNs, and output a pixel-super pixel soft correlation matrix Q corresponding to different time phases 1 ,Q 2 ∈R n×m And advanced features generated specifically for superpixel segmentation Where n represents the number of pixels, m represents the number of superpixels, and k represents the dimension of the SSN output advanced feature, which can be set manually.
Soft correlation matrix Q between ith pixel and jth superpixel ij Defined by a gaussian radial distance function, the expression is:
wherein D represents a distance operator, t represents the t-th iteration, K.epsilon.R n×c Representing pixel characteristics, S.epsilon.R m×c Representing a super-pixel feature representation, c representing the number of feature channels. Q εR in this embodiment n×m Is of the meter(s)Only 9 super-pixels around the pixel are considered to improve the calculation efficiency, i.e., m=9; super pixel centerCalculated by the following formula:
column normalized Q t Denoted as Q t And omitting the superscript t, and rewriting the above formula to the following formula, the mapping of the pixel feature representation to the super-pixel feature representation can be achieved by the following formula:
S＝Q T K
where T represents the matrix transpose. Representing the row normalized Q as Q, the inverse mapping of the super-pixel feature representation to the pixel feature representation may be achieved by:
K＝QS。
step 5, constructing a twin network and setting up advanced featuresAnd->Inputting into different branches of a twin network, and outputting a predictive probability map P; the untrained network formed by the super-pixel segmentation network and the twin network is recorded as an initial SAR image change detection network; the overall network structure is shown in FIG. 2, where T 1 Represents T 1 Time SAR image, T 2 Represents T 2 Time SAR images.
Specifically, the structure of the twin change detection network used in the embodiment is shown in fig. 4, explicit difference guiding information of multi-time phase data is added in the network, multi-scale information is fused by using jump connection, and meanwhile, each module of the network uses residual connection to relieve gradient disappearance, and the twin network can be replaced by other twin network models. Will be advanced featuresAndinto different branches of the twin network in fig. 4 (t in the figure 1 ，t 2 Corresponding to the input of different phases), the output is the predictive probability map P of the input SAR image pair.
Step 6, calculating a total loss function according to the predictive probability map P and the pseudo tag map G, wherein the total loss function belongs to omega i Pixels of (labeled "255") are ignored and do not participate in the calculation; and (3) training the initial image change detection network in a back propagation way, finishing training until a termination condition is reached, taking the trained network as a target SAR image change detection network, and detecting the end-to-end change of the SAR image pair by using the target SAR image change detection network.
Specifically, first, the cross entropy loss L between the predictive probability map P and the pseudo tag map G is calculated CE And the Dice loss L Dice Belonging to omega i Is ignored and does not participate in the calculation, the expressions are:
wherein N represents that the pseudo tag graph G belongs to Ω c And omega u I represents the pixel index, ω c And omega u Weights representing variant and invariant classes, G i One-hot encoding representing the ith pixel on pseudo tag map G, P i A softmax output representing the ith pixel on the predictive probability map P;
L Dice (P,G)＝1-Dice
in the formula, dice represents a Dice coefficient, and the expression is:
then, calculate the reconstruction loss L of the specific task rec And loss of compactness L cpt Belonging to omega i Is ignored and does not participate in the calculation, the expressions are:
L rec (G,Q)＝L CE (G,G*)＝L CE (G,QQ T G)
wherein, Q represents the pixel-super pixel soft correlation matrix Q after column normalization, and the mapping from the pixel characteristic representation to the super pixel characteristic representation can be realized by using Q; q represents a pixel-super pixel soft correlation matrix Q of line normalization, and reverse mapping from super pixel characteristic representation to pixel characteristic representation can be realized by using Q; mapping G to a superpixel feature representation using QI.e.T represents the matrix transpose, reuse Q will +.>Mapping back to pixel characteristic representation yields G, i.e +.>Calculating the cross entropy loss between G and G, namely reconstructing loss L rec ；
In the formula, I 2 Represents L 2 Norms, I xy The position characteristics of the input image are represented,obtained by the following steps: first, I is determined by Q xy Mapping to a super-pixel feature representation to obtain S xy The expression is:
S xy ＝Q T I xy
then, the absolute index of the super-pixel is given to the pixel by hard associationObtainingThe expression is:
in the method, in the process of the invention,representation->The value of the ith pixel, j, represents the superpixel index, H i Representing the super-pixel index to which the i-th pixel hard-association belongs, expressed as:
finally, the total loss function is calculated by integrating different phases, and the expression is as follows:
L Joint ＝L CE (P,G)+L Dice (P,G)+λ 1 (L rec (G,Q 1 )+L rec (G,Q 2 ))+λ 2 (L cpt (I xy ,Q 1 )+L cpt (I xy ,Q 2 ))
wherein lambda is 1 And lambda is 2 Representing weight factors, wherein the first term and the second term are main components of a total loss function after the equal sign, and punishing and training the whole model; the third term is used to penalize the super-pixel generation portion, calculates the sum of the different temporal reconstruction losses to facilitate the network to fully mine the multi-temporal information to generate super-pixels, and the fourth term facilitates the generation of compact super-pixels.
The effect of the invention can be further verified by the following experiments:
(1) Experimental environment
Pytorch1.8, graphic card NVIDIA Quadro RTX 8000GPU, system Ubuntu18.04.3;
(2) Experimental details
Performing a change detection on the Ottawa dataset and the Sulzberger dataset using the twin change detection network FC-sim-diff alone; by adopting the method, namely, a SAR variation detection model combining a super-pixel segmentation network and a twin network is constructed by connecting SSN and FC-Siam-diff in series, the SAR variation detection model is named as SSN-Siam-diff and is used for performing variation detection on two data sets, the comparison experimental result is shown in figure 5, figure 5 (a) is first time phase SAR data of Ottawa data set, figure 5 (b) is second time phase SAR data, figure 5 (c) is ground surface variation true value, figure 5 (d) is variation detection result of FC-Siam-diff, and figure 5 (e) is SSN-Siam-diff which is the experimental result of the method. Fig. 6 (a) shows first phase SAR data of the Sulzberger dataset, fig. 6 (b) shows second phase SAR data, fig. 6 (c) shows ground variation truth value, fig. 6 (d) shows variation detection result of FC-sialm-diff, and fig. 6 (e) shows SSN-sialm-diff, which is the experimental result of the method of the present invention.
(3) Evaluation of precision
For quantitative evaluation of the effects of the present invention, the present invention selects Overall Accuracy (OA), precision (Pre), recall (Recall), F1 score (F1), and Kappa coefficient as evaluation indexes, all indexes being percentages in table 1.
Table 1 comparison of the method of the present invention with the results of twin network variation detection for non-tandem superpixel split networks
(4) Analysis of experimental results
The experimental results of fig. 5, fig. 6 and table 1 show that the change detection result obtained by the method provided by the invention is closer to the ground surface true value, the detail keeping and boundary segmentation effects are better, the false alarm is greatly reduced, the noise immunity is improved, meanwhile, the small-area change area can be effectively detected, and the quantitative evaluation index is superior in absolute advantage. Therefore, the method provided by the invention well balances noise smoothing and detail reservation, is very friendly to SAR data with scarce samples, and remarkably improves SAR image change detection effect.
Claims (6)
1. The SAR image end-to-end change detection method combining super-pixel segmentation and twin networks is characterized by comprising the following steps of:
step 1, selecting SAR image I of two different simultaneous research areas 1 And I 2 And performing pretreatment;
step 2, utilizing the preprocessed image I 1 ' and I 2 ' generating a difference map and classifying the difference map into variation classes omega by using a clustering algorithm c Constant omega u And uncertainty class Ω i Will change omega c And invariant class Ω u Respectively used as training samples of a variable class and a constant class to obtain a pseudo tag graph G which belongs to an uncertain class omega i To be predicted;
step 3, expanding training samples by utilizing a data enhancement technology;
step 4, building a super-pixel segmentation network, and preprocessing the preprocessed image I 1 ' and I 2 ' respectively inputting to two weight sharing super-pixel segmentation networks, outputting pixel-super-pixel soft correlation matrix Q corresponding to different time phases 1 And Q 2 And advanced features including superpixel segmentation informationAnd->
Step 5, constructing a twin network and setting up advanced featuresAnd->Inputting into different branches of a twin network, and outputting a predictive probability map P; the untrained network formed by the super-pixel segmentation network and the twin network is recorded as an initial SAR image change detection networkComplexing;
and 6, calculating a total loss function according to the predictive probability map P and the pseudo-label map G, reversely transmitting and training an initial SAR image change detection network, ending training until reaching a termination condition, taking the trained network as a target SAR image change detection network, and carrying out end-to-end change detection on SAR images by utilizing the target SAR image change detection network.
2. The SAR image end-to-end variation detection method according to claim 1, wherein the preprocessing of step 1 comprises the steps of 1 And I 2 Geometric correction, radiation correction and high precision registration are performed.
3. The SAR image end-to-end variation detection method according to claim 1 or 2, wherein in step 2, the preprocessed image I is used 1 ' and I 2 ' generate a difference map and divide the difference map into variation classes Ω c Constant omega u And uncertainty class Ω i Comprising:
generating a difference graph by using a logarithmic ratio operator, wherein the expression is as follows:
wherein I is 1 ' representing the preprocessed image I 1 ，I 2 ' representing the preprocessed image I 2 Log represents natural logarithm;
dividing the difference graph into three classes, namely a variation class omega by adopting an unsupervised clustering method c Marked 1, unchanged omega u Marked 0, uncertainty class Ω i Marked 255.
4. The SAR image end-to-end variation detection method according to claim 3, wherein step 3 specifically comprises:
double-time-phase SAR image I 1 ' and I 2 ' normalization and pseudo-markingSignature G synchronously implements data enhancement, including random clipping and random rotation by a multiple of 90 degrees.
5. The SAR image end-to-end variation detection method according to claim 4, wherein in step 6, the total loss function is calculated according to the predictive probability map P and the pseudo tag map G, comprising:
calculating cross entropy loss L between predictive probability map P and pseudo tag map G CE And the Dice loss L Dice Belonging to omega i Is ignored and does not participate in the calculation, the expressions are respectively:
wherein N represents that the pseudo tag graph G belongs to Ω c And omega u I represents the pixel index, ω c And omega u Weights representing variant and invariant classes, G i One-hot encoding representing the ith pixel on pseudo tag map G, P i A softmax output representing the ith pixel on the predictive probability map P;
L Dice (P,G)＝1-Dice
in the formula, dice represents a Dice coefficient, and the expression is:
6. the SAR image end-to-end variation detection method according to claim 5, wherein in step 6, a total loss function is calculated according to the predictive probability map P and the pseudo tag map G, further comprising:
computing reconstruction loss L for a particular task rec And loss of compactness L cpt Belonging to omega i Is ignored and does not participate in the calculation, the expressions are respectively:
L rec (G,Q)＝L CE (G,G*)＝L CE (G,QQ T G)
wherein, Q represents the pixel-super pixel soft correlation matrix Q after column normalization, and the mapping from the pixel characteristic representation to the super pixel characteristic representation can be realized by using Q; q represents a pixel-super pixel soft correlation matrix Q of line normalization, and reverse mapping from super pixel characteristic representation to pixel characteristic representation can be realized by using Q; mapping G to a superpixel feature representation using QI.e. < ->T represents the matrix transpose, reuse Q will +.>Mapping back to pixel characteristic representation yields G, i.e +.>Calculating cross entropy loss between G and G, namely reconstruction loss;
in the formula, I 2 Represents L 2 Norms, I xy The position characteristics of the input image are represented,obtained by the following steps: first, I is determined by Q xy Mapping to a super-pixel feature representation to obtain S xy The expression is:
S xy ＝Q T I xy
then, the absolute index of the super pixel where the pixel is located is given by hard associationThe expression is:
in the method, in the process of the invention,representation->The value of the ith pixel, j, represents the superpixel index, H i The super-pixel index to which the i-th pixel hard association belongs is expressed as:
in which Q ij Representing soft association between the ith pixel and the jth superpixel, m representing the number of superpixels;
calculate the total loss function L by integrating different phases Joint The expression is:
L Joint ＝L CE (P,G)+L Dice (P,G)+λ 1 (L rec (G,Q 1 )+L rec (G,Q 2 ))+λ 2 (L cpt (I xy ,Q 1 )+L cpt (I xy ,Q 2 ))
wherein lambda is 1 And lambda is 2 Representing weight factors, wherein the first term and the second term are main components of a total loss function after the equal sign, and punishing and training the whole model; the third term is used to penalize the super-pixel generation portion, calculates the sum of the different temporal reconstruction losses to facilitate the network to fully mine the multi-temporal information to generate super-pixels, and the fourth term facilitates the generation of compact super-pixels.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310517307.8A CN116563308A (en) | 2023-05-09 | 2023-05-09 | SAR image end-to-end change detection method combining super-pixel segmentation and twin network |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310517307.8A CN116563308A (en) | 2023-05-09 | 2023-05-09 | SAR image end-to-end change detection method combining super-pixel segmentation and twin network |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116563308A true CN116563308A (en) | 2023-08-08 |
Family
ID=87499569
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310517307.8A Pending CN116563308A (en) | 2023-05-09 | 2023-05-09 | SAR image end-to-end change detection method combining super-pixel segmentation and twin network |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116563308A (en) |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US5635982A (en) * | 1994-06-27 | 1997-06-03 | Zhang; Hong J. | System for automatic video segmentation and key frame extraction for video sequences having both sharp and gradual transitions |
| CN106780485A (en) * | 2017-01-12 | 2017-05-31 | 西安电子科技大学 | SAR image change detection based on super-pixel segmentation and feature learning |
| CN108573276A (en) * | 2018-03-12 | 2018-09-25 | 浙江大学 | A kind of change detecting method based on high-resolution remote sensing image |
| US20190035431A1 (en) * | 2017-07-28 | 2019-01-31 | Adobe Systems Incorporated | Apparatus, systems, and methods for integrating digital media content |
| CN115173927A (en) * | 2022-07-11 | 2022-10-11 | 南京科伊星信息科技有限公司 | Satellite communication simulation terminal system for duplex transceiving of radio frequency signals in UHF frequency band |
-
2023
- 2023-05-09 CN CN202310517307.8A patent/CN116563308A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US5635982A (en) * | 1994-06-27 | 1997-06-03 | Zhang; Hong J. | System for automatic video segmentation and key frame extraction for video sequences having both sharp and gradual transitions |
| CN106780485A (en) * | 2017-01-12 | 2017-05-31 | 西安电子科技大学 | SAR image change detection based on super-pixel segmentation and feature learning |
| US20190035431A1 (en) * | 2017-07-28 | 2019-01-31 | Adobe Systems Incorporated | Apparatus, systems, and methods for integrating digital media content |
| CN108573276A (en) * | 2018-03-12 | 2018-09-25 | 浙江大学 | A kind of change detecting method based on high-resolution remote sensing image |
| CN115173927A (en) * | 2022-07-11 | 2022-10-11 | 南京科伊星信息科技有限公司 | Satellite communication simulation terminal system for duplex transceiving of radio frequency signals in UHF frequency band |
Non-Patent Citations (3)
| Title |
|---|
| HONGYAN ZHANG: "ESCNet: An End-to-End Superpixel-Enhanced Change Detection Network for Very-High-Resolution Remote Sensing Images", 《IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS》, pages 29 - 40 * |
| LINXIA JI: "A Novel End-to-End Unsupervised Change Detection Method with Self-Adaptive Superpixel Segmentation for SAR Images", 《MDPI》, 23 March 2023 (2023-03-23), pages 1 - 24 * |
| 王向阳;陈亮;王倩;王雪冰;杨红颖;: "基于TWSVM超像素分类的彩色图像分割算法", 辽宁师范大学学报(自然科学版), no. 01 * |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Liu et al. | Multi-scale attention integrated hierarchical networks for high-resolution building footprint extraction | |
| CN113298815A (en) | Semi-supervised remote sensing image semantic segmentation method and device and computer equipment | |
| CN111640121A (en) | Rectum CT image tumor segmentation method based on improved U-net | |
| WO2022141718A1 (en) | Method and system for assisting point cloud-based object detection | |
| Liu et al. | CT-UNet: Context-transfer-UNet for building segmentation in remote sensing images | |
| Khan et al. | An encoder–decoder deep learning framework for building footprints extraction from aerial imagery | |
| CN113988147A (en) | Multi-label classification method and device for remote sensing image scene based on graph network, and multi-label retrieval method and device | |
| Wang et al. | Double U-Net (W-Net): A change detection network with two heads for remote sensing imagery | |
| Wang et al. | STCD: efficient Siamese transformers-based change detection method for remote sensing images | |
| Zhong et al. | Lake water body extraction of optical remote sensing images based on semantic segmentation | |
| Li et al. | MF-SRCDNet: Multi-feature fusion super-resolution building change detection framework for multi-sensor high-resolution remote sensing imagery | |
| CN117788296B (en) | Infrared remote sensing image super-resolution reconstruction method based on heterogeneous combined depth network | |
| Chen et al. | Spatiotemporal fusion for spectral remote sensing: A statistical analysis and review | |
| Zhou et al. | A superior image inpainting scheme using Transformer-based self-supervised attention GAN model | |
| Liu et al. | SACTNet: Spatial attention context transformation network for cloud removal | |
| CN113781532A (en) | SAR satellite image and optical image automatic matching retrieval method | |
| Li et al. | Pillar‐based 3D object detection from point cloud with multiattention mechanism | |
| CN113408540A (en) | Synthetic aperture radar image overlap area extraction method and storage medium | |
| Wu et al. | Depth estimation from infrared video using local-feature-flow neural network | |
| Zheng et al. | Dcu-net: Self-supervised monocular depth estimation based on densely connected u-shaped convolutional neural networks | |
| CN117274627A (en) | Multi-temporal snow remote sensing image matching method and system based on image conversion | |
| CN111696167A (en) | Single image super-resolution reconstruction method guided by self-example learning | |
| CN116563308A (en) | SAR image end-to-end change detection method combining super-pixel segmentation and twin network | |
| Feng et al. | Improved deep fully convolutional network with superpixel-based conditional random fields for building extraction | |
| CN115984093A (en) | Depth estimation method based on infrared image, electronic device and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |