CN112330562A - Heterogeneous remote sensing image transformation method and system - Google Patents
Heterogeneous remote sensing image transformation method and system Download PDFInfo
- Publication number
- CN112330562A CN112330562A CN202011236018.3A CN202011236018A CN112330562A CN 112330562 A CN112330562 A CN 112330562A CN 202011236018 A CN202011236018 A CN 202011236018A CN 112330562 A CN112330562 A CN 112330562A
- Authority
- CN
- China
- Prior art keywords
- image
- visible light
- generator
- sar
- noise reduction
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000011426 transformation method Methods 0.000 title claims abstract description 22
- 230000009467 reduction Effects 0.000 claims abstract description 94
- 230000009466 transformation Effects 0.000 claims abstract description 47
- 238000012549 training Methods 0.000 claims description 69
- 239000010410 layer Substances 0.000 claims description 34
- 230000014759 maintenance of location Effects 0.000 claims description 11
- 208000037170 Delayed Emergence from Anesthesia Diseases 0.000 claims description 10
- 238000006243 chemical reaction Methods 0.000 claims description 8
- 239000002356 single layer Substances 0.000 claims description 7
- 230000006870 function Effects 0.000 description 22
- 238000000034 method Methods 0.000 description 20
- 238000013507 mapping Methods 0.000 description 12
- 238000010586 diagram Methods 0.000 description 7
- 230000003595 spectral effect Effects 0.000 description 6
- 230000000694 effects Effects 0.000 description 5
- 238000013135 deep learning Methods 0.000 description 3
- 238000013461 design Methods 0.000 description 3
- 239000011159 matrix material Substances 0.000 description 3
- 238000012544 monitoring process Methods 0.000 description 3
- 230000003042 antagnostic effect Effects 0.000 description 2
- 238000004364 calculation method Methods 0.000 description 2
- 238000000605 extraction Methods 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 230000008569 process Effects 0.000 description 2
- 238000012545 processing Methods 0.000 description 2
- 230000000750 progressive effect Effects 0.000 description 2
- 238000011084 recovery Methods 0.000 description 2
- 238000001228 spectrum Methods 0.000 description 2
- PXFBZOLANLWPMH-UHFFFAOYSA-N 16-Epiaffinine Natural products C1C(C2=CC=CC=C2N2)=C2C(=O)CC2C(=CC)CN(C)C1C2CO PXFBZOLANLWPMH-UHFFFAOYSA-N 0.000 description 1
- 241000209202 Bromus secalinus Species 0.000 description 1
- 230000003044 adaptive effect Effects 0.000 description 1
- 239000000654 additive Substances 0.000 description 1
- 230000000996 additive effect Effects 0.000 description 1
- 238000013459 approach Methods 0.000 description 1
- 238000013528 artificial neural network Methods 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000004040 coloring Methods 0.000 description 1
- 230000008878 coupling Effects 0.000 description 1
- 238000010168 coupling process Methods 0.000 description 1
- 238000005859 coupling reaction Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000005315 distribution function Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000003631 expected effect Effects 0.000 description 1
- 238000002474 experimental method Methods 0.000 description 1
- 238000011835 investigation Methods 0.000 description 1
- 238000013508 migration Methods 0.000 description 1
- 230000005012 migration Effects 0.000 description 1
- 230000008447 perception Effects 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
- 238000000844 transformation Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/70—Denoising; Smoothing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/29—Graphical models, e.g. Bayesian networks
- G06F18/295—Markov models or related models, e.g. semi-Markov models; Markov random fields; Networks embedding Markov models
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/40—Analysis of texture
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/90—Determination of colour characteristics
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Physics & Mathematics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Data Mining & Analysis (AREA)
- Bioinformatics & Computational Biology (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Evolutionary Biology (AREA)
- Evolutionary Computation (AREA)
- Artificial Intelligence (AREA)
- General Engineering & Computer Science (AREA)
- Life Sciences & Earth Sciences (AREA)
- Image Analysis (AREA)
Abstract
The invention relates to a heterogeneous remote sensing image transformation method and a system, wherein a visible light gray image is set as a middle state from an SAR image to a visible light color image, the SAR image is generated into the visible light gray image through an SAR image noise reduction generator and is identified through a first identifier, the visible light gray image is generated into the visible light color image through the visible light image generator and is identified through a second identifier, the distortion rate from the SAR image to the visible light color image is reduced, and the quality and the efficiency of remote sensing image transformation are improved.
Description
Technical Field
The invention relates to the technical field of image transformation, in particular to a heterogeneous remote sensing image transformation method and system.
Background
With the continuous development of information perception technology, the processing and application of remote sensing images gradually change from single mode to multi-mode. The reason is that the image acquired by a single sensor cannot meet the requirements of certain specific scenes, and under the condition that the performance of the sensor cannot be effectively improved temporarily, the heterogeneous images acquired by multiple types of sensors are combined into a feasible way.
Taking optical and Synthetic Aperture Radar (SAR) images as an example, optical images are sensitive to weather, especially cloud layers, which severely limits their use for observation and monitoring of ground targets. A possible alternative is to correlate images acquired by different modality sensors at the same time, using a weather insensitive SAR image as a compensation. In this respect, deep learning based approaches provide a good solution. Under heterogeneous image-driven coupling time series, deep learning based methods can learn the mapping from SAR images to visible light images and use it for the transformation of known SAR images to unknown available light images. The obtained result has the spectrum information of the visible light image and can also obtain the all-weather observation advantages of the SAR image all day long.
Heterogeneous transformations of remote sensing images can be classified as an image style transformation problem, and can be further classified into feature-based methods and methods based on generation of a countermeasure network GAN in terms of image style transformation. The method based on the characteristics adopts the idea that the style and semantic content of an original image are respectively extracted by utilizing the characteristic extraction capability of an artificial neural network, the local affine transformation from input to output is limited in a color space by considering the distortion of certain style and content, the constraint is expressed as a self-defined fully differentiable energy item, and then the combined reconstruction is carried out to generate a new style image. Another idea of style transformation is based on a method of generating a countermeasure network GAN. The idea of the method is to adopt a conditional generation countermeasure network cGAN to perform conditional generation and transformation on the image, so that the generated image has the semantic information of an original image domain and the style characteristics of a target image domain.
For the two existing methods, although the characteristic-based method has good effect on artistic image style transformation, when a new style photo is generated, the loss function needs to be derived for many times, so that the parameter calculation amount is large when the method is used for heterogeneous transformation of remote sensing images, and the processing instantaneity is poor. Compared with the former method, the method for generating the anti-network cGAN based on the condition has the advantages of low calculation cost, higher flexibility, ideal conversion result between general images with clear structures and symmetrical information, and serious texture loss and distortion when the method is used for heterogeneous conversion of remote sensing images, which influences final decision and judgment.
At present, the heterogeneous remote sensing image transformation for generating the countermeasure network based on the conditions has the following technical difficulties: firstly, due to the principle difference of the SAR image and the visible light image acquisition sensor, the SAR image and the visible light image acquisition sensor have the characteristic of information asymmetry, and experiments are carried out by utilizing the existing image transformation models such as pix2pix, Cycle-GAN and the like, so that the RGB visible light image generated by the SAR image has serious distortion phenomena such as characteristic loss and the like, so that the interpretability and the reliability of the final transformation result are low, and the subsequent research and decision are influenced. Therefore, the influence of multiplicative gamma noise pollution on the SAR image on image cross-modal transformation needs to be reduced, the retention and recovery capability of the model on the main characteristics of the SAR image is improved, and the finally transformed visible light image has the semantic information of the original SAR image and certain spectral information, so that more reliable image information is provided for geological disaster monitoring, site investigation, sea area monitoring and the like; and secondly, the information of the visible light image exists in a plurality of channels, the SAR image is a gray level image, the information of the visible light image exists in one channel, channel difference and semantic gap exist between the visible light image and the SAR image, and the problems of inaccurate mapping of spectral information and semantic information and the like exist when a generator and an identifier are directly adopted to generate an anti-network for heterogeneous transformation.
In terms of generating heterogeneous image transformation of a countermeasure network based on conditions, no transformation model aiming at SAR image to visible light image (RGB) can solve the problems well at present. In the aspects of accuracy and real-time performance of heterogeneous transformation, the existing methods cannot meet the requirements of complication and diversification of remote sensing images, so that the problem needs to be solved urgently.
Disclosure of Invention
Based on the above, the invention aims to provide a heterogeneous remote sensing image transformation method and system, which improve the quality and efficiency of remote sensing image transformation.
In order to achieve the purpose, the invention provides the following scheme:
a heterogeneous remote sensing image transformation method, comprising:
denoising the plurality of SAR images to obtain a visible light gray level image of each SAR image;
obtaining a visible light color image of each SAR image;
the SAR image is used as the input of an SAR image noise reduction generator, the visible light gray level image is used as the output of the SAR image noise reduction generator, a first gradient discriminator is used for discriminating the visible light gray level image output by the SAR image noise reduction generator, and the SAR image noise reduction generator and the first gradient discriminator are trained;
training the visible light image generator and a second gradient discriminator by taking the visible light gray scale image as the input of the visible light image generator, taking the visible light color image as the output of the visible light image generator and discriminating the visible light color image output by the visible light image generator by using the second gradient discriminator;
inputting the SAR image to be transformed into the trained SAR image noise reduction generator to obtain a visible light gray image to be transformed, and inputting the visible light gray image to be transformed into the trained visible light image generator to obtain a transformed visible light color image.
Optionally, the SAR image noise reduction generator is of an improved U-Net network structure, an input layer and an output layer of the SAR image noise reduction generator are both single channels, and each adjacent single-layer convolutional layer in the SAR image noise reduction generator is connected by adopting a residual block and a jump connection.
Optionally, the visible light image generator is of a modified U-Net network structure, and the number of output layer channels of the visible light image generator is 3.
Optionally, the first gradient discriminator and the second gradient discriminator are both markov discriminators.
Optionally, the first training loss function for training the SAR image noise reduction generator and the first gradient discriminator is:
wherein,
representing conditional generation-countering network loss of the SAR image noise reduction generator from inputting the SAR image to outputting a visible light grayscale image;
representing a reconstruction loss of the SAR image noise reduction generator from inputting the SAR image to outputting a visible light grayscale image,
representing a texture retention loss of the SAR image noise reduction generator from inputting the SAR image to outputting a visible light grayscale image;
representing the SAR image noise reduction generator, x(i)Representing the set of SAR images, y(i)Representing a set of visible light grayscale images;
the second training loss function training the visible light image generator and the second gradient discriminator is:
wherein,
representing conditional generation-countering network loss of the visible light image generator from inputting the visible light grayscale image to outputting a visible light color image;
representing a reconstruction loss of the visible light image generator from inputting the visible light gray scale image to outputting a visible light color image;
representing a texture-preserving loss of the visible light image generator from inputting the visible light grayscale image to outputting a visible light color image;
representing a visible light image generator, z(i)Representing a set of visible color images;
α1denotes a first weight parameter, β1Representing a second weight parameter, α2Denotes a third weight parameter, β2Representing a fourth weight parameter;
stopping training the SAR image noise reduction generator, the first gradient discriminator, the visible light image generator and the second gradient discriminator when the sum of the first training loss function and the second training loss function is less than a set threshold.
The invention also discloses a heterogeneous remote sensing image transformation system, which comprises:
and the visible light gray level image acquisition module is used for denoising the SAR images to obtain the visible light gray level image of each SAR image.
The visible light color image acquisition module is used for acquiring a visible light color image of each SAR image;
the first training module is used for taking the SAR image as the input of the SAR image noise reduction generator, taking the visible light gray image as the output of the SAR image noise reduction generator, identifying the visible light gray image output by the SAR image noise reduction generator by using a first gradient identifier, and training the SAR image noise reduction generator and the first gradient identifier;
a second training module, which takes the visible light gray scale image as the input of a visible light image generator, takes the visible light color image as the output of the visible light image generator, and uses a second gradient discriminator to discriminate the visible light color image output by the visible light image generator, so as to train the visible light image generator and the second gradient discriminator;
and the SAR image conversion module is used for inputting the SAR image to be converted into the trained SAR image noise reduction generator to obtain the visible light gray image to be converted, and inputting the visible light gray image to be converted into the trained visible light image generator to obtain the converted visible light color image.
Optionally, the SAR image noise reduction generator is of an improved U-Net network structure, an input layer and an output layer of the SAR image noise reduction generator are both single channels, and each adjacent single-layer convolutional layer in the SAR image noise reduction generator is connected by adopting a residual block and a jump connection.
Optionally, the visible light image generator is of a modified U-Net network structure, and the number of output layer channels of the visible light image generator is 3.
Optionally, the first gradient discriminator and the second gradient discriminator are both markov discriminators.
Optionally, the first training loss function for training the SAR image noise reduction generator and the first gradient discriminator is:
wherein,
representing conditional generation-countering network loss of the SAR image noise reduction generator from inputting the SAR image to outputting a visible light grayscale image;
representing a reconstruction loss of the SAR image noise reduction generator from inputting the SAR image to outputting a visible light grayscale image,
representing a texture retention loss of the SAR image noise reduction generator from inputting the SAR image to outputting a visible light grayscale image;
representing the SAR image noise reduction generator, x(i)Representing the set of SAR images, y(i)A set of visible light grayscale images is represented,
the second training loss function training the visible light image generator and the second gradient discriminator is:
wherein,
representing conditional generation-countering network loss of the visible light image generator from inputting the visible light grayscale image to outputting a visible light color image;
representing a reconstruction loss of the visible light image generator from inputting the visible light gray scale image to outputting a visible light color image;
representing a texture-preserving loss of the visible light image generator from inputting the visible light grayscale image to outputting a visible light color image;
representing a visible light image generator, z(i)Representing a set of visible color images;
α1denotes a first weight parameter, β1Representing a second weight parameter, α2Denotes a third weight parameter, β2Representing a fourth weight parameter;
stopping training the SAR image noise reduction generator, the first gradient discriminator, the visible light image generator and the second gradient discriminator when the sum of the first training loss function and the second training loss function is less than a set threshold.
According to the specific embodiment provided by the invention, the invention discloses the following technical effects:
the invention discloses a heterogeneous remote sensing image transformation method and a system, wherein a visible light gray image is set as a middle state from an SAR image to a visible light color image, the SAR image is generated into the visible light gray image through an SAR image noise reduction generator, the visible light gray image is identified through a first identifier, the visible light gray image is generated into the visible light color image through the visible light image generator, the visible light color image is identified through a second identifier, the distortion rate from the SAR image to the visible light color image is reduced, and the quality and the efficiency of remote sensing image transformation are improved.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings needed to be used in the embodiments will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings without inventive exercise.
FIG. 1 is a schematic flow chart of a heterogeneous remote sensing image transformation method of the invention;
FIG. 2 is a schematic diagram of the heterogeneous image transformation from SAR images to visible color images of the present invention;
FIG. 3 is a schematic diagram of a network structure of an SAR image noise reduction generator according to the present invention;
FIG. 4 is a schematic diagram of a network structure of a visible light image generator according to the present invention;
FIG. 5 is a diagram of a first authenticator network architecture in accordance with the present invention;
FIG. 6 is a diagram of a second discriminator network according to the invention;
FIG. 7 is a schematic structural diagram of a heterogeneous remote sensing image transformation system according to the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The invention aims to provide a heterogeneous remote sensing image transformation method and a heterogeneous remote sensing image transformation system, which improve the quality and efficiency of remote sensing image transformation.
In order to make the aforementioned objects, features and advantages of the present invention comprehensible, embodiments accompanied with figures are described in further detail below.
Fig. 1 is a schematic flow chart of a heterogeneous remote sensing image transformation method of the present invention, and as shown in fig. 1, the heterogeneous remote sensing image transformation method includes:
step 101: and denoising the plurality of SAR images to obtain a visible light gray level image of each SAR image.
Step 102: and obtaining a visible light color image of each SAR image.
Step 103: and taking the SAR image as the input of an SAR image noise reduction generator, taking the visible light gray image as the output of the SAR image noise reduction generator, identifying the visible light gray image output by the SAR image noise reduction generator by using a first gradient identifier, and training the SAR image noise reduction generator and the first gradient identifier.
Step 104: and training the visible light image generator and a second gradient discriminator by taking the visible light gray scale image as the input of the visible light image generator, taking the visible light color image as the output of the visible light image generator, and discriminating the visible light color image output by the visible light image generator by using the second gradient discriminator.
Step 105: inputting the SAR image to be transformed into the trained SAR image noise reduction generator to obtain a visible light gray image to be transformed, and inputting the visible light gray image to be transformed into the trained visible light image generator to obtain a transformed visible light color image.
The SAR image noise reduction generator is of an improved U-Net network structure, an input layer and an output layer of the SAR image noise reduction generator are single channels, and each adjacent single-layer convolution layer in the SAR image noise reduction generator is connected through a residual block and a jump connection.
The visible light image generator is of an improved U-Net network structure, and the number of output layer channels of the visible light image generator is 3.
The first gradient discriminator and the second gradient discriminator are both Markov discriminators.
The first training loss function for training the SAR image noise reduction generator and the first gradient discriminator is as follows:
wherein,
representing conditional generation-countering network loss of the SAR image noise reduction generator from inputting the SAR image to outputting a visible light grayscale image;
representing a reconstruction loss of the SAR image noise reduction generator from inputting the SAR image to outputting a visible light grayscale image,
representing a texture retention loss of the SAR image noise reduction generator from inputting the SAR image to outputting a visible light grayscale image;
representing the SAR image noise reduction generator, x(i)Representing the set of SAR images, y(i)Representing a set of visible light grayscale images.
The second training loss function training the visible light image generator and the second gradient discriminator is:
wherein,
representing conditional generation-countering network loss of the visible light image generator from inputting the visible light grayscale image to outputting a visible light color image;
representing a reconstruction loss of the visible light image generator from inputting the visible light gray scale image to outputting a visible light color image;
representing a texture-preserving loss of the visible light image generator from inputting the visible light grayscale image to outputting a visible light color image;
to representVisible light image generator, z(i)Representing a collection of visible color images.
α1Denotes a first weight parameter, β1Representing a second weight parameter, α2Denotes a third weight parameter, β2Representing a fourth weight parameter.
Stopping training the SAR image noise reduction generator, the first gradient discriminator, the visible light image generator and the second gradient discriminator when the sum of the first training loss function and the second training loss function is less than a set threshold.
The following is a detailed description of a heterogeneous remote sensing image transformation method of the present invention.
As shown in FIG. 2, for ease of illustration, the following convention is made: let X, Y, Z denote the SAR image domain, the visible light grayscale image domain (intermediate visible light image domain) and the visible light color image domain, x, respectively(i)∈X、y(i)∈Y、z(i)E Z are the dataset samples for the corresponding image domain, respectively (i ═ 1,2, …, N denotes the total number of samples of the dataset). Therefore, the transformation of the SAR image into a visible light grayscale image (intermediate-state visible light image) can be defined as a mapping transformation of the source domain X into the target domain Y
Let x be(i)Is a random sample taken from the SAR image domain X and has a distribution function of
x(i)The random sample mapped to the visible light image domain is y(i). The remote sensing image transformation network provided by the invention is finally realized
Final z(i)Is derived from x by a deep learning network(i)Generating and obtaining:
remote sensing image changeThe switching network comprises two generators and two discriminators which are respectively used for reducing noise of the SAR image
Visible light image generator
Gradient discriminator
(first gradient discriminator) and gradient discriminator
(second gradient discriminator). SAR image noise reduction generator
Short for generator
Visible light image generator
Short for generator
Generator
The SAR image is converted into a visible light gray level image, so that high-frequency details and low-frequency semantic information of the SAR image are recovered, and a semantic gap from the SAR image to the visible light image is reduced; generator
Converting the visible light gray level image into a visible light color image with spectral information, so that the recovered visible light color image has the semantic information of the original SAR image domain and the spectral details of the visible light image; discriminator
The gradient details of the visible light gray scale image in the data set Y are used for identification, so that the generator
Obtaining stronger noise reduction capability of the SAR image, and simultaneously reserving semantic information in the original SAR image; discriminator
And discriminator
Similarly, the gradient information of the visible color image is used for discrimination, thus the generator
The resulting image allows for better recovery of spectral information while preserving detailed information.
As shown in fig. 3, the generator
By adopting an improved U-Net network structure, an input layer and an output layer are single channels, and each adjacent single-layer convolutional layer is connected by a Residual Block and a Skip Connection, so that the learning capability of the generator can be fully enhanced while the network depth is expanded. Meanwhile, instead of converting a multiplicative speckle model into an additive speckle model through input logarithmic conversion, a residual error learning strategy the same as SAR-DRN (SAR-Dual regression networks) is adopted, and a generator
The output layer is Gamma noise extracted from the input image, and the clear visible light gray image can be obtained by subtracting the input image from the output image.
As shown in fig. 4, the generator
And generator
The structure of the network is similar, an improved U-Net network structure is adopted, an input layer is a single Channel, the difference is that the number of output layer channels of the network is 3(Channel is 3), and the input layer is a generator
And generating a visible light gray scale image. In principle, the generator
The learning is the mapping from the SAR image contaminated by Gamma noise to the visible light gray scale image, its essence is the noise reduction of the image, and the generator
Learned is the mapping from the visible light gray-scale image to the visible light color image. Therefore, the progressive cross-mode transformation can obtain a better transformation effect, the expression capability of the model from SAR image reconstruction with less information amount to visible light color image is enhanced by increasing the layer number of the network, and the heterogeneous transformation of the image is realized.
As shown in fig. 5-6, discriminator D1And discriminator D2The Markov judgers are adopted, which are completely composed of convolution layers, and finally, an n-by-n matrix is output. The input of the Markov discriminator is 256 multiplied by 256 to randomly cut a 70 multiplied by 70 image Patch, the output of the Markov discriminator is the mean value of an output matrix (4 multiplied by 4), the final discrimination result is True/False, the True represents that the picture generated by the generator cheats the discriminator, the expected effect is achieved, and the training of the generator is finished; "False" means that the picture generated by the generator has not "spoofed" the discriminator, the generated picture is not as expected, and the generator needs to continue training. Since each output in the output matrix represents a field of view in the original image and corresponds to a slice (patch) of the original image, the GAN constructed in this way may be called patch GAN. In a different aspect, the discriminator
Is for discriminating the generator
Generated image (True/False), and discriminator
Is for discriminating the generator
The resulting image (True/False). In the heterogeneous image transformation process, the migration transformation of the content part and the texture part is included. The content portion is the similarity in content (semantic) between the generated image and the original image, and the texture portion is the similarity in texture between the generated image and the target image. Theory shows that with PatchGAN, texture detail can be maintained.
In the remote sensing image transformation network training stage, a generator is trained firstly
And a discriminator
Generator
Intermediate-state visible light image deception discriminator generated by learning speckle noise removal and super-resolution reconstruction (equal-scale) mapping of SAR image from paired SAR and visible light gray level image
Making it impossible to distinguish between the visible light gray scale image and the generated intermediate state image; and a discriminator
Judging whether the intermediate state image generated by the generator is close to the visible light corresponding to the original input SAR image or notA grayscale image. Then, the generator is fixed
And a discriminator
Is a training generator
And a discriminator
Generator
And learning the spectral information mapping of the visible light image from the paired visible light gray level image and visible light color image to complete the coloring function of the image. During the training process, the generator
And a discriminator
After the balance is achieved, the generator is carried out
And a discriminator
And (4) training.
A testing phase, the input SAR image is processed by a generator
Generating a denoised intermediate visible light gray scale image, and inputting the image to a generator
Via a generator
And generating a visible light color image, and completing the conversion from the SAR image to the visible light image.
Design for loss function
1) cGAN (Conditional generational adaptive networks) loss
Under the premise that paired training data exists, the cGAN is adopted by the method to improve the performance of the generator. By generating a countermeasure network, a generator
Will learn the mapping from X to Y and spoof the discriminator
Generator
Will learn the mapping from Y to Z and spoof the discriminator
The network structure of the discriminator has a function of distinguishing a forged image from an actual image. Thus, the cGAN losses from X to Y and Y to Z are:
in the first half of training, when the discriminator
Learn to make
When maximizing, the generator
Attempt to make
Minimized until both reach an optimal state (nash equilibrium); in the second half of training, when the discriminator
Learn to make
When maximizing, the generator
Attempt to make
Minimized until both reach an optimum (Nash equilibrium),
representing a mathematical expectation, a smaller mathematical expectation is better, and a smaller mathematical expectation is less loss.
2) Loss of reconstruction
Because the invention adopts the improved U-Net structure, the introduced residual block and jump connection increase the network depth, so the L is adopted in the reconstruction loss design1The loss is used for minimizing the distance between the reconstructed image and the original image, and the reconstruction losses of the two training stages are respectively:
3) texture retention loss
As shown in fig. 2, in the optimum conversion state
Output of the network
Should approximate the visible light image y(i). The invention provides gradient-guided texture retention loss on the basis of the traditional reconstruction loss. Assuming that M (-) represents an image gradient operation, the texture retention penalty for the two training phases is:
for the image I, M (-) the operation of extracting the image gradient is calculated as follows:
Ix(x,y)＝I(x+1,y)-I(x-1,y),
Iy(x,y)＝I(x,y+1)-I(x,y-1),
in a specific implementation, the M (-) operation is implemented by convolution with a fixed convolution kernel, and (x, y) represents the pixel coordinates.
Thus, the first half of the model training phase, the generator
And a discriminator
Formation of antagonistic networks GAN1The total training loss is:
the second half of the model training phase, the generator
And a discriminator
Formation of antagonistic networks GAN2The total training loss is:
wherein,
representing conditional generation-countering network loss of the SAR image noise reduction generator from inputting the SAR image to outputting a visible light grayscale image;
representing a reconstruction loss of the SAR image noise reduction generator from inputting the SAR image to outputting a visible light grayscale image,
representing a texture retention loss of the SAR image noise reduction generator from inputting the SAR image to outputting a visible light grayscale image;
representing the SAR image noise reduction generator, x(i)Representing the set of SAR images, y(i)Representing a set of visible light grayscale images.
α1Denotes a first weight parameter, β1Representing a second weight parameter, α2Denotes a third weight parameter, β2Representing a fourth weight parameter.
The technical effect of the heterogeneous remote sensing image transformation method of the invention comprises the following points:
1. a heterogeneous remote sensing image transformation method constructs a weighted target function with reconstruction loss, cGAN loss and texture preserving gradient loss, realizes natural transformation from an SAR image to a visible light image by weighting three constraint conditions of comprehensive semantic information, style characteristics and one-dimensional gradient characteristics, has obvious advantages in overcoming noise and distortion, and has strong adaptability, wide application range and strong robustness.
2. A heterogeneous remote sensing image transformation method is characterized in that a residual block (Residualblock) and a Skip Connection (Skip Connection) are used for improving U-Net and a generator for an SAR image to a visible light gray image and a generator for a visible light gray image to a visible light color image for the first time, so that the network depth and the multi-scale feature extraction capability and the mapping capability are increased, the mapping transformation capability of a single-channel gray remote sensing image to an RGB channel remote sensing image is remarkably improved, and the spectrum mapping distortion is greatly reduced.
3. A heterogeneous remote sensing image transformation method optimally designs a total objective function, and based on a traditional method, a result generated by one-dimensional gradient feature constraint retains the detail features of an original SAR image, and improves the quality of heterogeneous transformation.
4. A heterogeneous remote sensing image transformation method overcomes the distortion phenomenon of SAR image to RGB visible light image transformation by adjusting the final transformation effect of network structure constraint aiming at the characteristic of asymmetric information between an SAR image and a visible light color image.
Fig. 7 is a schematic structural diagram of a heterogeneous remote sensing image transformation system according to the present invention, and as shown in fig. 7, the heterogeneous remote sensing image transformation system includes:
a visible light grayscale image obtaining module 201, configured to denoise multiple SAR images to obtain a visible light grayscale image of each SAR image;
a visible light color image obtaining module 202, configured to obtain a visible light color image of each SAR image;
the first training module 203 is configured to use the SAR image as an input of a SAR image noise reduction generator, use the visible light grayscale image as an output of the SAR image noise reduction generator, and use a first gradient discriminator to discriminate the visible light grayscale image output by the SAR image noise reduction generator, so as to train the SAR image noise reduction generator and the first gradient discriminator;
a second training module 204, configured to train the visible light image generator and the second gradient discriminator by using the visible light grayscale image as an input of the visible light image generator, using the visible light color image as an output of the visible light image generator, and discriminating the visible light color image output by the visible light image generator by using a second gradient discriminator;
the SAR image conversion module 205 is configured to input the SAR image to be converted into the trained SAR image noise reduction generator to obtain a visible light grayscale image to be converted, and input the visible light grayscale image to be converted into the trained visible light image generator to obtain a converted visible light color image.
The SAR image noise reduction generator is of an improved U-Net network structure, an input layer and an output layer of the SAR image noise reduction generator are single channels, and each adjacent single-layer convolution layer in the SAR image noise reduction generator is connected through a residual block and a jump connection.
The visible light image generator is of an improved U-Net network structure, and the number of output layer channels of the visible light image generator is 3.
The first gradient discriminator and the second gradient discriminator are both Markov discriminators.
The first training loss function for training the SAR image noise reduction generator and the first gradient discriminator is as follows:
wherein,
representing conditional generation-countering network loss of the SAR image noise reduction generator from inputting the SAR image to outputting a visible light grayscale image;
representing a reconstruction loss of the SAR image noise reduction generator from inputting the SAR image to outputting a visible light grayscale image,
representing a texture retention loss of the SAR image noise reduction generator from inputting the SAR image to outputting a visible light grayscale image;
representing the SAR image noise reduction generator, x(i)Representing the set of SAR images, y(i)A set of visible light grayscale images is represented,
the second training loss function training the visible light image generator and the second gradient discriminator is:
wherein,
representing conditional generation-countering network loss of the visible light image generator from inputting the visible light grayscale image to outputting a visible light color image;
representing a reconstruction loss of the visible light image generator from inputting the visible light gray scale image to outputting a visible light color image;
representing a texture-preserving loss of the visible light image generator from inputting the visible light grayscale image to outputting a visible light color image;
representing a visible light image generator, z(i)Representing a set of visible color images;
α1denotes a first weight parameter, β1Representing a second weight parameter, α2Denotes a third weight parameter, β2Representing a fourth weight parameter;
stopping training the SAR image noise reduction generator, the first gradient discriminator, the visible light image generator and the second gradient discriminator when the sum of the first training loss function and the second training loss function is less than a set threshold.
The embodiments in the present description are described in a progressive manner, each embodiment focuses on differences from other embodiments, and the same and similar parts among the embodiments are referred to each other. For the system disclosed by the embodiment, the description is relatively simple because the system corresponds to the method disclosed by the embodiment, and the relevant points can be referred to the method part for description.
The principles and embodiments of the present invention have been described herein using specific examples, which are provided only to help understand the method and the core concept of the present invention; meanwhile, for a person skilled in the art, according to the idea of the present invention, the specific embodiments and the application range may be changed. In view of the above, the present disclosure should not be construed as limiting the invention.
Claims (10)
1. A heterogeneous remote sensing image transformation method is characterized by comprising the following steps:
denoising the plurality of SAR images to obtain a visible light gray level image of each SAR image;
obtaining a visible light color image of each SAR image;
the SAR image is used as the input of an SAR image noise reduction generator, the visible light gray level image is used as the output of the SAR image noise reduction generator, a first gradient discriminator is used for discriminating the visible light gray level image output by the SAR image noise reduction generator, and the SAR image noise reduction generator and the first gradient discriminator are trained;
training the visible light image generator and a second gradient discriminator by taking the visible light gray scale image as the input of the visible light image generator, taking the visible light color image as the output of the visible light image generator and discriminating the visible light color image output by the visible light image generator by using the second gradient discriminator;
inputting the SAR image to be transformed into the trained SAR image noise reduction generator to obtain a visible light gray image to be transformed, and inputting the visible light gray image to be transformed into the trained visible light image generator to obtain a transformed visible light color image.
2. The heterogeneous remote sensing image transformation method according to claim 1, wherein the SAR image noise reduction generator is of an improved U-Net network structure, an input layer and an output layer of the SAR image noise reduction generator are both single channels, and each adjacent single-layer convolution layer in the SAR image noise reduction generator is connected through a residual block and a jump connection.
3. The heterogeneous remote sensing image transformation method according to claim 1, wherein the visible light image generator is of an improved U-Net network structure, and the number of output layer channels of the visible light image generator is 3.
4. The heterogeneous remote sensing image transformation method of claim 1, wherein the first gradient discriminator and the second gradient discriminator are both Markov discriminators.
5. The heterogeneous remote sensing image transformation method of claim 1, wherein a first training loss function for training the SAR image noise reduction generator and the first gradient discriminator is:
wherein,
representing conditional generation-countering network loss of the SAR image noise reduction generator from inputting the SAR image to outputting a visible light grayscale image;
representing a reconstruction loss of the SAR image noise reduction generator from inputting the SAR image to outputting a visible light grayscale image,
representing the SAR image noise reduction generator slave inputTexture retention loss from entering the SAR image to outputting a visible light gray level image;
representing the SAR image noise reduction generator, x(i)Representing the set of SAR images, y(i)A set of visible light grayscale images is represented,
the second training loss function training the visible light image generator and the second gradient discriminator is:
wherein,
representing conditional generation-countering network loss of the visible light image generator from inputting the visible light grayscale image to outputting a visible light color image;
representing a reconstruction loss of the visible light image generator from inputting the visible light gray scale image to outputting a visible light color image;
representing a texture-preserving loss of the visible light image generator from inputting the visible light grayscale image to outputting a visible light color image;
representing a visible light image generator, z(i)Representing a set of visible color images;
α1denotes a first weight parameter, β1Representing a second weight parameter, α2Denotes a third weight parameter, β2Representing a fourth weight parameter;
stopping training the SAR image noise reduction generator, the first gradient discriminator, the visible light image generator and the second gradient discriminator when the sum of the first training loss function and the second training loss function is less than a set threshold.
6. A heterogeneous remote sensing image transformation system, the system comprising:
the visible light gray level image acquisition module is used for denoising the SAR images to obtain visible light gray level images of the SAR images;
the visible light color image acquisition module is used for acquiring a visible light color image of each SAR image;
the first training module is used for taking the SAR image as the input of the SAR image noise reduction generator, taking the visible light gray image as the output of the SAR image noise reduction generator, identifying the visible light gray image output by the SAR image noise reduction generator by using a first gradient identifier, and training the SAR image noise reduction generator and the first gradient identifier;
the second training module is used for taking the visible light gray-scale image as the input of the visible light image generator, taking the visible light color image as the output of the visible light image generator, and identifying the visible light color image output by the visible light image generator by using a second gradient identifier to train the visible light image generator and the second gradient identifier;
and the SAR image conversion module is used for inputting the SAR image to be converted into the trained SAR image noise reduction generator to obtain the visible light gray image to be converted, and inputting the visible light gray image to be converted into the trained visible light image generator to obtain the converted visible light color image.
7. The heterogeneous remote sensing image transformation system of claim 6, wherein the SAR image noise reduction generator is of an improved U-Net network structure, an input layer and an output layer of the SAR image noise reduction generator are both single channels, and each adjacent single-layer convolution layer in the SAR image noise reduction generator is connected through a residual block and a jump connection.
8. The heterogeneous remote sensing image transformation system of claim 6, wherein the visible light image generator is of a modified U-Net network structure, and the number of output layer channels of the visible light image generator is 3.
9. The heterogeneous remote sensing image transformation system of claim 6, wherein the first gradient discriminator and the second gradient discriminator are both Markov discriminators.
10. The heterogeneous remote sensing image transformation system of claim 6, wherein the first training loss function for training the SAR image noise reduction generator and the first gradient discriminator is:
wherein,
representing conditional generation-countering network loss of the SAR image noise reduction generator from inputting the SAR image to outputting a visible light grayscale image;
representing a reconstruction loss of the SAR image noise reduction generator from inputting the SAR image to outputting a visible light grayscale image,
representing a texture retention loss of the SAR image noise reduction generator from inputting the SAR image to outputting a visible light grayscale image; p denotes the SAR image noise reduction generator, x(i)Representing the set of SAR images, y(i)A set of visible light grayscale images is represented,
the second training loss function training the visible light image generator and the second gradient discriminator is:
wherein,
representing conditional generation-countering network loss of the visible light image generator from inputting the visible light grayscale image to outputting a visible light color image;
representing a reconstruction loss of the visible light image generator from inputting the visible light gray scale image to outputting a visible light color image;
representing a texture-preserving loss of the visible light image generator from inputting the visible light grayscale image to outputting a visible light color image; q denotes a visible light image generator, z(i)Representing a set of visible color images;
α1denotes a first weight parameter, β1Representing a second weight parameter, α2Denotes a third weight parameter, β2Representing a fourth weight parameter;
stopping training the SAR image noise reduction generator, the first gradient discriminator, the visible light image generator and the second gradient discriminator when the sum of the first training loss function and the second training loss function is less than a set threshold.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011236018.3A CN112330562B (en) | 2020-11-09 | 2020-11-09 | Heterogeneous remote sensing image transformation method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011236018.3A CN112330562B (en) | 2020-11-09 | 2020-11-09 | Heterogeneous remote sensing image transformation method and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112330562A true CN112330562A (en) | 2021-02-05 |
| CN112330562B CN112330562B (en) | 2022-11-15 |
Family
ID=74316926
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011236018.3A Active CN112330562B (en) | 2020-11-09 | 2020-11-09 | Heterogeneous remote sensing image transformation method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112330562B (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113012251A (en) * | 2021-03-17 | 2021-06-22 | 厦门大学 | SAR image automatic colorization method based on generation countermeasure network |
| CN113554671A (en) * | 2021-06-23 | 2021-10-26 | 西安电子科技大学 | Method and device for converting SAR image into visible light image based on contour enhancement |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105976345A (en) * | 2016-04-27 | 2016-09-28 | 北京航空航天大学 | Visible light remote sensing image synthesis method |
| CN109636742A (en) * | 2018-11-23 | 2019-04-16 | 中国人民解放军空军研究院航空兵研究所 | The SAR image of network and the mode conversion method of visible images are generated based on confrontation |
| CN110473142A (en) * | 2019-05-22 | 2019-11-19 | 南京理工大学 | Single image super resolution ratio reconstruction method based on deep learning |
| CN110751699A (en) * | 2019-10-15 | 2020-02-04 | 西安电子科技大学 | Color reconstruction method of optical remote sensing image based on convolutional neural network |
-
2020
- 2020-11-09 CN CN202011236018.3A patent/CN112330562B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105976345A (en) * | 2016-04-27 | 2016-09-28 | 北京航空航天大学 | Visible light remote sensing image synthesis method |
| CN109636742A (en) * | 2018-11-23 | 2019-04-16 | 中国人民解放军空军研究院航空兵研究所 | The SAR image of network and the mode conversion method of visible images are generated based on confrontation |
| CN110473142A (en) * | 2019-05-22 | 2019-11-19 | 南京理工大学 | Single image super resolution ratio reconstruction method based on deep learning |
| CN110751699A (en) * | 2019-10-15 | 2020-02-04 | 西安电子科技大学 | Color reconstruction method of optical remote sensing image based on convolutional neural network |
Non-Patent Citations (4)
| Title |
|---|
| CHENG MA ET AL.: ""Structure-Preserving Super Resolution with Gradient Guidance"", 《ARXIV》 * |
| LONG GUO ET AL.: ""Gray Scale Image Coloring Method Based on GAN"", 《DSIT 2020: PROCEEDINGS OF THE 3RD INTERNATIONAL CONFERENCE ON DATA SCIENCE AND INFORMATION》 * |
| PHILLIP ISOLA ET AL.: ""Image-to-Image Translation with Conditional Adversarial Networks"", 《IEEE》 * |
| PUYANG WANG ET AL.: ""Generating high quality visible images from SAR"", 《IEEE》 * |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113012251A (en) * | 2021-03-17 | 2021-06-22 | 厦门大学 | SAR image automatic colorization method based on generation countermeasure network |
| CN113012251B (en) * | 2021-03-17 | 2022-05-03 | 厦门大学 | SAR image automatic colorization method based on generation countermeasure network |
| CN113554671A (en) * | 2021-06-23 | 2021-10-26 | 西安电子科技大学 | Method and device for converting SAR image into visible light image based on contour enhancement |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112330562B (en) | 2022-11-15 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110956094B (en) | RGB-D multi-mode fusion personnel detection method based on asymmetric double-flow network | |
| CN111539316B (en) | High-resolution remote sensing image change detection method based on dual-attention twin network | |
| Lei et al. | Multiscale superpixel segmentation with deep features for change detection | |
| Liu et al. | Remote sensing image change detection based on information transmission and attention mechanism | |
| CN109871902B (en) | SAR small sample identification method based on super-resolution countermeasure generation cascade network | |
| CN114841972A (en) | Power transmission line defect identification method based on saliency map and semantic embedded feature pyramid | |
| CN109558806A (en) | The detection method and system of high score Remote Sensing Imagery Change | |
| CN109741340B (en) | Ice cover radar image ice layer refined segmentation method based on FCN-ASPP network | |
| CN113838064B (en) | Cloud removal method based on branch GAN using multi-temporal remote sensing data | |
| CN109635726B (en) | Landslide identification method based on combination of symmetric deep network and multi-scale pooling | |
| Wang et al. | The poor generalization of deep convolutional networks to aerial imagery from new geographic locations: an empirical study with solar array detection | |
| CN114255403A (en) | Optical remote sensing image data processing method and system based on deep learning | |
| CN112330562B (en) | Heterogeneous remote sensing image transformation method and system | |
| CN113887472B (en) | Remote sensing image cloud detection method based on cascade color and texture feature attention | |
| CN117765361B (en) | Method for detecting building change area in double-time-phase remote sensing image based on residual neural network | |
| CN115049841A (en) | Depth unsupervised multistep anti-domain self-adaptive high-resolution SAR image surface feature extraction method | |
| CN109977968A (en) | A kind of SAR change detecting method of deep learning classification and predicting | |
| CN105512622A (en) | Visible remote-sensing image sea-land segmentation method based on image segmentation and supervised learning | |
| Ge et al. | Coarse-to-fine foraminifera image segmentation through 3D and deep features | |
| CN115861756A (en) | Earth background small target identification method based on cascade combination network | |
| CN114842524A (en) | Face false distinguishing method based on irregular significant pixel cluster | |
| Zhang et al. | Spatial–temporal gray-level co-occurrence aware CNN for SAR image change detection | |
| CN116168240A (en) | Arbitrary-direction dense ship target detection method based on attention enhancement | |
| CN112784777B (en) | Unsupervised hyperspectral image change detection method based on countermeasure learning | |
| CN117994240A (en) | Multi-scale two-level optical remote sensing image stripe noise intelligent detection method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |