CN117314751A - Remote sensing image super-resolution reconstruction method based on generation type countermeasure network - Google Patents

Remote sensing image super-resolution reconstruction method based on generation type countermeasure network Download PDF

Info

Publication number
CN117314751A
CN117314751A CN202311357268.6A CN202311357268A CN117314751A CN 117314751 A CN117314751 A CN 117314751A CN 202311357268 A CN202311357268 A CN 202311357268A CN 117314751 A CN117314751 A CN 117314751A
Authority
CN
China
Prior art keywords
image
training
super
remote sensing
resolution
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202311357268.6A
Other languages
Chinese (zh)
Inventor
戴鹏远
刘文杰
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Nanjing University of Information Science and Technology
Original Assignee
Nanjing University of Information Science and Technology
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Nanjing University of Information Science and Technology filed Critical Nanjing University of Information Science and Technology
Priority to CN202311357268.6A priority Critical patent/CN117314751A/en
Publication of CN117314751A publication Critical patent/CN117314751A/en
Pending legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformation in the plane of the image
    • G06T3/40—Scaling the whole image or part thereof
    • G06T3/4053—Super resolution, i.e. output image resolution higher than sensor resolution
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/0464—Convolutional networks [CNN, ConvNet]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/0475—Generative networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/048—Activation functions
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/084—Backpropagation, e.g. using gradient descent
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/094—Adversarial learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/764—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
    • G06V10/765—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects using rules for classification or partitioning the feature space
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10032—Satellite or aerial image; Remote sensing
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
    • Y02T10/00—Road transport of goods or passengers
    • Y02T10/10—Internal combustion engine [ICE] based vehicles
    • Y02T10/40—Engine management systems

Abstract

The invention discloses a remote sensing image super-resolution reconstruction method based on a generated type countermeasure network, which comprises the following steps: acquiring an actual remote sensing image to be reconstructed, and inputting the actual remote sensing image to a trained super-resolution reconstruction model to obtain a reconstructed actual remote sensing image; the training method comprises the following steps: acquiring a training set; inputting the training set into a pre-constructed super-resolution reconstruction model to perform alternating antagonism iterative training of the generator and the discriminator until a preset iterative condition is met, and outputting a trained super-resolution reconstruction model; the pre-constructed super-resolution reconstruction model comprises a classifier, a plurality of generators and a discriminator. Classifying the received images through a classifier to determine the image type of the images, and improving the efficiency and quality of image processing; meanwhile, final parameters of the generator are determined through alternate antagonism iterative training of the generator and the discriminator, so that the efficiency and quality of image processing are further improved, and the image effect is better.

Description

Remote sensing image super-resolution reconstruction method based on generation type countermeasure network
Technical Field
The invention relates to a remote sensing image super-resolution reconstruction method based on a generated type countermeasure network, and belongs to the technical field of image reconstruction.
Background
Under the big data age, image data are widely used by people, but in the process of collecting images, the imaging effect of the images is not ideal due to the influence of a series of factors such as equipment performance, cost, image compression and the like; therefore, the requirements in a certain special scene, such as medicine, satellite remote sensing, biological feature recognition and the like, cannot be met. The super-resolution reconstruction of the remote sensing image is a technology for processing a low-resolution remote sensing image with complementary information to obtain a high-resolution remote sensing image. The remote sensing image is a data support and application foundation of the remote sensing technology, provides rich information for monitoring the earth surface, and has wide application in the fields of disaster monitoring, urban economic level evaluation, resource exploration and the like. The quality of the remote sensing image obtained from the satellite at present cannot meet the application requirements of ground object target identification, detailed land detection and the like, and the remote sensing image is high in cost and difficult to lift from the hardware, so that the remote sensing image resolution is lifted by attempting to lift from software through an image processing technology, the image resolution is improved at a lower economic cost, and the image utilization benefit is improved. At present, the remote sensing image super-resolution reconstruction method with better effect is a method based on machine learning and recent deep learning, wherein the method comprises an image super-resolution method based on a generated type countermeasure network. However, this type of method often has the following problems in performing super resolution of remote sensing images:
1) The diversity of data categories prevents the performance of super-resolution models from being improved. The remote sensing images shot by satellite remote sensing show different appearances in different areas, and the diversity of the image categories is mainly reflected on the content shown by the remote sensing images, for example, the images shot in the north-south polar region are mainly glacial rivers, the images shot in the eastern Africa are all grasslands, and the images shot in the Amazon zone are more related to tropical rainforests and the like. However, there is a large deviation between these different classes of images, whether texture details or color differences, resulting in a very unstable model training effect. Even if the image content is similar, such as desert and gobi, the difference of image textures can exist due to the difference of illumination intensity and climate in different areas, and the diversity of the image categories can influence the model training result. The diversity of image categories during model training can lead to model parameters that favor texture information for images with larger amounts of data. However, the types of the remote sensing images photographed by the satellites have a certain randomness, so that it is a difficult matter to process the remote sensing images of different types.
2) Noise-based feature imbalance can also affect the quality of the resulting model-generated image. The conventional model contrast generation type countermeasure network is in a dilemma of insufficient discrimination capability when dealing with unnecessary or redundant interference information in an image. In the process of image superprocessing, the model needs to aggregate data key features in time. However, designing such a filter requires a lot of expertise in signal processing, and it is not so easy to combine two different fields together.
3) The generated image is not clear enough, and the existence of artifacts is also an urgent problem to be solved. In the process of satellite remote sensing shooting, high-frequency signals are attenuated too much due to the fact that the signal transmission distance is too far, and an amplifying and compensating device is not arranged in the middle of the satellite remote sensing shooting, so that a shot image is very fuzzy, the quality of the shot image cannot meet the requirements of ground object target recognition, detailed land detection and the like, and good effects cannot be obtained in the application fields of disaster monitoring, urban economic level assessment, resource exploration and the like.
The traditional remote sensing image super-resolution reconstruction method mainly comprises 3 types, namely a frequency domain method, a space domain method and a frequency domain-space domain combined method. The frequency domain method is mainly divided into a frequency spectrum aliasing algorithm, a recursive least square method and the like, and the theoretical premise based on the frequency domain method is too ideal and cannot be effectively applied to a plurality of occasions; the airspace method has good flexibility, but the airspace method involves a plurality of factors such as intra-frame motion blur, optical blur and other complex degradation models, so that the optimization method is complex and the calculation cost is high; the frequency domain-space domain combining method combines the advantages of the frequency domain method and the space domain method, but has the problem of huge calculation amount. At present, the most commonly used remote sensing image super-resolution reconstruction algorithm based on deep learning mainly has two architectures, namely a convolutional neural network (convolutional neural networks, CNN) method and a GAN (generative adversarial network, GAN) method, and compared with the traditional bicubic interpolation (Bicubic Interpolation) method, the image generated by SRCNN has higher definition, but the method does not consider the characteristic of remote sensing image scale diversity.
The information disclosed in this background section is only for enhancement of understanding of the general background of the invention and should not be taken as an acknowledgement or any form of suggestion that this information forms the prior art already known to a person of ordinary skill in the art.
Disclosure of Invention
The invention aims to overcome the defects in the prior art and provides a remote sensing image super-resolution reconstruction method based on a generation type countermeasure network, which classifies received images through a classifier to adaptively determine the image type of the images and improve the efficiency and quality of image processing; on the other hand, through the alternate antagonism iterative training of the generator and the discriminator, the final parameters of the generator are determined, the efficiency and the quality of image processing are further improved, and the image effect is better.
In order to achieve the above purpose, the invention is realized by adopting the following technical scheme:
the invention discloses a remote sensing image super-resolution reconstruction method based on a generated type countermeasure network, which comprises the following steps:
acquiring an actual remote sensing image to be reconstructed;
inputting the actual remote sensing image to be reconstructed into a trained super-resolution reconstruction model to obtain a reconstructed actual remote sensing image;
the training method of the super-resolution reconstruction model comprises the following steps of:
acquiring a training set, wherein the training set comprises an original training image and a corresponding preprocessing training image;
inputting the training set into a pre-constructed super-resolution reconstruction model for iterative training, wherein the pre-constructed super-resolution reconstruction model comprises a classifier, generators of different image types and a discriminator; the training steps comprise:
inputting the preprocessing training image into a classifier to obtain a corresponding image type;
inputting the preprocessed training image into a generator of a corresponding image type to obtain a reconstructed training image;
inputting the reconstructed training image and the original training image into a discriminator, and carrying out image discrimination processing to obtain a discrimination result;
and according to the judging result, optimizing the parameter setting of the generator by minimizing the loss of the generator or optimizing the parameter setting of the discriminator by minimizing the loss of the discriminator, repeating the alternating antagonism iterative training of the generator and the discriminator until the preset iteration termination condition is met, and outputting a trained super-resolution reconstruction model.
Further, the training set comprises a Sentinel satellite remote sensing image data set and an AID satellite remote sensing image data set.
Further, the preprocessing step of preprocessing the training image includes:
acquiring an original training image;
performing image standardization processing according to the original training image to obtain a standard training image;
and reducing the resolution ratio by bilinear interpolation according to the standard training image to obtain a preprocessed training image.
Further, the bilinear interpolation is expressed as follows:
wherein f (x ', y') represents the interpolated data; (x ', y') represents the interpolated coordinates; f (x) 1 ，y 1 ) Representing first known data; (x) 1 ，y 1 ) Representing a first known coordinate; f (x) 2 ，y 1 ) Representing second known data; (x) 2 ，y 1 ) Representing a second known coordinate; f (x) 1 ，y 2 ) Representing third known data; (x) 1 ，y 2 ) Representing a third known coordinate; f (x) 2 ，y 2 ) Representing fourth known data; (x) 2 ，y 2 ) Representing fourth known coordinates.
Further, the classifier includes:
ResNet module: the feature map is obtained through convolution operation according to the preprocessing training image;
global average pooling layer: the pooling operation is carried out according to the feature map to obtain feature vectors;
full tie layer: and the image type corresponding to the preprocessing training image is obtained by performing classification probability mapping operation according to the feature vector.
Further, the image types include ocean, forest, desert, school and highway; one image type corresponds to each of the generators.
Further, the generator includes:
a first input module: for inputting a pre-processed training image;
and the feature extraction module is used for: the method comprises the steps of obtaining a preliminary extraction feature by using a preprocessing training image;
residual error module: the method comprises the steps of carrying out deep extraction operation by learning residual information of a preprocessing training image according to the preliminary extraction characteristics to obtain deep extraction characteristics;
up-sampling module: the method is used for carrying out up-sampling operation according to the deep extraction characteristics to obtain a reconstructed training image;
a first output module: for outputting the reconstructed training image.
Further, the discriminator includes:
a second input module: the method comprises the steps of inputting a reconstructed training image and an original training image;
and a convolution module: the method comprises the steps of carrying out image feature extraction operation through convolution layer combination according to the reconstructed training image and the original training image to obtain convolution extraction features;
full tie layer module: the method comprises the steps of extracting features according to convolution, and mapping probability scores to obtain probability scores;
and an output module: and the image judgment module is used for carrying out mapping between the generated image and the real image based on the Sigmoid activation function according to the probability score and outputting an image judgment result.
Further, the alternating challenge iterative training includes:
the parameters of the fixed discriminant are input into a pre-constructed super-resolution reconstruction model to obtain the discrimination result of the fixed discriminant; optimizing the parameter setting of the generator by using a counter propagation and optimization algorithm to minimize the loss of the generator according to the discrimination result of the fixed discriminator, and obtaining the optimized generator parameters;
fixing the optimized generator parameters, and inputting the training set into a pre-constructed super-resolution reconstruction model to obtain a discrimination result of a fixed generator; optimizing the parameter setting of the discriminator by minimizing the discriminator loss through a back propagation and optimization algorithm according to the discrimination result of the fixed generator, and obtaining optimized discriminator parameters;
and repeating the alternating antagonism iterative training of the generator and the discriminator until the preset iteration termination condition is met, and outputting a trained super-resolution reconstruction model.
Further, the preset iteration termination condition includes:
and responding to the iteration times reaching a preset iteration threshold, and taking a super-resolution reconstruction model with highest peak signal-to-noise ratio and structural similarity as a trained super-resolution reconstruction model based on the generated countermeasure network.
Compared with the prior art, the invention has the beneficial effects that:
according to the remote sensing image super-resolution reconstruction method based on the generation type countermeasure network, on one hand, the classifier is arranged in the super-resolution reconstruction model, the received images can be classified, the image types of the images are determined in a self-adaptive mode, and corresponding generators are selected according to the data types, so that the efficiency and quality of image processing are improved; on the other hand, through the alternate antagonism iterative training of the generator and the discriminator, the balance of the generator and the discriminator is realized, the final parameters of the generator are determined, the efficiency and the quality of image processing are further improved, and the image effect is better.
Meanwhile, in model training, the invention carries out preprocessing on the original training image, reduces the resolution ratio of the original training image, then carries out subsequent parameter adjustment of generators with different data types, and improves the training speed. The invention solves the problems of insufficient definition and noise and artifacts of the quality of the remote sensing images from various types at present, and can be suitable for the super-resolution reconstruction task of the remote sensing images of different types.
Drawings
FIG. 1 is a schematic structural diagram of a pre-constructed super-resolution reconstruction model provided in an embodiment;
FIG. 2 is a schematic diagram of a generator provided by an embodiment;
FIG. 3 is a schematic diagram of a arbiter provided in an embodiment;
FIG. 4 is an actual remote sensing image to be reconstructed provided by an embodiment;
fig. 5 is a reconstructed actual remote sensing image provided by the embodiment.
Detailed Description
The invention is further described below with reference to the accompanying drawings. The following examples are only for more clearly illustrating the technical aspects of the present invention, and are not intended to limit the scope of the present invention.
The embodiment provides a remote sensing image super-resolution reconstruction method based on a generated type countermeasure network, which comprises the following steps:
acquiring an actual remote sensing image to be reconstructed;
inputting an actual remote sensing image to be reconstructed into a trained super-resolution reconstruction model to obtain a reconstructed actual remote sensing image;
the training method of the super-resolution reconstruction model comprises the following steps of:
acquiring a training set, wherein the training set comprises an original training image and a corresponding preprocessing training image;
inputting the training set into a pre-constructed super-resolution reconstruction model for iterative training, wherein the pre-constructed super-resolution reconstruction model comprises a classifier, generators of different image types and a discriminator; the training steps comprise:
inputting the preprocessed training image into a classifier to obtain a corresponding image type;
inputting the preprocessed training image into a generator of a corresponding image type to obtain a reconstructed training image;
inputting the reconstructed training image and the original training image into a discriminator, and carrying out image discrimination processing to obtain discrimination results;
and according to the judging result, optimizing the parameter setting of the generator by minimizing the loss of the generator or optimizing the parameter setting of the discriminator by minimizing the loss of the discriminator, repeating the alternating antagonism iterative training of the generator and the discriminator until the preset iteration termination condition is met, and outputting a trained super-resolution reconstruction model.
The technical conception of the invention is as follows: on one hand, the received images are classified through the classifier, so that the data type of the images is determined in a self-adaptive mode, and corresponding generators are selected according to the data type, so that the efficiency and the quality of image processing are improved; on the other hand, through the alternate antagonism iterative training of the generator and the discriminator, the balance of the generator and the discriminator is realized, the final parameters of the generator are determined, the efficiency and the quality of image processing are further improved, and the image effect is better;
specifically, the training method of the super-resolution reconstruction model comprises the following steps:
step 1: and (5) data processing.
1.1: and acquiring an original data set, wherein the original data set comprises a Sentinel satellite remote sensing image data set and an AID satellite remote sensing image data set.
Dividing an original data set into an original training set and a testing set according to a set duty ratio, wherein the ratio of the original training set to the testing set is 8:2;
in order to prevent the occurrence of the fitting phenomenon in the training process, the original training set is divided into a training set and a verification set again;
overall, the ratio of training set, validation set and test set was 16:4:5.
1.2: the image data is subjected to normalization processing.
Because the images in the original data set may have the problem of inconsistent sizes, errors occur in the process of extracting the image features by the subsequent training task filter, and the normalization of the image sizes is helpful to accelerate the training speed of the model, so that the quality of the generated images is improved.
Taking the training set as an example, performing image standardization processing on the original training images in the training set to obtain standard training images.
1.3: bilinear interpolation is used to reduce the resolution of the image.
It should be noted that, the images obtained in the steps 1.1 and 1.2 are high resolution images, and the high resolution images are processed by bilinear interpolation to obtain the corresponding low resolution version images.
Taking the training set as an example, according to the standard training image, reducing the resolution by a bilinear interpolation method to obtain a corresponding preprocessing training image.
The expression for bilinear interpolation is as follows:
wherein f (x ', y') represents the interpolated data; (x ', y') represents the interpolated coordinates; f (x) 1 ，y 1 ) Representing first known data; (x) 1 ，y 1 ) Representing a first known coordinate; f (x) 2 ，y 1 ) Representing second known data; (x) 2 ，y 1 ) Representing a second known coordinate; f (x) 1 ，y 2 ) Representing third known data; (x) 1 ，y 2 ) Representing a third known coordinate; f (x) 2 ，y 2 ) Representing fourth known data; (x) 2 ，y 2 ) Representing fourth known coordinates.
Step 2: and (5) model training.
A super-resolution reconstruction model is pre-constructed as shown in fig. 1. The system comprises a classifier, a plurality of generators and a discriminator, wherein one generator corresponds to one image type, and the image type comprises ocean, forest, desert, school and expressway.
2.1: the classifier is used for obtaining a corresponding image type according to the preprocessed training image; obtaining a corresponding generator according to the data type;
specifically, the classifier includes:
ResNet module: the method comprises the steps of obtaining a generated feature map through convolution operation according to a preprocessed training image;
global average pooling layer: the pooling operation is carried out according to the generated feature map to obtain a generated feature vector;
full tie layer: and the method is used for carrying out classification probability mapping operation according to the generated feature vector to obtain the image type corresponding to the preprocessing training image.
Specifically, first, a preprocessing training image x is input to a res net module, and a feature map F (x) is obtained through a series of convolution operations.
Then, the feature map F (x) is transferred to the global averaging pooling layer (Global Average Pooling layer) for pooling operation, so as to obtain a K-dimensional feature vector F, where K is the number of classifications of data types.
Finally, the feature vector f is mapped to the classification probability P (y p I x) P (yi x), the image type corresponding to the preprocessed training image is obtained, as shown in formula (2) and formula (3):
wherein F (x) represents a feature map; f represents a feature vector; k represents the number of classifications of the image types; h represents the width of the feature map and has a value of 256; w represents the high of the feature map, and the value is 256; f (F) i，j Feature vectors representing the ith row and the jth column on the feature map;
P(y p |x) represents the classification probability; y is p Representing a corresponding image type; x represents the input pre-processed training image; w (w) p A weight representing the image type p; b p Representing the number of deviations of the image type p; w (w) q A weight representing the image type q; b q A deviation number representing the image type q;
w p f+b p a score representing the image type p, a measure representing the likelihood that the model considers that the input x belongs to the image type p;
w q f+b q a score representing image type q, a measure of the likelihood that the model considers that the input x belongs to image type q.
The image types arranged in the classifier comprise ocean, forest, desert, school and expressway; one image type corresponds to each of the generators. The pre-processed training images are input to the corresponding generators according to the image type.
2.2: the generator is used for obtaining a reconstructed training image according to the preprocessed training image; the training image is reconstructed into a clearer super-resolution image.
As shown in fig. 2, the generator includes:
a first input module: for inputting a pre-processed training image;
and the feature extraction module is used for: the method is used for primarily extracting the features of the preprocessing training image to obtain primarily extracted features, and is helpful for capturing basic structure and texture information of the image;
residual error module: the method comprises the steps of carrying out deep extraction operation through residual information of a learning preprocessing training image according to the preliminary extraction features to obtain deep extraction features so as to add the deep extraction features into a generated reconstructed training image to improve details and quality;
up-sampling module: the method is used for carrying out up-sampling operation according to the deep extracted features to obtain a reconstructed training image, so that the spatial resolution of the image can be increased, and meanwhile, the depth of a feature image is kept;
a first output module: for outputting the reconstructed training image.
The specific principle of the generator comprises that firstly, in a residual error module, an attention mechanism is integrated into a convolutional neural network of the generator, and the attention mechanism has a very remarkable effect in improving the algorithm performance. And secondly, directly carrying out one-dimensional convolution on the channel characteristic diagram after global pooling, and not compressing the characteristic diagram, so that the characteristic information is completely reserved, and in the whole process, the training speed of the model is faster because the two-dimensional convolution operation is not carried out on the whole characteristic diagram due to the one-dimensional convolution.
2.3: a discriminator: and the image judging device is used for carrying out image judging processing according to the original training image and the reconstructed training image to obtain judging results. Specifically, as shown in fig. 3, the discriminator includes:
a second input module: the method comprises the steps of inputting a super-resolution reconstructed training image and a high-resolution original training image;
and a convolution module: the method comprises the steps of carrying out image feature extraction operation through convolution layer combination according to a reconstructed training image and an original training image to obtain convolution extracted features; specifically, the method comprises 8 convolution layer combinations, wherein each convolution layer combination consists of one convolution layer and one Mish activation function, and the convolution layer combinations are used for extracting image features and helping a discriminator to distinguish differences between an original training image and a reconstructed training image.
Full tie layer module: the method comprises the steps of performing mapping of probability scores according to convolution extraction characteristics to obtain probability scores; the probability score can represent the probability of whether the input reconstructed training image is a true high resolution original training image.
And an output module: the method is used for mapping the generated image and the real image based on the Sigmoid activation function according to the probability score and outputting an image discrimination result. The output of the arbiter is mapped into the range of 0 to 1 using a Sigmoid activation function, representing the probability that the input reconstructed training image is a true high resolution original training image. 0 represents the generated image, i.e. the reconstructed training image, and 1 represents the real image, i.e. the original training image.
The network model structure of the discriminator mainly comprises a linear classifier composed of 8 convolution layer combinations, a full-connection layer module and an output module, wherein the convolution layer combination is composed of a layer of convolution with a convolution kernel of 3 and a Mish activation function; to reduce the loss of information, a convolution layer with a step length of 2 is used to replace the pooling layer, so that the size of the feature map is reduced; in order to maintain better feature expression capability, the number of channels is increased while the feature map is reduced; and finally, changing the feature map into a one-dimensional vector, and outputting a classification value for distinguishing the input image through a two-layer fully connected network and a Sigmoid activation function.
2.4 alternate challenge iterative training.
2.4.1 the input of the generator is the original training image I of high resolution HR Corresponding low resolution pre-processed training image I LR 。
For images with C color channels, we describe with tensors of size w×h×c, the generation of the super-resolution reconstructed training image can be represented by rW ×rh×c. We design the generator network as a feed forward neural network with weight parameters θ G The representation is: θ G ＝{W L ；b L The weight and bias of the generator layer L are represented. Its value may pass through a specific perceptual loss function L SR Obtained as shown in formula (4):
wherein,representing the optimal value of the weighting parameter of the generator, i.e. when the perceptual loss function +.> Weight parameter theta when taking minimum value G Is a value of (2); l (L) SR Representing a perceived loss function that is weighted by a plurality of loss function components; />Representing an nth preprocessed training image; />Representing an nth original training image; n represents the number of images of the training set.
The goal of the generator is to generate the super-resolution reconstructed training images so that they can fool the arbiter so that it cannot distinguish between the generated super-resolution reconstructed training images and the high-resolution original training images. The loss of the generator in this embodiment is represented using the generator counter loss to encourage the generated image to be closer to the real image. The loss function of the generator against the loss can be expressed using the following manner:
wherein L is adv (G) Watch (watch)The indicator generator combat losses; i HR Representing an original training image; p (P) train (I HR ) A dataset representing an original training image; i LR Representing a pre-processed training image; d (G (I) LR ) Representing the evaluation of the reconstructed training image generated by the generator by the arbiter.
The task of the 2.4.2 discriminant is to distinguish between the generated super-resolution reconstructed training image and the high-resolution original training image. The loss of the discriminator also includes a discriminator fight loss, which is the inverse of the generator fight loss, the loss function of the discriminator fight loss is as follows:
wherein L is adv (D) Representing the discriminator against loss;representing the expectations of the original training image;representing a desire to pre-process the training image; d (I) HR ) Representing the evaluation of the original training image by the discriminator; d (G (I) LR ) Representing the evaluation of the reconstructed training image generated by the generator by the arbiter.
2.4.3 during actual training, the generator and the discriminant are optimized by alternate training. The method comprises the following specific steps:
the parameters of the fixed discriminant are input into a pre-constructed super-resolution reconstruction model to obtain the discrimination result of the fixed discriminant; optimizing the parameter setting of the generator by using a counter propagation and optimization algorithm to minimize the loss of the generator according to the discrimination result of the fixed discriminator, and obtaining the optimized generator parameters;
fixing the optimized generator parameters, and inputting the training set into a pre-constructed super-resolution reconstruction model to obtain a discrimination result of a fixed generator; optimizing the parameter setting of the discriminator by minimizing the discriminator loss through a back propagation and optimization algorithm according to the discrimination result of the fixed generator, and obtaining optimized discriminator parameters;
the iterative training of alternating antagonism of the generator and the arbiter is repeated to balance the generator and the arbiter. The contrast training process is iterated continuously until the generator generates a super-resolution reconstructed training image, the discriminator cannot accurately distinguish the generated super-resolution reconstructed training image from the high-resolution original training image, the preset iteration termination condition is met, and a trained super-resolution reconstruction model is output.
The generator and the arbiter optimize parameters in the same way, the optimized parameters mainly comprise weight parameters w and bias parameters b, and the parameters are adjusted by back propagation algorithm (Backpropagation), gradient Descent (Gradient device) and other optimization methods when the neural network is trained, and the specific steps are as follows:
1. initializing: at the beginning of training, it is often necessary to initialize the weight parameter w and the bias parameter b.
2. Forward propagation: forward propagation computation is performed by the input data and the current weight and bias parameters w. This includes passing the input data to the various layers in the neural network and calculating the final model predictions.
3. Calculating loss: the value of the loss function of the loss of the generator or the arbiter is calculated from the generated super-resolution reconstructed training image and the high-resolution original training image. The loss function measures the gap between model predictions and actual labels.
4. Back propagation: the gradient of the loss with respect to the weight parameter w and the bias parameter b is calculated using a back propagation algorithm. This gradient represents the rate of change of the losses with respect to these parameters.
5. Parameter updating: the weight parameter w and the bias parameter b are updated according to a gradient descent or other optimization algorithm. The update rule is as follows:
wherein: w (w) new Representing the updated weight parameters; w (w) old Representing the original weight parameters; b new Representing the updated bias parameters; b old Representing the original bias parameters; alpha represents the step length of the learning rate controlling parameter updating;representing the gradient of the loss with respect to the weight parameter; />Representing the gradient of the loss with respect to the bias parameter.
6. Repeating the iteration: steps 2 to 5 are repeated until the loss reaches a satisfactory value or the model converges.
The whole process is an iterative optimization process, and the neural network gradually adjusts weights and biases by continuously calculating gradients and updating parameters so as to improve the fitting capability and performance of the data. The goal of the adjustment is to minimize the loss function so that the model can be efficiently predicted and generalized. In practical applications, this needs to be solved by two lines of code in the pytorch described below.
Backsaward () # calculate gradient
The optimizer step () # update parameter
The number of iterations in this embodiment is set to 100, adam is selected by the generator's optimizer, the learning rate is set to 0.0005, the convolution kernel size is 3*3, the number of convolution kernels is 3, the number of channels is 64, and the step size is 1.
And responding to the iteration times reaching a preset iteration threshold 100, saving the model parameters of each iteration and the scores of the peak signal-to-noise ratio and the structural similarity thereof, and taking the super-resolution reconstruction model with the highest peak signal-to-noise ratio and the highest structural similarity as a trained super-resolution reconstruction model based on the generated type countermeasure network.
2.5: model verification
Inputting the verification set into a trained super-resolution reconstruction model based on the generated type countermeasure network, and obtaining a verification result.
2.6: and (5) model testing.
Inputting the test set into a trained super-resolution reconstruction model based on the generated type countermeasure network, and obtaining a test result.
Step 3: the model is practically applied.
Acquiring an actual remote sensing image to be reconstructed, as shown in fig. 4;
and inputting the actual remote sensing image to be reconstructed into a trained super-resolution reconstruction model to obtain a reconstructed actual remote sensing image, wherein the effect is shown in figure 5.
In summary, most of the existing deep learning models achieve the purpose of generating super-resolution images with better effect by fixing image types, and the method needs actual experience and multiple parameter adjustment, which tends to increase cost consumption. The received image is classified by the classifier, so that the data type of the image is determined in a self-adaptive mode, a specific generator is specified for the data type, and the generated super-resolution image is better in effect.
Meanwhile, in model training, the method carries out preprocessing on the original training image, reduces the resolution of the original training image, then carries out subsequent parameter adjustment of generators of different data types, solves the problems of insufficient definition, noise, artifacts and the like of the quality of the remote sensing images of various types, and can be suitable for super-resolution reconstruction tasks of the remote sensing images of different types.
It will be appreciated by those skilled in the art that embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
The foregoing is only a preferred embodiment of the invention, it being noted that: it will be apparent to those skilled in the art that various modifications and adaptations can be made without departing from the principles of the present invention, and such modifications and adaptations are intended to be comprehended within the scope of the invention.

Claims (10)

1. A remote sensing image super-resolution reconstruction method based on a generated type countermeasure network is characterized by comprising the following steps:
acquiring an actual remote sensing image to be reconstructed;
inputting the actual remote sensing image to be reconstructed into a trained super-resolution reconstruction model to obtain a reconstructed actual remote sensing image;
the training method of the super-resolution reconstruction model comprises the following steps of:
acquiring a training set, wherein the training set comprises an original training image and a corresponding preprocessing training image;
inputting the training set into a pre-constructed super-resolution reconstruction model for iterative training, wherein the pre-constructed super-resolution reconstruction model comprises a classifier, generators of different image types and a discriminator; the training steps comprise:
inputting the preprocessing training image into a classifier to obtain a corresponding image type;
inputting the preprocessed training image into a generator of a corresponding image type to obtain a reconstructed training image;
inputting the reconstructed training image and the original training image into a discriminator, and carrying out image discrimination processing to obtain a discrimination result;
and according to the judging result, optimizing the parameter setting of the generator by minimizing the loss of the generator or optimizing the parameter setting of the discriminator by minimizing the loss of the discriminator, repeating the alternating antagonism iterative training of the generator and the discriminator until the preset iteration termination condition is met, and outputting a trained super-resolution reconstruction model.
2. The method for reconstructing the super-resolution of the remote sensing image based on the generated type countermeasure network according to claim 1, wherein the training set comprises a Sentinel satellite remote sensing image dataset and an AID satellite remote sensing image dataset.
3. The method for reconstructing a super-resolution of a remote sensing image based on a generated type countermeasure network according to claim 1, wherein the preprocessing step of preprocessing the training image comprises:
acquiring an original training image;
performing image standardization processing according to the original training image to obtain a standard training image;
and reducing the resolution ratio by bilinear interpolation according to the standard training image to obtain a preprocessed training image.
4. The method for reconstructing a super-resolution of a remote sensing image based on a generated type countermeasure network according to claim 3, wherein the expression of the bilinear interpolation method is as follows:
wherein f (x ', y') represents the interpolated data; (x ', y') represents the interpolated coordinates; f (x) 1 ，y 1 ) Representing first known data; (x) 1 ，y 1 ) Representing a first known coordinate; f (x) 2 ，y 1 ) Representing second known data; (x) 2 ，y 1 ) Representing a second known coordinate; f (x) 1 ，y 2 ) Representing third known data; (x) 1 ，y 2 ) Representing a third known coordinate; f (x) 2 ，y 2 ) Representing fourth known data; (x) 2 ，y 2 ) Representing fourth known coordinates.
5. The method for reconstructing a super-resolution of a remote sensing image based on a generated type countermeasure network according to claim 1, wherein the classifier comprises:
ResNet module: the feature map is obtained through convolution operation according to the preprocessing training image;
global average pooling layer: the pooling operation is carried out according to the feature map to obtain feature vectors;
full tie layer: and the image type corresponding to the preprocessing training image is obtained by performing classification probability mapping operation according to the feature vector.
6. The method for reconstructing a super-resolution of a remote sensing image based on a generated type countermeasure network according to claim 1, wherein the image types include ocean, forest, desert, school and expressway; one image type corresponds to each of the generators.
7. The method for reconstructing a super-resolution of a remote sensing image based on a generated type countermeasure network according to claim 1, wherein the generator comprises:
a first input module: for inputting a pre-processed training image;
and the feature extraction module is used for: the method comprises the steps of obtaining a preliminary extraction feature by using a preprocessing training image;
residual error module: the method comprises the steps of carrying out deep extraction operation by learning residual information of a preprocessing training image according to the preliminary extraction characteristics to obtain deep extraction characteristics;
up-sampling module: the method is used for carrying out up-sampling operation according to the deep extraction characteristics to obtain a reconstructed training image;
a first output module: for outputting the reconstructed training image.
8. The method for reconstructing a super-resolution of a remote sensing image based on a generated type countermeasure network according to claim 1, wherein the discriminator comprises:
a second input module: the method comprises the steps of inputting a reconstructed training image and an original training image;
and a convolution module: the method comprises the steps of carrying out image feature extraction operation through convolution layer combination according to the reconstructed training image and the original training image to obtain convolution extraction features;
full tie layer module: the method comprises the steps of extracting features according to convolution, and mapping probability scores to obtain probability scores;
and an output module: and the image judgment module is used for carrying out mapping between the generated image and the real image based on the Sigmoid activation function according to the probability score and outputting an image judgment result.
9. The method for reconstructing the super-resolution of the remote sensing image based on the generated countermeasure network according to claim 1, wherein the alternating countermeasure iterative training comprises:
the parameters of the fixed discriminant are input into a pre-constructed super-resolution reconstruction model to obtain the discrimination result of the fixed discriminant; optimizing the parameter setting of the generator by using a counter propagation and optimization algorithm to minimize the loss of the generator according to the discrimination result of the fixed discriminator, and obtaining the optimized generator parameters;
fixing the optimized generator parameters, and inputting the training set into a pre-constructed super-resolution reconstruction model to obtain a discrimination result of a fixed generator; optimizing the parameter setting of the discriminator by minimizing the discriminator loss through a back propagation and optimization algorithm according to the discrimination result of the fixed generator, and obtaining optimized discriminator parameters;
and repeating the alternating antagonism iterative training of the generator and the discriminator until the preset iteration termination condition is met, and outputting a trained super-resolution reconstruction model.
10. The method for reconstructing a super-resolution of a remote sensing image based on a generated type countermeasure network according to claim 1, wherein the preset iteration termination condition includes:
and responding to the iteration times reaching a preset iteration threshold, and taking a super-resolution reconstruction model with highest peak signal-to-noise ratio and structural similarity as a trained super-resolution reconstruction model based on the generated countermeasure network.
CN202311357268.6A 2023-10-19 2023-10-19 Remote sensing image super-resolution reconstruction method based on generation type countermeasure network Pending CN117314751A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202311357268.6A CN117314751A (en) 2023-10-19 2023-10-19 Remote sensing image super-resolution reconstruction method based on generation type countermeasure network

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202311357268.6A CN117314751A (en) 2023-10-19 2023-10-19 Remote sensing image super-resolution reconstruction method based on generation type countermeasure network

Publications (1)

Publication Number Publication Date
CN117314751A true CN117314751A (en) 2023-12-29

Family

ID=89273567

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202311357268.6A Pending CN117314751A (en) 2023-10-19 2023-10-19 Remote sensing image super-resolution reconstruction method based on generation type countermeasure network

Country Status (1)

Country Link
CN (1) CN117314751A (en)

Similar Documents

Publication Publication Date Title
CN111724478B (en) Point cloud up-sampling method based on deep learning
CN112052886B (en) Intelligent human body action posture estimation method and device based on convolutional neural network
CN109345508B (en) Bone age evaluation method based on two-stage neural network
CN110599401A (en) Remote sensing image super-resolution reconstruction method, processing device and readable storage medium
CN113313657B (en) Unsupervised learning method and system for low-illumination image enhancement
CN115661144B (en) Adaptive medical image segmentation method based on deformable U-Net
CN112132959B (en) Digital rock core image processing method and device, computer equipment and storage medium
Zhao et al. ADRN: Attention-based deep residual network for hyperspectral image denoising
CN115565056A (en) Underwater image enhancement method and system based on condition generation countermeasure network
CN116563682A (en) Attention scheme and strip convolution semantic line detection method based on depth Hough network
CN115063318A (en) Adaptive frequency-resolved low-illumination image enhancement method and related equipment
CN114897728A (en) Image enhancement method and device, terminal equipment and storage medium
CN114037888A (en) Joint attention and adaptive NMS (network management System) -based target detection method and system
CN117151990B (en) Image defogging method based on self-attention coding and decoding
CN112132207A (en) Target detection neural network construction method based on multi-branch feature mapping
CN116542924A (en) Prostate focus area detection method, device and storage medium
CN116385281A (en) Remote sensing image denoising method based on real noise model and generated countermeasure network
CN114463176B (en) Image super-resolution reconstruction method based on improved ESRGAN
CN114511798B (en) Driver distraction detection method and device based on transformer
CN117314751A (en) Remote sensing image super-resolution reconstruction method based on generation type countermeasure network
CN115223033A (en) Synthetic aperture sonar image target classification method and system
CN114529730A (en) Convolutional neural network ground material image classification method based on LBP (local binary pattern) features
CN113989256A (en) Detection model optimization method, detection method and detection device for remote sensing image building
Wu et al. Fish Target Detection in Underwater Blurred Scenes Based on Improved YOLOv5
Malathi et al. Optimzied resnet model of convolutional neural network for under sea water object detection and classification

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination