CN113516591B - Remote sensing image super-resolution reconstruction method, device, equipment and storage medium - Google Patents
Remote sensing image super-resolution reconstruction method, device, equipment and storage medium Download PDFInfo
- Publication number
- CN113516591B CN113516591B CN202110818992.9A CN202110818992A CN113516591B CN 113516591 B CN113516591 B CN 113516591B CN 202110818992 A CN202110818992 A CN 202110818992A CN 113516591 B CN113516591 B CN 113516591B
- Authority
- CN
- China
- Prior art keywords
- remote sensing
- image
- super
- sensing image
- unmanned aerial
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 79
- 238000012549 training Methods 0.000 claims abstract description 63
- 238000012545 processing Methods 0.000 claims abstract description 23
- 238000007781 pre-processing Methods 0.000 claims abstract description 21
- 238000000605 extraction Methods 0.000 claims description 21
- 230000004913 activation Effects 0.000 claims description 14
- 230000006870 function Effects 0.000 claims description 14
- 238000004590 computer program Methods 0.000 claims description 9
- 230000008569 process Effects 0.000 claims description 8
- 238000011176 pooling Methods 0.000 claims description 7
- 238000010276 construction Methods 0.000 claims description 3
- 238000012937 correction Methods 0.000 claims description 3
- 230000004927 fusion Effects 0.000 claims description 3
- 238000011478 gradient descent method Methods 0.000 claims description 3
- 238000010586 diagram Methods 0.000 description 7
- 230000009471 action Effects 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 2
- 230000006872 improvement Effects 0.000 description 2
- 238000012952 Resampling Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000007796 conventional method Methods 0.000 description 1
- 238000013136 deep learning model Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000011156 evaluation Methods 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000011835 investigation Methods 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T3/4053—Scaling of whole images or parts thereof, e.g. expanding or contracting based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/174—Segmentation; Edge detection involving the use of two or more images
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10032—Satellite or aerial image; Remote sensing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20021—Dividing image into blocks, subimages or windows
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
- Y02T10/00—Road transport of goods or passengers
- Y02T10/10—Internal combustion engine [ICE] based vehicles
- Y02T10/40—Engine management systems
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- General Engineering & Computer Science (AREA)
- Evolutionary Computation (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Computational Linguistics (AREA)
- Software Systems (AREA)
- Mathematical Physics (AREA)
- Health & Medical Sciences (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Computing Systems (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Evolutionary Biology (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Image Processing (AREA)
- Image Analysis (AREA)
Abstract
The application discloses a remote sensing image super-resolution reconstruction method, a device, equipment and a storage medium, which comprise the following steps: collecting satellite remote sensing images and unmanned aerial vehicle images in the same area, and performing preprocessing and dicing processing to obtain satellite remote sensing image dicing and unmanned aerial vehicle image dicing; according to the satellite remote sensing image dicing and the unmanned aerial vehicle image dicing, a training sample set is established; constructing a ESRGAN-based super-resolution reconstruction model and training; preprocessing and dicing the remote sensing image to be processed to obtain a remote sensing image dicing to be processed; performing super-resolution reconstruction on the remote sensing image cut blocks to be processed by using the trained super-resolution reconstruction model; and cutting the reconstructed remote sensing image to be processed into blocks, and adding coordinate information to generate a new remote sensing image. Therefore, the multisource multiscale remote sensing image is used as a data source, super-resolution reconstruction is carried out through the super-resolution reconstruction model, the super-resolution result is good, and the problem of insufficient resolution of the remote sensing image is solved.
Description
Technical Field
The present invention relates to the field of remote sensing technologies, and in particular, to a method, an apparatus, a device, and a storage medium for reconstructing a remote sensing image with super resolution.
Background
Along with the rapid development and popularization of remote sensing technology, particularly along with the improvement of the spatial resolution of images, the application requirements of people on remote sensing data are increasing, and the requirements on the resolution of remote sensing images are also increasing from large-scale ecological environments to mesoscale resource investigation to higher-scale image classification. At present, the global remote sensing satellites better than 1 meter are few, the requirements of all users in the world are difficult to meet at the same time, for some small-scale areas, unmanned aerial vehicles can be used for aerial photography, the unmanned aerial vehicles are used for aerial photography, the space resolution is higher, the flexibility is high, the atmospheric influence is small, but the shooting cost is high by using the unmanned aerial vehicles, and some areas are inconvenient to reach and cannot acquire images. Based on the two situations, in order to simultaneously consider the problems of resolution, shooting cost, feasibility and the like, the resolution of the satellite image is enlarged to that of the unmanned aerial vehicle image, the conventional method for improving the resolution of the remote sensing image is interpolation, and the most adjacent, bilinear, cubic interpolation and other methods can be used, and in addition, super-resolution reconstruction is a method for improving the resolution of the image.
In terms of super-resolution reconstruction of remote sensing images, many studies have been conducted, and a single data source is generally used, so that although good results have been obtained, the super-resolution reconstruction is performed by using a single resolution, and the final result never exceeds the initial resolution. For example, when using a remote sensing image with a resolution of 5 meters for super-resolution reconstruction, the image is firstly downsampled to 10 meters, and then a model of 10 meters super-resolution to 5 meters is trained, and the model is only suitable for data of 10 meters to 5 meters, and when using data of 5 meters as test data, the effect is far from ideal. In cases where the data is not inherently sufficient, the practical significance of such an algorithm is not great.
Therefore, how to solve the problem of insufficient resolution of the remote sensing image is a technical problem to be solved by those skilled in the art.
Disclosure of Invention
In view of the above, the present invention aims to provide a remote sensing image super-resolution reconstruction method, apparatus, device and storage medium, which can effectively improve the image resolution and alleviate the problem of insufficient resolution of remote sensing images. The specific scheme is as follows:
A remote sensing image super-resolution reconstruction method comprises the following steps:
collecting satellite remote sensing images and unmanned aerial vehicle images in the same area, and performing preprocessing and dicing processing to obtain satellite remote sensing image dicing and unmanned aerial vehicle image dicing;
according to the satellite remote sensing image dicing and the unmanned aerial vehicle image dicing, a training sample set is established;
establishing a ESRGAN-based super-resolution reconstruction model, and training the super-resolution reconstruction model through the training sample set;
Preprocessing and dicing the remote sensing image to be processed to obtain a remote sensing image dicing to be processed;
performing super-resolution reconstruction on the remote sensing image cut blocks to be processed by using the trained super-resolution reconstruction model;
And cutting the reconstructed remote sensing image to be processed into blocks, and adding coordinate information to generate a new remote sensing image.
Preferably, in the above remote sensing image super-resolution reconstruction method provided by the embodiment of the present invention, the constructing a ESRGAN-based super-resolution reconstruction model includes:
constructing a generator; the generator comprises a convolution layer, an activation function, four block extraction feature layers, a convolution layer, a pooling layer, an activation function and a pixelshuffer layer which are connected in sequence; the block extraction feature layer is a feature extraction part of EFFICIENTNET-B4;
constructing a discriminator; the arbiter includes a VGG19 feature layer.
Preferably, in the above remote sensing image super-resolution reconstruction method provided by the embodiment of the present invention, the training the super-resolution reconstruction model through the training sample set includes:
The satellite remote sensing images in the training sample set are cut into blocks to serve as input data of the generator, and super-resolution reconstruction results are generated after the satellite remote sensing images are processed by the grower;
Taking the superdivision reconstruction result generated by the generator and the unmanned aerial vehicle image cut corresponding to the training sample set as input data of the discriminator, and extracting features by using a VGG19 feature layer;
calculating a loss of the generator and a loss of the arbiter, respectively; wherein the loss of the generator comprises a perceptual loss and an LI norm;
And respectively and alternately training the generator and the discriminator to obtain the super-resolution reconstruction model after training.
Preferably, in the above remote sensing image super-resolution reconstruction method provided by the embodiment of the present invention, while the generator and the arbiter are alternately trained, the method further includes:
parameter adjustment of the super-resolution reconstruction model is carried out by using a random gradient descent method;
and determining that the training of the super-resolution reconstruction model is finished by using an early-stop method.
Preferably, in the above remote sensing image super-resolution reconstruction method provided by the embodiment of the present invention, in a process of preprocessing the satellite remote sensing image and the unmanned aerial vehicle image, the method includes:
Performing image fusion, geographic registration, radiometric calibration, atmospheric correction and image stretching pretreatment on the satellite remote sensing image;
performing three-dimensional reconstruction, orthographic image processing and image mosaic processing on the unmanned aerial vehicle image;
And registering the satellite remote sensing image with the unmanned aerial vehicle image by using the unmanned aerial vehicle image as a reference.
Preferably, in the above method for reconstructing a super-resolution of a remote sensing image according to the embodiment of the present invention, in a process of performing dicing processing on the satellite remote sensing image and the unmanned aerial vehicle image to obtain a satellite remote sensing image diced piece and an unmanned aerial vehicle image diced piece, the method includes:
cutting the satellite remote sensing image according to the set cutting size to obtain satellite remote sensing image cutting blocks;
Acquiring picture coordinates of the satellite remote sensing image to obtain geographic coordinates corresponding to the satellite remote sensing image dicing;
Obtaining corresponding picture coordinates of the unmanned aerial vehicle image according to the geographic coordinates and a six-parameter coordinate system of the unmanned aerial vehicle image;
And cutting the unmanned aerial vehicle image by using the picture coordinates of the unmanned aerial vehicle image to obtain unmanned aerial vehicle image cutting blocks.
Preferably, in the above remote sensing image super-resolution reconstruction method provided by the embodiment of the present invention, the creating a training sample set according to the satellite remote sensing image dicing and the unmanned aerial vehicle image dicing includes:
The satellite remote sensing image cut blocks and the unmanned aerial vehicle image cut blocks which correspond to each other in position are made into the same numbers, and an image pair is formed;
establishing a training sample set formed by a plurality of image pairs; each training is performed using one of the pairs of images in the training sample set.
The embodiment of the invention also provides a remote sensing image super-resolution reconstruction device, which comprises:
The multi-scale image processing module is used for acquiring satellite remote sensing images and unmanned aerial vehicle images in the same area, and preprocessing and dicing the satellite remote sensing images and the unmanned aerial vehicle images to obtain satellite remote sensing image dicing and unmanned aerial vehicle image dicing;
the sample set establishing module is used for establishing a training sample set according to the satellite remote sensing image dicing and the unmanned aerial vehicle image dicing;
The model construction training module is used for constructing a ESRGAN-based super-resolution reconstruction model and training the super-resolution reconstruction model through the training sample set;
the to-be-processed image processing module is used for preprocessing and dicing the to-be-processed remote sensing image to obtain to-be-processed remote sensing image dicing;
The super-resolution reconstruction module is used for performing super-resolution reconstruction on the remote sensing image cut blocks to be processed by using the trained super-resolution reconstruction model;
And the remote sensing image generation module is used for splicing the reconstructed remote sensing images to be processed into blocks, adding coordinate information and generating new remote sensing images.
The embodiment of the invention also provides a remote sensing image super-resolution reconstruction device, which comprises a processor and a memory, wherein the remote sensing image super-resolution reconstruction method provided by the embodiment of the invention is realized when the processor executes a computer program stored in the memory.
The embodiment of the invention also provides a computer readable storage medium for storing a computer program, wherein the computer program realizes the remote sensing image super-resolution reconstruction method provided by the embodiment of the invention when being executed by a processor.
From the above technical solution, the remote sensing image super-resolution reconstruction method provided by the invention includes: collecting satellite remote sensing images and unmanned aerial vehicle images in the same area, and performing preprocessing and dicing processing to obtain satellite remote sensing image dicing and unmanned aerial vehicle image dicing; according to the satellite remote sensing image dicing and the unmanned aerial vehicle image dicing, a training sample set is established; establishing a ESRGAN-based super-resolution reconstruction model, and training the super-resolution reconstruction model through a training sample set; preprocessing and dicing the remote sensing image to be processed to obtain a remote sensing image dicing to be processed; performing super-resolution reconstruction on the remote sensing image cut blocks to be processed by using the trained super-resolution reconstruction model; and cutting the reconstructed remote sensing image to be processed into blocks, and adding coordinate information to generate a new remote sensing image.
According to the remote sensing image super-resolution reconstruction method provided by the invention, the multi-source multi-scale remote sensing image is used as a data source, the super-resolution reconstruction is carried out on the remote sensing image to be processed through the ESRGAN-based super-resolution reconstruction model, the super-resolution result is good, the problem that the resolution of the high-resolution remote sensing image cannot meet the requirement is solved, and the application of the high-resolution remote sensing image is further promoted. In addition, the invention also provides a corresponding device, equipment and a computer readable storage medium for the remote sensing image super-resolution reconstruction method, so that the method has more practicability, and the device, equipment and computer readable storage medium have corresponding advantages.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the related art, the drawings that are required to be used in the embodiments or the related technical descriptions will be briefly described, and it is apparent that the drawings in the following description are only embodiments of the present invention, and other drawings may be obtained according to the provided drawings without inventive effort for those skilled in the art.
FIG. 1 is a flowchart of a remote sensing image super-resolution reconstruction method according to an embodiment of the present invention;
Fig. 2 is a schematic diagram of a remote sensing image super-resolution reconstruction method according to an embodiment of the present invention;
Fig. 3 is a schematic structural diagram of a generator of a super-resolution reconstruction model according to an embodiment of the present invention;
fig. 4 is a schematic structural diagram of a block extraction feature layer of a generator according to an embodiment of the present invention;
Fig. 5 is a schematic structural diagram of a discriminator of a super-resolution reconstruction model according to the embodiment of the invention;
Fig. 6 to 8 are respectively front and rear contrast diagrams of super-resolution reconstruction provided in the embodiment of the present invention;
fig. 9 is a schematic structural diagram of a remote sensing image super-resolution reconstruction device according to an embodiment of the present invention.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
The invention provides a remote sensing image super-resolution reconstruction method, which is shown in fig. 1 and 2 and comprises the following steps:
s101, acquiring satellite remote sensing images and unmanned aerial vehicle images in the same area, and performing preprocessing and dicing processing to obtain satellite remote sensing image dicing and unmanned aerial vehicle image dicing;
it will be appreciated that the resolution of the satellite remote sensing image is lower than the resolution of the unmanned aerial vehicle image, i.e. the satellite remote sensing image is a low resolution image and the unmanned aerial vehicle image is a high resolution image. The satellite remote sensing image and the unmanned aerial vehicle image have two different scales, and the remote sensing image is too large to be processed at one time and needs to be cut into one piece for dicing. The invention takes the high-resolution image as the label of the super-resolution model, and takes the low-resolution image as the original data of the super-resolution model.
S102, according to satellite remote sensing image dicing and unmanned aerial vehicle image dicing, a training sample set is established;
in practical application, each satellite remote sensing image cut and each unmanned aerial vehicle image cut can store corresponding image indexes and geographic information, so that subsequent model operation is facilitated.
S103, constructing a super-resolution reconstruction model based on ESRGAN, and training the super-resolution reconstruction model through a training sample set;
It should be noted that, the Super-Resolution reconstruction model constructed in the present invention is based on ESRGAN (Enhanced Super-Resolution GENERATIVE ADVERSARIAL Networks), and the ESRGAN model is an improvement on SRGAN (Super-Resolution GENERATIVE ADVERSARIAL Networks), by generating an countermeasure network, training a high-Resolution (SR) picture, generating a high-Resolution image, and calculating a feature difference between the generated high-Resolution image and a real picture to calculate a loss to train the model.
S104, preprocessing and dicing the remote sensing image to be processed to obtain a remote sensing image dicing to be processed;
s105, performing super-resolution reconstruction on the remote sensing image cut block to be processed by using the trained super-resolution reconstruction model;
S106, splicing the reconstructed remote sensing images to be processed, cutting the remote sensing images into blocks, and adding coordinate information to generate new remote sensing images.
In the remote sensing image super-resolution reconstruction method provided by the embodiment of the invention, the multi-source multi-scale remote sensing image is used as a data source, the super-resolution reconstruction is carried out on the remote sensing image to be processed through the ESRGAN-based super-resolution reconstruction model, the super-resolution result is good, the problem that the resolution of the high-resolution remote sensing image cannot meet the requirement is solved, and the application of the high-resolution remote sensing image is further promoted.
In a specific implementation, in the above remote sensing image super-resolution reconstruction method provided by the embodiment of the present invention, in a process of performing step S101 to preprocess a satellite remote sensing image and an unmanned aerial vehicle image, the method may specifically include: firstly, preprocessing such as image fusion, geographic registration, radiometric calibration, atmosphere correction, image stretching and the like is carried out on a satellite remote sensing image; then, carrying out three-dimensional reconstruction, orthographic image processing, image mosaic processing and the like on the unmanned aerial vehicle image; finally, the unmanned aerial vehicle image is used as a reference, the satellite remote sensing image and the unmanned aerial vehicle image are registered with high precision, and the precision requirement is controlled within 0.5 pixel. In practical applications, for images with large scale differences, accurate registration is required, and in low resolution images, the error is not more than 0.5 pixel.
In a specific implementation, in the above remote sensing image super-resolution reconstruction method provided by the embodiment of the present invention, in a process of executing step S101 to perform dicing processing on a satellite remote sensing image and an unmanned aerial vehicle image to obtain a satellite remote sensing image dicing and an unmanned aerial vehicle image dicing, the method specifically may include the following steps:
Firstly, cutting a satellite remote sensing image according to a set cutting size to obtain satellite remote sensing image cutting blocks; specifically, with the satellite remote sensing image with low resolution as a reference, the set dicing size can be set to 512×512, and the dicing is performed once every 512 pixels without overlapping;
Then, obtaining picture coordinates of the satellite remote sensing image, and obtaining geographic coordinates corresponding to the satellite remote sensing image cutting blocks; specifically, a six-parameter coordinate system (t 0, t1, t2, t3, t4, t 5) corresponding to the satellite remote sensing image with low resolution is used to obtain geographic coordinates corresponding to the satellite remote sensing image dicing;
then, according to the geographic coordinates and a six-parameter coordinate system of the unmanned aerial vehicle image, obtaining the picture coordinates of the corresponding unmanned aerial vehicle image; specifically, using the obtained geographic coordinates and a six-parameter coordinate system (f 0, f1, f2, f3, f4, f 5) of the unmanned aerial vehicle image, finding out the picture coordinates of the corresponding range in the unmanned aerial vehicle image;
Finally, the unmanned aerial vehicle image is cut by using the picture coordinates of the unmanned aerial vehicle image, and the unmanned aerial vehicle image cutting block is obtained.
In a specific implementation, in the above remote sensing image super-resolution reconstruction method provided by the embodiment of the present invention, because the spatial resolution of the unmanned aerial vehicle image is high, the sizes of the cut blocks of the satellite image and the unmanned aerial vehicle image with the same geographic coordinates are different, and step S102 establishes a training sample set according to the cut blocks of the satellite remote sensing image and the cut blocks of the unmanned aerial vehicle image, and may specifically include: the method comprises the steps of cutting satellite remote sensing images and unmanned aerial vehicle images which correspond to each other in position into identical numbers to form an image pair, namely, using the satellite images and the unmanned aerial vehicle images which correspond to each other in position to form image pairs with different resolutions at the same place; establishing a training sample set formed by a plurality of image pairs; each training is performed using an image pair in the training sample set.
It should be noted that, after forming an image pair, the method may further include: resampling drone data. For high Resolution remote sensing images, the Resolution is typically around 1 meter, for unmanned aerial vehicle data, the Resolution is around 0.1 meter, in super Resolution models, the Resolution of the picture of SR is N times 2 of LR (Low Resolution), N is typically 2, 3, 4. For remote sensing data, the resolution multiples of the unmanned plane data and the satellite data are not the power of N of 2 due to the influence of a data source, so that the unmanned plane data are firstly interpolated to the resolution of 8 times of the satellite image by an interpolation (such as bilinear interpolation) method, and the unmanned plane data are interpolated to 0.125 m for the remote sensing image with the resolution of 1 m.
Further, in a specific implementation, in the above remote sensing image super-resolution reconstruction method provided by the embodiment of the present invention, step S103 builds a ESRGAN-based super-resolution reconstruction model, which may specifically include: constructing a generator; as shown in fig. 3, the generator includes a convolution layer, an activation function, four block extraction feature layers, a convolution layer, a pooling layer, an activation function, and a pixelshuffer layer connected in sequence; as shown in fig. 4, the block extraction feature layer is the feature extraction portion of EFFICIENTNET-B4; constructing a discriminator; as shown in fig. 5, the arbiter includes a VGG19 feature layer.
It should be noted that the super-resolution reconstruction model is constructed by introducing EFFICIENTNET-B4 as a backbone part (backbone) of the network, so that the capability of feature extraction is improved. The generator is used for generating super-resolution images, and the discriminator is used for calculating characteristic differences between the generated data and the real unmanned aerial vehicle data. In the generator, the improved DenseBlock is used to extract the multiscale features of the remote sensing image, and in the arbiter, the VGG19 is used to extract the features to calculate the loss.
In a specific implementation, in the above remote sensing image super-resolution reconstruction method provided by the embodiment of the present invention, step S103 trains a super-resolution reconstruction model through a training sample set, and may specifically include:
Step one, cutting satellite remote sensing images in a training sample set into blocks to serve as input data of a generator, and processing the blocks by a grower to generate a superdivision reconstruction result;
Specifically, a satellite remote sensing image slice of four bands (RGB and near infrared bands) can be used as input data, converted into three bands by a layer of 1×1 convolution, and then passed through a layer of activation functions Relu, followed by a series of block extraction features. For each block, using the feature extraction part of EFFICIENTNET-B4 as DenseBlock of the super-resolution reconstruction model, fully using the multi-scale features of the remote sensing image, connecting the feature extraction parts of four EFFICIENTNET-B4, and connecting each extracted feature with the previous feature. Then through a convolution layer, a pooling layer, an activation function Relu, and using pixelshuffer, the super resolution image and the drone image are the same size.
Step two, taking the superdivision reconstruction result generated by the generator and unmanned aerial vehicle image cut blocks corresponding to the training sample set as input data of a discriminator, and extracting features by using a VGG19 feature layer;
Step three, calculating the loss of the generator and the loss of the discriminator respectively;
for the arbiter, its loss is:
Wherein x r represents real data, x f represents generated data, a similarity evaluation index between the real data and the generated data is constructed using the idea of RELATIVISTIC GAN, Represents the degree to which the real unmanned aerial vehicle image approximates the generated data,Representing the distance between the generated data and the real data.
For the generator, its loss is:
the perceived loss L precep in the super-resolution reconstruction model is increased, and an L1 norm is increased to increase the robustness of the super-resolution reconstruction model, using lambda and eta as super-parameters. The Loss of the final generator is:
and step four, respectively and alternately training a generator and a discriminator to obtain a super-resolution reconstruction model after training.
In a specific implementation, in the above remote sensing image super-resolution reconstruction method provided by the embodiment of the present invention, while respectively alternately training the generator and the arbiter in the fourth step, the method may further include: parameter adjustment of the super-resolution reconstruction model is carried out by using a random gradient descent method (Stochastic GRADIENT DESCENT, SGD); the learning rate (LEARNINGRATE) may be set to 0.025; and determining that the training of the super-resolution reconstruction model is completed by using an early-stop method (EarlyStopping) so as to prevent the model from being fitted excessively.
In a specific implementation, when executing the processing of dicing the remote sensing image to be processed in step S104, the same slice size as the training sample is used to cut the remote sensing image into 512×512 without overlapping, and the six parameters of coordinates of the remote sensing image to be processed are stored.
In a specific implementation, after performing step S105 to perform super-resolution reconstruction on the remote sensing image to be processed and the diced piece by using the trained super-resolution reconstruction model, a super-resolution image with the same resolution as the unmanned aerial vehicle may be obtained.
In a specific implementation, when executing step S106, the resolution of the new data may be modified by using the super-resolution image obtained in step S105 and the six parameters of coordinates stored in step S104, remote sensing data may be generated by using the image start coordinates and the new resolution in the six parameters of coordinates, and finally a new image in tiff format may be generated.
Fig. 6 to 8 show the contrast diagrams before and after super-resolution reconstruction, and it can be seen that the super-resolution reconstruction method and the device for the remote sensing image by using the deep learning model perform super-resolution reconstruction, have a better super-resolution result, improve the resolution of the image, and are beneficial to further application.
Based on the same inventive concept, the embodiment of the invention also provides a remote sensing image super-resolution reconstruction device, and because the principle of solving the problem of the device is similar to that of the remote sensing image super-resolution reconstruction method, the implementation of the device can be referred to the implementation of the remote sensing image super-resolution reconstruction method, and the repetition is omitted.
In a specific implementation, the remote sensing image super-resolution reconstruction device provided by the embodiment of the present invention, as shown in fig. 9, specifically includes:
the multi-scale image processing module 11 is used for acquiring satellite remote sensing images and unmanned aerial vehicle images in the same area, and performing preprocessing and dicing processing to obtain satellite remote sensing image dicing and unmanned aerial vehicle image dicing;
The sample set establishing module 12 is used for establishing a training sample set according to the satellite remote sensing image dicing and the unmanned aerial vehicle image dicing;
the model construction training module 13 is used for constructing a ESRGAN-based super-resolution reconstruction model and training the super-resolution reconstruction model through a training sample set;
the to-be-processed image processing module 14 is used for preprocessing and dicing the to-be-processed remote sensing image to obtain to-be-processed remote sensing image diced pieces;
The super-resolution reconstruction module 15 is used for performing super-resolution reconstruction on the remote sensing image cut blocks to be processed by using the trained super-resolution reconstruction model;
the remote sensing image generating module 16 is configured to splice the reconstructed to-be-processed remote sensing image into blocks, and add coordinate information to generate a new remote sensing image.
In the remote sensing image super-resolution reconstruction device provided by the embodiment of the invention, the multi-source multi-scale remote sensing image can be used as a data source through interaction of the six modules, the super-resolution reconstruction is carried out on the remote sensing image to be processed through the ESRGAN-based super-resolution reconstruction model, the super-resolution result is good, the problem that the resolution of the high-resolution remote sensing image cannot meet the requirement is solved, and the application of the high-resolution remote sensing image is further promoted.
For more specific working procedures of the above modules, reference may be made to the corresponding contents disclosed in the foregoing embodiments, and no further description is given here.
Correspondingly, the embodiment of the invention also discloses a remote sensing image super-resolution reconstruction device which comprises a processor and a memory; the remote sensing image super-resolution reconstruction method disclosed in the foregoing embodiment is implemented when the processor executes the computer program stored in the memory.
For more specific procedures of the above method, reference may be made to the corresponding contents disclosed in the foregoing embodiments, and no further description is given here.
Further, the invention also discloses a computer readable storage medium for storing a computer program; the computer program when executed by the processor realizes the remote sensing image super-resolution reconstruction method disclosed in the prior art.
For more specific procedures of the above method, reference may be made to the corresponding contents disclosed in the foregoing embodiments, and no further description is given here.
In this specification, each embodiment is described in a progressive manner, and each embodiment is mainly described in a different point from other embodiments, so that the same or similar parts between the embodiments are referred to each other. The apparatus, device, and storage medium disclosed in the embodiments are relatively simple to describe, and the relevant parts refer to the description of the method section because they correspond to the methods disclosed in the embodiments.
Those of skill would further appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or combinations of both, and that the various illustrative elements and steps are described above generally in terms of functionality in order to clearly illustrate the interchangeability of hardware and software. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the solution. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.
The steps of a method or algorithm described in connection with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. The software modules may be disposed in Random Access Memory (RAM), memory, read Only Memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.
The embodiment of the invention provides a remote sensing image super-resolution reconstruction method, which comprises the following steps: collecting satellite remote sensing images and unmanned aerial vehicle images in the same area, and performing preprocessing and dicing processing to obtain satellite remote sensing image dicing and unmanned aerial vehicle image dicing; according to the satellite remote sensing image dicing and the unmanned aerial vehicle image dicing, a training sample set is established; establishing a ESRGAN-based super-resolution reconstruction model, and training the super-resolution reconstruction model through a training sample set; preprocessing and dicing the remote sensing image to be processed to obtain a remote sensing image dicing to be processed; performing super-resolution reconstruction on the remote sensing image cut blocks to be processed by using the trained super-resolution reconstruction model; and cutting the reconstructed remote sensing image to be processed into blocks, and adding coordinate information to generate a new remote sensing image. Therefore, the multisource multiscale remote sensing image is used as a data source, the super-resolution reconstruction is carried out on the remote sensing image to be processed through the ESRGAN-based super-resolution reconstruction model, the super-resolution result is good, the problem that the resolution of the high-resolution remote sensing image cannot meet the requirement is solved, and the application of the high-resolution remote sensing image is further promoted. In addition, the invention also provides a corresponding device, equipment and a computer readable storage medium for the remote sensing image super-resolution reconstruction method, so that the method has more practicability, and the device, equipment and computer readable storage medium have corresponding advantages.
Finally, it is further noted that relational terms such as first and second, and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.
The above describes the remote sensing image super-resolution reconstruction method, device, equipment and storage medium provided by the invention in detail, and specific examples are applied to describe the principle and implementation of the invention, and the description of the above examples is only used for helping to understand the method and core idea of the invention; meanwhile, as those skilled in the art will have variations in the specific embodiments and application scope in accordance with the ideas of the present invention, the present description should not be construed as limiting the present invention in view of the above.
Claims (8)
1. The method for reconstructing the super-resolution of the remote sensing image is characterized by comprising the following steps of:
collecting satellite remote sensing images and unmanned aerial vehicle images in the same area, and performing preprocessing and dicing processing to obtain satellite remote sensing image dicing and unmanned aerial vehicle image dicing;
according to the satellite remote sensing image dicing and the unmanned aerial vehicle image dicing, a training sample set is established;
Constructing a ESRGAN-based super-resolution reconstruction model, which comprises the following steps: constructing a generator; the generator comprises a convolution layer, an activation function, four block extraction feature layers, a convolution layer, a pooling layer, an activation function and a pixelshuffer layer which are connected in sequence; the block extraction feature layer is a feature extraction part of EFFICIENTNET-B4, and the feature extraction part of EFFICIENTNET-B4 is used as DenseBlock of the super-resolution reconstruction model; constructing a discriminator; the discriminator comprises a VGG19 feature layer;
The satellite remote sensing images in the training sample set are cut into blocks to serve as input data of the generator, and a superdivision reconstruction result is generated after the blocks sequentially pass through a convolution layer, an activation function, four block extraction feature layers, a convolution layer, a pooling layer, an activation function and a pixelshuffer layer of the generator;
Taking the superdivision reconstruction result generated by the generator and the unmanned aerial vehicle image cut corresponding to the training sample set as input data of the discriminator, and extracting features by using a VGG19 feature layer;
calculating a loss of the generator and a loss of the arbiter, respectively; wherein the loss of the generator comprises a perceptual loss and an L1 norm; the loss of the generator is:
Wherein L G is the loss of the generator, L precep is the perceptual loss, lambda and eta are hyper-parameters, L 1 is L1 norm, x r represents real data, x f represents generated data, Represents the degree to which the real unmanned aerial vehicle image approximates the generated data,Representing the distance between the generated data and the real data,Representing the expected value;
alternately training the generator and the discriminator respectively to obtain the super-resolution reconstruction model after training;
Preprocessing and dicing the remote sensing image to be processed to obtain a remote sensing image dicing to be processed;
performing super-resolution reconstruction on the remote sensing image cut blocks to be processed by using the trained super-resolution reconstruction model;
And cutting the reconstructed remote sensing image to be processed into blocks, and adding coordinate information to generate a new remote sensing image.
2. The method of claim 1, wherein the alternately training the generator and the arbiter, respectively, further comprises:
parameter adjustment of the super-resolution reconstruction model is carried out by using a random gradient descent method;
and determining that the training of the super-resolution reconstruction model is finished by using an early-stop method.
3. The method of claim 1, wherein the preprocessing of the satellite remote sensing image and the unmanned aerial vehicle image comprises:
Performing image fusion, geographic registration, radiometric calibration, atmospheric correction and image stretching pretreatment on the satellite remote sensing image;
performing three-dimensional reconstruction, orthographic image processing and image mosaic processing on the unmanned aerial vehicle image;
And registering the satellite remote sensing image with the unmanned aerial vehicle image by using the unmanned aerial vehicle image as a reference.
4. The method for reconstructing a super-resolution image of claim 3, wherein in the process of dicing the satellite remote sensing image and the unmanned aerial vehicle image to obtain a satellite remote sensing image diced and an unmanned aerial vehicle image diced, the method comprises:
cutting the satellite remote sensing image according to the set cutting size to obtain satellite remote sensing image cutting blocks;
Acquiring picture coordinates of the satellite remote sensing image to obtain geographic coordinates corresponding to the satellite remote sensing image dicing;
Obtaining corresponding picture coordinates of the unmanned aerial vehicle image according to the geographic coordinates and a six-parameter coordinate system of the unmanned aerial vehicle image;
And cutting the unmanned aerial vehicle image by using the picture coordinates of the unmanned aerial vehicle image to obtain unmanned aerial vehicle image cutting blocks.
5. The method of claim 4, wherein the creating a training sample set from the satellite remote sensing image dicing and the unmanned aerial vehicle image dicing comprises:
The satellite remote sensing image cut blocks and the unmanned aerial vehicle image cut blocks which correspond to each other in position are made into the same numbers, and an image pair is formed;
establishing a training sample set formed by a plurality of image pairs; each training is performed using one of the pairs of images in the training sample set.
6. The utility model provides a remote sensing image super-resolution reconstruction device which characterized in that includes:
The multi-scale image processing module is used for acquiring satellite remote sensing images and unmanned aerial vehicle images in the same area, and preprocessing and dicing the satellite remote sensing images and the unmanned aerial vehicle images to obtain satellite remote sensing image dicing and unmanned aerial vehicle image dicing;
the sample set establishing module is used for establishing a training sample set according to the satellite remote sensing image dicing and the unmanned aerial vehicle image dicing;
The model construction training module is used for constructing a super-resolution reconstruction model based on ESRGAN, and comprises the following steps: constructing a generator; the generator comprises a convolution layer, an activation function, four block extraction feature layers, a convolution layer, a pooling layer, an activation function and a pixelshuffer layer which are connected in sequence; the block extraction feature layer is a feature extraction part of EFFICIENTNET-B4, and the feature extraction part of EFFICIENTNET-B4 is used as DenseBlock of the super-resolution reconstruction model; constructing a discriminator; the discriminator comprises a VGG19 feature layer; the satellite remote sensing images in the training sample set are cut into blocks to serve as input data of the generator, and a superdivision reconstruction result is generated after the blocks sequentially pass through a convolution layer, an activation function, four block extraction feature layers, a convolution layer, a pooling layer, an activation function and a pixelshuffer layer of the generator; taking the superdivision reconstruction result generated by the generator and the unmanned aerial vehicle image cut corresponding to the training sample set as input data of the discriminator, and extracting features by using a VGG19 feature layer; calculating a loss of the generator and a loss of the arbiter, respectively; wherein the loss of the generator comprises a perceptual loss and an L1 norm; the loss of the generator is:
Wherein L G is the loss of the generator, L precep is the perceptual loss, lambda and eta are hyper-parameters, L 1 is L1 norm, x r represents real data, x f represents generated data, Represents the degree to which the real unmanned aerial vehicle image approximates the generated data,Representing the distance between the generated data and the real data,Representing the expected value; alternately training the generator and the discriminator respectively to obtain the super-resolution reconstruction model after training;
the to-be-processed image processing module is used for preprocessing and dicing the to-be-processed remote sensing image to obtain to-be-processed remote sensing image dicing;
The super-resolution reconstruction module is used for performing super-resolution reconstruction on the remote sensing image cut blocks to be processed by using the trained super-resolution reconstruction model;
And the remote sensing image generation module is used for splicing the reconstructed remote sensing images to be processed into blocks, adding coordinate information and generating new remote sensing images.
7. A remote sensing image super-resolution reconstruction device, comprising a processor and a memory, wherein the processor implements the remote sensing image super-resolution reconstruction method according to any one of claims 1 to 5 when executing a computer program stored in the memory.
8. A computer readable storage medium for storing a computer program, wherein the computer program when executed by a processor implements the remote sensing image super-resolution reconstruction method according to any one of claims 1 to 5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110818992.9A CN113516591B (en) | 2021-07-20 | 2021-07-20 | Remote sensing image super-resolution reconstruction method, device, equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110818992.9A CN113516591B (en) | 2021-07-20 | 2021-07-20 | Remote sensing image super-resolution reconstruction method, device, equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113516591A CN113516591A (en) | 2021-10-19 |
| CN113516591B true CN113516591B (en) | 2024-07-09 |
Family
ID=78067440
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110818992.9A Active CN113516591B (en) | 2021-07-20 | 2021-07-20 | Remote sensing image super-resolution reconstruction method, device, equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113516591B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115187463A (en) * | 2022-09-13 | 2022-10-14 | 云南省交通规划设计研究院有限公司 | Landslide remote sensing image set super-resolution reconstruction method and system |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111179172A (en) * | 2019-12-24 | 2020-05-19 | 浙江大学 | Remote sensing satellite super-resolution implementation method and device based on unmanned aerial vehicle aerial data, electronic equipment and storage medium |
| CN112270697A (en) * | 2020-10-13 | 2021-01-26 | 清华大学 | Satellite sequence image moving target detection method combined with super-resolution reconstruction |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113034361B (en) * | 2021-03-03 | 2022-10-14 | 深圳市数字城市工程研究中心 | Remote sensing image super-resolution reconstruction method based on improved ESRGAN |
-
2021
- 2021-07-20 CN CN202110818992.9A patent/CN113516591B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111179172A (en) * | 2019-12-24 | 2020-05-19 | 浙江大学 | Remote sensing satellite super-resolution implementation method and device based on unmanned aerial vehicle aerial data, electronic equipment and storage medium |
| CN112270697A (en) * | 2020-10-13 | 2021-01-26 | 清华大学 | Satellite sequence image moving target detection method combined with super-resolution reconstruction |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113516591A (en) | 2021-10-19 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109685842B (en) | Sparse depth densification method based on multi-scale network | |
| CN109508681B (en) | Method and device for generating human body key point detection model | |
| CN111127538B (en) | Multi-view image three-dimensional reconstruction method based on convolution cyclic coding-decoding structure | |
| CN102859535B (en) | Daisy descriptor is produced from precalculated metric space | |
| CN110264416A (en) | Sparse point cloud segmentation method and device | |
| CN108596108B (en) | Aerial remote sensing image change detection method based on triple semantic relation learning | |
| US10339707B2 (en) | Automated generation of digital elevation models | |
| WO2023000158A1 (en) | Super-resolution reconstruction method, apparatus and device for remote sensing image, and storage medium | |
| CN112257605B (en) | Three-dimensional target detection method, system and device based on self-labeling training sample | |
| CN108304593B (en) | Method for interactive display of paper map and electronic map | |
| CN103606188A (en) | Geographical information on-demand acquisition method based on image point cloud | |
| CN113610070A (en) | Landslide disaster identification method based on multi-source data fusion | |
| CN114758337A (en) | Semantic instance reconstruction method, device, equipment and medium | |
| CN115116049A (en) | Target detection method and device, electronic equipment and storage medium | |
| CN114926734B (en) | Solid waste detection device and method based on feature aggregation and attention fusion | |
| CN115565154A (en) | Feasible region prediction method, device, system and storage medium | |
| Mel et al. | Workflows for virtual reality visualisation and navigation scenarios in earth sciences | |
| CN113516591B (en) | Remote sensing image super-resolution reconstruction method, device, equipment and storage medium | |
| CN102506828A (en) | Geometric correction system of traffic accident scene map based on line features | |
| CN109829421B (en) | Method and device for vehicle detection and computer readable storage medium | |
| CN117132737B (en) | Three-dimensional building model construction method, system and equipment | |
| CN116543165B (en) | Remote sensing image fruit tree segmentation method based on dual-channel composite depth network | |
| CN111311732B (en) | 3D human body grid acquisition method and device | |
| CN117557983A (en) | Scene reconstruction method and driving assistance system based on depth forward projection and query back projection | |
| Zhou et al. | Real-time ortho-rectification for remote-sensing images |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |