CN109447089B

CN109447089B - High-resolution arctic sea ice type extraction method based on super-resolution technology

Info

Publication number: CN109447089B
Application number: CN201811203230.2A
Authority: CN
Inventors: 冯甜甜; 刘小敏
Original assignee: Tongji University
Current assignee: Tongji University
Priority date: 2018-10-16
Filing date: 2018-10-16
Publication date: 2021-09-14
Anticipated expiration: 2038-10-16
Also published as: CN109447089A

Abstract

The invention relates to a polar sea ice type remote sensing monitoring method for performing super-resolution reconstruction on a passive microwave remote sensing image of an arctic and performing high-resolution sea ice type extraction on the passive microwave remote sensing image by using a super-resolution technology, in particular to a high-resolution arctic sea ice type extraction method based on the super-resolution technology. The method is characterized in that the passive microwave image is subjected to hyper-resolution reconstruction, and then the sea ice type is extracted based on the obtained hyper-resolution image with higher resolution. The method utilizes a super-resolution reconstruction strategy to firstly carry out super-resolution reconstruction on the passive microwave remote sensing image of the polar sea ice to obtain an image with higher resolution, and then further realizes high-resolution extraction of the polar sea ice type on the basis.

Description

High-resolution arctic sea ice type extraction method based on super-resolution technology

Technical Field

The invention relates to a polar sea ice type remote sensing monitoring method which utilizes a super-resolution technology to carry out super-resolution reconstruction on a passive microwave remote sensing image of the north pole and carries out high-resolution sea ice type extraction on the image.

Background

The type of polar sea ice is one of the key parameters of a freezing circle and polar environment change, and plays a significant role in global climate change research. Therefore, the polar sea ice type is accurately extracted, the change trend of the polar sea ice is accurately described and reasonably predicted, and reliable data are provided for applications such as polar channel design.

The remote sensing data for sea ice monitoring mainly comprises optical image data, passive microwave data, SAR, satellite height measurement data and the like. The passive microwave data has the advantages of strong ground surface penetration capacity, all-weather work, wide coverage range and the like, overcomes the defect that optical images (such as MODIS) are influenced by cloud and weather, and provides an important data source for sea ice extraction particularly in the polar night of the south and north poles. However, the spatial resolution of passive microwave data is low (12.5 km-25 km), and high-resolution extraction of polar sea ice types is difficult to achieve.

The super-resolution reconstruction method is to obtain a high-resolution image by eliminating frequency-frequency aliasing in a frequency domain or constructing a motion degradation model in a spatial domain by using a low-resolution image, and has wide application in the field of remote sensing.

Disclosure of Invention

The invention aims to overcome the defects of the prior art and provides a high-resolution extraction method of an arctic sea ice type based on a passive microwave image. The method utilizes a super-resolution reconstruction strategy to firstly carry out super-resolution reconstruction on the passive microwave remote sensing image of the polar sea ice to obtain an image with higher resolution, and then further realizes high-resolution extraction of the polar sea ice type on the basis.

In order to solve the above-mentioned invention task, the technical scheme adopted by the invention is as follows:

a high-resolution arctic sea ice type extraction method based on a super-resolution technology is characterized in that passive microwave images are subjected to super-resolution reconstruction, and then sea ice types are extracted based on the obtained super-resolution images with higher resolution. The passive microwave image is an AMSR2 image, and the resolution of the original image is10 km; the super-resolution reconstruction is based on a super-resolution reconstruction method for generating a countermeasure network, and the resolution of an original image can be improved by 4 times to obtain an image with the resolution of 2.5 km; the sea ice types are annual ice and perennial ice; the sea ice type extraction is that sea ice type extraction is carried out on the over-separation result of the passive microwave of the arctic sea ice by a deep learning method based on semantic segmentation, and the sea ice distribution of one year ice and multiple years ice in a research area is obtained.

The specific implementation steps are as follows:

step one, passive microwave image data preparation

The passive microwave image data is a data product of a passive microwave scanning radiometer (AMSR2) on a GCOM-W polar orbit satellite platform in a Global Change Observation Mission (GCOM) of Japan aviation research and development organization (JAXA).

Step two, passive microwave image hyper-resolution reconstruction

Step 2.1, building a deep learning framework: ubuntu16.4+ NVIDIA GTX 1080Ti GPUs + Python2.7+ CUDA8.0+ Cudnn5.1+ Tensorflow 1.2.

Step 2.2, training sample preparation

The method comprises the following specific steps:

(1) converting the original passive microwave data downloaded in the step one into three-channel RGB images in batch;

(2) the RGB images obtained in the step (1) can be overlapped and cut into images with a certain pixel size, and the requirement that each image covers a sea ice area is met;

step 2.3, hyper-resolution model training

The generation of a countering network (SRGAN, a deep network model, set as model 1) is applied to passive microwave imagery for hyper-resolution reconstruction in order to fool a discriminator with resolution capability, which is trained to distinguish between real images and super-resolution images. Based on the method, the generator can learn to generate a result which is highly similar to a real image, so that the result is difficult to distinguish by a discriminator, and the optimal solution in the subspace is a natural image.

The specific training process is as follows:

(1) setting network parameters: the initial weight is a VGG19 network initial weight, the learning rate is 0.0001, the iteration times are 40000, the batch processing image parameter is set to be 16, and the up-sampling factor is 4;

(2) acquiring a low-resolution image: inputting a high-resolution image (denoted as HR) by a network, namely a passive microwave image with the resolution of 10km in the step 2.2, and performing down sampling on the passive microwave image by four times at the beginning of training to obtain a corresponding low-resolution image (denoted as LR) with the resolution of 40 km;

(3) training a generator: inputting the low-resolution image LR in the step (2) as a generation network into a training generator for super-resolution reconstruction, and outputting a corresponding super-resolution image SR through the network;

(4) training a discriminator: inputting the super-resolution image SR output by the generator in the step (3) as a discrimination network to discriminate whether the images are passive wavelet images HR in the step (2), and outputting the probability value for judging whether the input images are high-resolution images;

(5) and (3) model alternating optimization training: network parameters of the generator model (3) and the discriminator model (4) are alternately optimized and trained, and the two models can be improved until the generator generates an image which cannot be distinguished from a natural image (high-resolution image) through the learning of an optimal discriminator, namely the training is finished.

Step 2.4, the hyper-resolution model test:

(1) and (3) super-resolution reconstruction of low-resolution images: and (3) inputting the passive microwave image downloaded in the step one based on the network model trained in the step 2.3, wherein the original spatial resolution is10 km, and obtaining a super-resolution image with the spatial resolution of 2.5km under the condition of a quadruple up-sampling factor.

(2) And (5) carrying out qualitative evaluation on the over-separation result.

(3) And (3) quantitative evaluation of the overdue result:

1) and selecting the evaluation standard of the hyper-score result, wherein the evaluation standard comprises a peak signal to noise ratio (PNSR) and a Structural Similarity (SSIM), and the larger the value of the evaluation standard is, the more the hyper-score test result tends to the real situation of the image.

2) Low/high resolution image pair acquisition

And (4) inputting the passive microwave image obtained in the first step as a high-resolution image, and inputting the quadruple passive microwave image as a low-resolution image.

3) Inputting the low-score image obtained in the last step into the SRGAN network model trained in the step 2.5, and performing quantitative analysis by adopting two evaluation indexes of PNSR and SSIM.

Step three, sea ice type extraction based on super-resolution images

Step 3.1, arctic sea ice type: relates to three types of annual ice, perennial ice and ice-free areas.

Step 3.2, training sample preparation: the FCN samples are divided into two types, namely, a super-resolution image and a corresponding labeled graph with semantic labels. The preparation method specifically comprises the following steps:

(1) obtaining a super-resolution image: applying the passive microwave image downloaded in the step one to the processing method in the step 2.2 to obtain an RGB three-channel image, and then outputting an ultra-resolution image with a certain resolution after ultra-resolution reconstruction based on the ultra-resolution model (model 1) trained in the step 2.3;

(2) obtaining a label graph: labeling is carried out based on three types, which are respectively as follows: annual ice, perennial ice and ice free areas. The software is Arcgis10.2, and the specific steps are as follows:

1) coordinate transformation, namely transforming the AARI Sea Ice distribution diagram coordinate system to a coordinate system NSIDC _ Sea _ Ice _ Polar _ Stereographic _ North used in the step I of the passive microwave image, and recording the coordinate system as AARI.

2) Creating a new surface element, which is recorded as background, wherein the range size is consistent with the passive microwave image in the step I, and the coordinate system is also consistent;

3) vector transformation grating, namely rasterizing aari.shp and background.shp obtained in the previous step, setting the converted resolution to be 2500m as same as the spatial resolution of the super-resolution image, namely the size of the converted resolution is 3040 multiplied by 4480 pixels, and outputting TIFF files in the format of aari.GIF and background.GIF respectively;

4) classifying ari.GIF, referring to the attribute table and the frequency histogram, wherein the total number of the attribute table and the frequency histogram has six types of values which correspond to six sea ice types and are output as ari _ coast.GIF;

5) layer embedding, namely merging layers of background.GIF and ari _ coast.GIF by using a Mosaic tool, and outputting the merged layers as ari _ new.GIF;

6) image reclassification, wherein six sea ice types are totally extracted from ari _ new.GIF, and the extraction targets of the sea ice types are as follows: the corresponding values of the three types of ice-free areas, annual ice and perennial ice are 0, 1 and 2, and the output is ari _ new _ retrieval.GIF;

7) and (4) label graph derivation, namely deriving ari _ new _ retrieval. tif, storing the ari _ new _ retrieval. tif in png format, wherein the resolution of the derivative graph is consistent with that of the super-resolution image and is 2.5 km.

(3) Image cutting: and respectively overlapping and cutting the super-resolution image and the corresponding label graph.

Step 3.3, training a semantic segmentation model: the sea ice type is extracted by applying a semantic segmentation method of a full convolution neural network (FCN, a deep network model, the invention is set as a model 2). The FCN introduces an end-to-end full convolutional network, namely, all three full connection layers in the traditional Convolutional Neural Network (CNN) are converted into convolutional layers, so that the calculation is more efficient.

Step 3.4, testing a semantic segmentation model: and (4) inputting the hyper-resolution image in the step 3.2 based on the network model trained in the step 3.3 to obtain an arctic sea ice type distribution map.

(1) And (5) qualitatively evaluating the classification result.

(2) And (5) quantitatively analyzing the classification result.

Drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below. It is obvious that the drawings in the following description are only some embodiments of the invention, and that for a person skilled in the art, other drawings can be derived from them without inventive effort.

Fig. 1 is a general flow diagram.

FIG. 2 is a flow chart of step two.

FIG. 3 is a flow chart showing three steps.

Figure 4 step 3.2 flow chart.

Figure 5 hyper-resolution reconstruction results. (a) The low resolution image is input as a passive microwave image with a resolution of 10 km. (b) And (d) respectively enlarging corresponding areas of a green frame and a red frame in the input image in the super-resolution reconstruction result. (c) And (e) the corresponding images after the super-resolution, the resolution is 2.5 km.

FIG. 6AARI sea ice distribution plot.

FIG. 7 sea Ice type extraction results: (a) the passive microwave hyperspectral image (b) is a classification true value (c) and the classification result (d) is a classification error map.

Detailed Description

The preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings so that the advantages and features of the present invention can be more easily understood by those skilled in the art, and the scope of the present invention will be more clearly and clearly defined.

As shown in fig. 1, the overall process of the embodiment of the present invention includes two parts: the method comprises the following specific implementation steps of hyper-resolution reconstruction of passive microwave images and sea ice type extraction based on the hyper-resolution images:

step one, passive microwave image data preparation

The passive microwave image data used in this embodiment is a data product of a passive microwave scanning radiometer (AMSR2) on a GCOM-W polar satellite platform in the japan aviation research and development organization (JAXA) Global Change Observation Mission (GCOM), and can be downloaded freely. The downloaded product data is AMSR2 bright temperature product with level 3 horizontal polarization of 36.5GHz, image size 760X 1120 pixels, spatial coverage in the northern hemisphere, and spatial resolution of 10 km.

Step two, passive microwave image hyper-resolution reconstruction

Step 2.2, training sample preparation: deep learning requires a large number of external data sets to train the model, and ensures training efficiency, and the specific steps are as follows:

(1) the original passive microwave data downloaded in the step one is a 16-bit single-channel gray image with the size of 760 x 1120 pixels, and is converted into a three-channel RGB image with the size of 760 x 1120 x 3 in batches;

(2) the RGB images obtained in the step (1) can be overlapped and cut into images with the size of 400 x 400 pixels, and the requirement that each image covers a sea ice area is met;

the final training set was 912 sheets and the test set was 72 sheets.

Step 2.3, training a hyper-resolution model: this embodiment applies a generation countermeasure network (SRGAN, a deep network model, set as model 1) to the passive microwave image for the purpose of performing a super-resolution reconstruction, in order to fool a discriminator with resolution capability that is trained to discriminate between true images and super-resolution images. Based on the method, the generator can learn to generate a result which is highly similar to a real image, so that the result is difficult to distinguish by a discriminator, and the optimal solution in the subspace is a natural image. The specific training process is as follows:

Step 2.4, the hyper-resolution model test:

(2) And (3) carrying out qualitative evaluation on the over-separation result: fig. 5 is a result of the super-resolution reconstruction, comparing the original low-resolution image with the higher-resolution image obtained after the super-resolution reconstruction, the super-resolution image obtained by the SRGAN is finer, and the texture is restored clearly.

(3) And (3) quantitative evaluation of the overdue result:

2) For obtaining the low/high-resolution images, the current experimental data AMSR2 is the best quality in the passive microwave data, and the passive microwave data with high resolution cannot be found for a while to quantitatively analyze the hyper-resolution images with resolution improved to 2.5 km. Therefore, the passive microwave image with the resolution of 10km obtained in the first step is used as a high-resolution image input, and the image is down-sampled four times by a bicubic interpolation method, so that a 40km image is obtained and used as a low-resolution image input.

3) Inputting the low-resolution image (40km) obtained in the last step into the SRGAN network model trained in the step 2.5, and performing quantitative analysis by adopting two evaluation indexes of PNSR and SSIM, as shown in the results in Table 1, the reconstructed image has higher signal-to-noise ratio and structural similarity.

Step three, sea ice type extraction based on super-resolution images

Step 3.1, arctic sea ice type: in polar regions, two sea ice types, namely annual ice and perennial ice, are relatively representative and are main influence factors, and in large-scale and remote sensing monitoring means, the sea ice types are main sea ice types, so that the extraction of the sea ice types mainly relates to three types of annual ice, perennial ice and ice-free areas.

Step 3.2, training sample preparation: the FCN samples are divided into two categories, namely, the super-resolution image and the label graph with semantic labels, as shown in fig. 4. The preparation method specifically comprises the following steps:

(1) obtaining a super-resolution image: applying the passive microwave image (with the resolution of 10km) downloaded in the step one to the processing method in the step 2.2 to obtain an RGB three-channel image, and then outputting a hyper-resolution image with the resolution of 2.5km after hyper-resolution reconstruction based on the hyper-resolution model (model 1) trained in the step 2.3;

(2) obtaining a label graph: for the output super-resolution image in (1), class labeling is performed pixel by pixel, and the labeling is performed mainly based on three types, which are: annual ice, perennial ice and ice free areas. The annotated map is a sea ice product published by the institute for north and south poles of the russian federal national science center with AARI sea ice distribution diagram, the time resolution is 7 days, the north sea ice region is basically covered, and the data format is Shapefile, as shown in fig. 6. The software is Arcgis10.2, and the specific steps are as follows:

(3) Image cutting: the super-resolution image with size of 3040 × 4480 pixels and the corresponding labeled graph can be respectively cut into 2048 × 2048 pixels in an overlapping manner, and the final sample number is 420 training sets and 42 testing sets.

Step 3.3, training a semantic segmentation model: this embodiment applies a semantic segmentation method of a full convolutional neural network (FCN, a deep network model, the present invention is set as model 2) to extract the sea ice type. The FCN introduces an end-to-end full convolutional network, namely, all three full connection layers in the traditional Convolutional Neural Network (CNN) are converted into convolutional layers, so that the calculation is more efficient. The method comprises the following specific steps:

(1) setting network parameters, wherein the initial weight is the initial weight of the VGG19 network, the learning rate is 0.0001, the iteration times are 100000 times, and the batch processing image parameters are set to be 2. (network parameters finally determined in the embodiment case)

(2) Inputting the data, namely inputting the hyper-resolution images processed in the step 3.2 and the corresponding labeled graphs into a network

(3) Model training, namely performing FCN network model training, wherein one time is about one day.

Step 3.4, testing a semantic segmentation model: based on the network model trained in step 3.3, the hyper-resolution image in step 3.2 is input to obtain an arctic sea ice type distribution map, as shown in fig. 7- (c).

(1) And (3) qualitatively evaluating a classification result: comparing the manual labeling chart (fig. 7- (b)) and the test classification result chart (7- (c)) in the experimental result and the classification error chart (fig. 7- (d)) obtained by making a difference between fig. 7- (b) and 7- (c), the distribution of ice in a large range of one year and multiple years can be accurately obtained based on the FCN.

(2) And (4) quantitatively analyzing classification results: as shown in table 2, the overall accuracy of the experimental result is about 93%, and the Kappa coefficient reaches 0.86, which indicates that the classification result is highly consistent with the true value on the whole, and further indicates that a scheme for implementing high-resolution sea ice monitoring by using the super-resolution technology is feasible.

TABLE 1 evaluation results of the accuracy of the super-resolution reconstructed images

TABLE 2 confusion matrix and segmentation accuracy

The above description is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiment, and all technical solutions belonging to the principle of the present invention belong to the protection scope of the present invention. It will be apparent to those skilled in the art that various modifications and adaptations can be made without departing from the principles of the invention and these are intended to be within the scope of the invention.

Claims

1. A high-resolution arctic sea ice type extraction method based on a super-resolution technology is characterized in that a passive microwave image is subjected to super-resolution reconstruction, and then a sea ice type is extracted based on a super-resolution image with a higher resolution;

the specific implementation steps are as follows:

step one, passive microwave image data preparation

Step two, passive microwave image hyper-resolution reconstruction

Step 2.1, deep learning framework construction

Step 2.2, training sample preparation

The method comprises the following specific steps:

step 2.3, hyper-resolution model training

The method comprises the steps that an antagonistic network model SRGAN is generated and is used for performing super-resolution reconstruction on a passive microwave image, and the purpose is to deceive a discriminator with resolution capability, wherein the discriminator is trained to distinguish a real image or a super-resolution image; based on the method, the generator learns to generate a result which is highly similar to a real image, so that the result is difficult to distinguish by a discriminator, and an optimal solution in a subspace is a natural image;

step 2.4, the hyper-resolution model test:

(1) and (3) super-resolution reconstruction of low-resolution images: inputting the passive microwave image downloaded in the step one based on the network model trained in the step 2.3, wherein the original spatial resolution is10 km, and obtaining a super-resolution image with the spatial resolution of 2.5km under the condition of a quadruple up-sampling factor;

(2) carrying out qualitative evaluation on the over-separation result;

(3) and (3) quantitative evaluation of the overdue result:

1) selecting the evaluation standard of the hyper-resolution result, wherein the evaluation standard comprises a peak signal-to-noise ratio PNSR and a structural similarity SSIM, and the larger the value of the evaluation standard is, the more the hyper-resolution test result tends to the real situation of the image;

2) low/high resolution image pair acquisition

Inputting the passive microwave image obtained in the first step as a high-resolution image, and inputting the quadruple passive microwave image as a low-resolution image;

3) inputting the low-score image obtained in the last step into the SRGAN trained in the step 2.5, and carrying out quantitative analysis by adopting two evaluation indexes of PNSR and SSIM;

step three, sea ice type extraction based on super-resolution images

Step 3.1, arctic sea ice type: three types of annual ice, perennial ice and ice-free areas are involved;

step 3.2, training sample preparation: the FCN samples are divided into two types, namely, the super-resolution images and the corresponding labeled graphs with semantic labels; the preparation method specifically comprises the following steps:

(1) obtaining a super-resolution image: applying the passive microwave image downloaded in the step one to the processing method in the step 2.2 to obtain an RGB three-channel image, and then outputting the RGB three-channel image as a super-resolution image with a certain resolution after super-resolution reconstruction based on the super-resolution model trained in the step 2.3;

(2) obtaining a label graph: labeling is carried out based on three types, which are respectively as follows: annual ice, perennial ice and ice free areas,

(3) image cutting: respectively cutting the super-resolution image and the corresponding label graph in an overlapping manner;

step 3.3, training a semantic segmentation model: extracting sea ice types by applying a semantic segmentation method of a full convolution neural network (FCN); the FCN introduces an end-to-end full convolutional network, namely, all three full connection layers in the traditional convolutional neural network CNN are converted into convolutional layers, so that the calculation is more efficient; training a network model;

step 3.4, testing a semantic segmentation model: inputting the super-resolution image in the step 3.2 based on the network model trained in the step 3.3 to obtain an arctic sea ice type distribution map;

(1) qualitatively evaluating the classification result;

(2) and (5) quantitatively analyzing the classification result.

2. The method for extracting the type of the arctic sea ice with high resolution based on the hyper-resolution technology as claimed in claim 1, wherein the passive microwave image data preparation in the first step:

the passive microwave image data is a data product of a passive microwave scanning radiometer AMSR2 on a GCOM-W polar orbit satellite platform in a JAXA global change observation task GCOM of Japan aviation research and development organization.

3. The method for extracting the type of the high-resolution arctic sea ice based on the super resolution technology as claimed in claim 1, wherein in step 2.3, the specific training process of the super resolution model training is as follows:

step 2.3.1 network parameter setting: the initial weight is a VGG19 network initial weight, the learning rate is 0.0001, the iteration times are 40000, the batch processing image parameter is set to be 16, and the up-sampling factor is 4;

step 2.3.2 obtaining low-resolution images: inputting a high-resolution image by a network, recording the high-resolution image as HR (high resolution ratio), namely a passive microwave image with the resolution of 10km in the step 2.2, starting training to down-sample the passive microwave image by four times to obtain a corresponding low-resolution image, recording the low-resolution image as LR (low resolution ratio), and recording the low-resolution image as 40 km;

step 2.3.3 generator training: inputting the low-score image LR in the step 2.3.2 as a generation network into a training generator for super-resolution reconstruction, and outputting a corresponding super-score image SR by the network;

step 2.3.4 discriminant training: the super-resolution image SR output by the generator in the step 2.3.3 is input as a discrimination network to discriminate whether the images are passive micro-shadow HR in the step 2.3.2, and the output is a probability value for judging whether the input image is a high-resolution image;

step 2.3.5 model alternating optimization training: the network parameters of the generator model step 2.3.3 and the discriminator model step 2.3.4 are alternately optimized and trained, and both models can be improved until the generator generates an image which cannot be distinguished from a natural image through the learning of an optimal discriminator, namely the training is finished.

4. The method for extracting the type of the high-resolution arctic sea ice based on the hyper-resolution technology according to claim 1, wherein the label graph is obtained by: the method adopts the labeling software Arcgis10.2, and comprises the following specific steps: