WO2023000505A1

WO2023000505A1 - Two-order lightweight network panchromatic sharpening method combining guided filtering and nsct

Info

Publication number: WO2023000505A1
Application number: PCT/CN2021/122464
Authority: WO
Inventors: 黄梦醒; 吴园园; 李玉春; 冯思玲; 毋媛媛; 吴迪
Original assignee: 海南大学
Priority date: 2021-07-19
Filing date: 2021-09-30
Publication date: 2023-01-26
Also published as: CN113643197B; CN113643197A

Abstract

Provided in the present invention is a two-order lightweight network panchromatic sharpening method combining guided filtering and NSCT. The advantage of retaining better edge and detail information by guided filtering and the advantage of multi-scale multi-directional decomposition of NSCT are combined with a CNN, so as to construct a two-order lightweight network model to fuse an MS image and a PAN image, wherein by means of guided filtering, the panchromatic image MLPAN, which has been subjected to histogram matching, is filtered, so as to obtain a multi-scale high-frequency component and low-frequency component; by means of NSCT, an I-component image extracted from the MS image is filtered, so as to obtain a multi-scale multi-directional high-frequency directional sub-band image and low-frequency sub-band image; a detail extraction network ResCNN is constructed by using the advantage of a residual module, so as to extract in-details; and finally, a nonlinear model NLCNN is constructed by means of taking the in-details and a DUMS image as inputs, and the NLCNN is fully trained to obtain an optimal model. By means of the method, spectral information is retained while spatial resolution can be increased to a greater extent, the network structure is simple, the training time is reduced, an over-fitting phenomenon is prevented from occuring, and the fusion performance is improved.

Description

A Two-Stage Lightweight Network Panchromatic Sharpening Method Combining Guided Filtering and NSCT

technical field

The invention relates to the technical field of remote sensing image processing, in particular to a two-stage lightweight network panchromatic sharpening method combined with guided filtering and NSCT.

Background technique

Remote sensing images are widely used in various industries, such as yield prediction, forestry pest detection, forest natural disaster prediction, geological exploration, national security, land use, environmental change detection, etc., but limited by satellite sensor technology, it is impossible to obtain Images with high spatial resolution and high spectral resolution can only obtain panchromatic images (PAN) with high spatial resolution and low spectral resolution and multispectral images (MS) with low spatial resolution and high spectral resolution. However, in practice In applications, images with both high spatial resolution and high spectral resolution are often required, and even images with high temporal resolution are required. At present, the commonly used method is to use the redundant and complementary information of PAN and MS images to obtain high spatial resolution High spectral resolution images (HSHM) can generally be realized by the following technologies: image enhancement, super-resolution reconstruction, image fusion, etc., among which the mainstream research technology is image fusion technology, which refers to the multi-source image through a certain The method generates a higher-quality, more informative image that matches people's visual perception, so that decision makers can make more precise decisions with clearer images.

The fusion of MS images and PAN images, also known as panchromatic sharpening, is one of the hot spots and focuses in the field of remote sensing image processing. Fusion methods can be summarized as component replacement methods, multi-resolution analysis methods, variational methods, and deep learning. Component replacement methods, such as IHS, GIHS, AIHS, PCA, Brovey, GS, etc., although these methods can improve spatial resolution, they generally have different degrees of distortion of spectral information; multi-resolution analysis methods such as wavelet (wavelet) transformation, pull Laplacian Pyramid decomposition (Laplacian Pyramid, LP), contourlet (contourlet) transformation, curvelet (curvelet) transformation, non-subsampling contourlet transformation (NSCT) (such as the multi-focus image fusion algorithm based on NSCT whose publication number is CN103632353A ) and so on to reduce spectral distortion to a certain extent, but the spatial resolution is relatively low, and artifacts may also occur; the rapid development of deep learning in the field of computer vision has made various networks begin to be applied in the direction of remote sensing image fusion, such as PNN , PCNN (such as the image fusion method based on gradient domain-guided filtering and improved PCNN whose publication number is CN112184646A), DRPNN, PanNet, PanGAN and other networks have achieved certain effects for panchromatic sharpening, but there will still be spectral distortion , Low spatial resolution, low fusion quality, overfitting, and long training time.

Contents of the invention

In view of this, the present invention proposes a two-stage lightweight network panchromatic sharpening method combining guided filtering and NSCT, which preserves spectral information while improving spatial resolution, and has high fusion quality, and the two-stage lightweight network Simple, short training time, prevent overfitting phenomenon.

The present invention is achieved through the following technical solutions:

A two-stage lightweight network panchromatic sharpening method combining guided filtering and NSCT, including the following steps:

Step S1, obtaining satellite remote sensing images, and preprocessing the MS images and PAN images in the remote sensing images;

Step S2, according to the Wald criterion, the preprocessed MS image and PAN image are subjected to resolution reduction processing, and a simulation training set, a simulation test set and a real test set are constructed, wherein the simulation training set and the simulation test set include DUMS images and LPAN images As well as MS images, the real test set includes UMS images and PAN images;

Step S3, using AIHS transformation on the DUMS image in the simulation training set to obtain the brightness I component image, and using the I component image to perform histogram equalization processing on the LPAN image to obtain the MLPAN image;

Step S4, filtering the MLPAN image by using a guided filter to obtain multi-scale high-frequency component MLPAN _Hn and low-frequency component MLPAN _Ln ;

Step S5, using NSCT to filter the I component image to obtain multi-scale and multi-directional high-frequency direction sub-band images I _Hn and low-frequency sub-band images I _Ln ;

Step S6, constructing a detail extraction network ResCNN according to the DUMS image, the MLPAN image, the high-frequency component MLPAN _Hn , the low-frequency component MLPAN _Ln , the high-frequency direction sub-band image I _Hn and the low-frequency sub-band image I _Ln , and obtain injection details In-details;

Step S7, inject details In-details and DUMS images as the input of the shallow CNN network, MS images as the output, establish a nonlinear model NLCNN network, fully train the NLCNN network, obtain the optimal nonlinear model, and use the optimal non-linear The parameters of the linear model are frozen, and the pan-sharpened image is obtained using the optimal nonlinear model.

Preferably, the preprocessing in step S1 includes: atmospheric correction and spatial registration.

Preferably, the specific steps of said step S2 include:

Step S21, according to the Wald criterion and the spatial resolution ratio between the panchromatic image and the multispectral image, the MS image and the PAN image are down-sampled using the bicubic interpolation method, and the reduced-resolution LPAN image and the DMS image are obtained;

Step S22. Upsampling the DMS image using a bicubic interpolation method according to the Wald criterion, and obtaining a DUMS image;

Step S23, using bicubic interpolation method to upsample the MS image according to the Wald criterion, and obtain the UMS image;

Step S24, constructing a simulation training set and a simulation test set from DUMS images, LPAN images and MS images, and constructing a real test set from UMS images and PAN images.

Preferably, the AIHS transformation in the step S3 obtains the expression of the I component image as:

Where i is the i-th channel, a _i is the adaptive coefficient, and N is the total number of channels.

Preferably, the specific steps of said step S4 are: use a guide filter to filter the MLPAN image, the input image of the guide filter is an MLPAN image, and the guide image is an I component image, and the low frequency component MLPAN _i =GF( MLPAN _i-1 , I), wherein GF is a guide filter, and MLPAN _i-1 is the output image of the i-1 filtering, when i=1, it is an MLPAN image, then the i low-frequency component MLPAN _Li = MLPAN _i , the i-th high-frequency component MLPAN _Hi =MLPAN _Li-1 −MLPAN _Li , n high-frequency components MLPAN _Hn and n low-frequency components MLPAN _Ln are obtained after n times of filtering.

Preferably, the NSCT in step S5 includes a non-downsampling pyramid filter bank NSPFB and a non-downsampling directional filter bank NSDFB.

Preferably, the specific steps of said step S5 include:

Step S51, using NSPFB to decompose the I component image to obtain the low-frequency sub-band image I _Li and the high-frequency sub-band image I _Hi ;

Step S52, using NSPFB to decompose the low-frequency sub-band image, and obtain the low-frequency sub-band image and the high-frequency sub-band image of the next layer;

Step S53, using NSDFB to filter the high-frequency sub-band images of each layer respectively, to obtain the high-frequency direction sub-band images of each layer.

Preferably, the specific steps of said step S6 include:

Step S61, using the DUMS image, the MLPAN image, the high frequency component MLPAN _Hn , the low frequency component MLPAN _Ln , the high frequency direction subband image I _Hn and the low frequency subband image I _Ln as the input of the ResCNN network;

Step S62, using the details of the difference between the DUMS image and the MS image as a label;

Step S63, train the ResCNN network, and after minimizing the loss function, freeze the training parameters to obtain the optimal model;

Step S64, obtaining injection details In-details according to the optimal model.

Preferably, the specific steps of said step S7 include:

Step S71, using the In-details and DUMS images injected as the input of the nonlinear model NLCNN network;

Step S72, using the MS image as a label;

Step S73, train the above network, and after minimizing the loss function, freeze the training parameters to obtain the optimal nonlinear model;

Step S74, using the optimal nonlinear model to obtain a pan-sharpened image.

Compared with prior art, the beneficial effect of the present invention is:

The present invention provides a two-stage lightweight network panchromatic sharpening method combining guided filtering and NSCT, effectively combining guided filtering and NSCT, wherein guided filtering is used to extract multi-scale high-frequency components and low-frequency components of MLPAN images, Can maintain edge features; use NSCT to extract multi-scale and multi-directional high-frequency sub-band images and low-frequency sub-band images of I-component images, and then use ResCNN's residual characteristics and nonlinear characteristics to extract richer detail information and construct shallow layers The network is easy to train and prevents overfitting; since the relationship between DUMS images and LPAN images is nonlinear, the nonlinearity of the shallow CNN network is used to inject details and DUMS images for training to obtain the final fusion result. The network designed by the invention is composed of a two-order lightweight network, the network is relatively simple, easy to train, prevents overfitting, has strong generalization ability, and preserves spectral information while improving spatial resolution.

Description of drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the drawings that need to be used in the description of the embodiments. Obviously, the drawings in the following description are only preferred embodiments of the present invention. For those skilled in the art, other drawings can also be obtained based on these drawings without any creative effort.

Fig. 1 is a flow chart of a two-stage lightweight network panchromatic sharpening method combining guided filtering and NSCT of the present invention;

Fig. 2 is a schematic diagram of NSCT filtering of a two-order lightweight network panchromatic sharpening method combining guided filtering and NSCT of the present invention;

FIG. 3 is a structural schematic diagram of a two-stage lightweight network panchromatic sharpening method ResCNN combining guided filtering and NSCT according to the present invention.

detailed description

In order to better understand the technical content of the present invention, a specific embodiment is provided below, and the present invention is further described in conjunction with the accompanying drawings.

Referring to Fig. 1, a two-stage lightweight network panchromatic sharpening method combining guided filtering and NSCT provided by the present invention includes the following steps:

Step S1, obtaining Landsat-8, Landsat-7, Quickbird, GF-2 satellite remote sensing original images, the remote sensing original images include MS images and PAN images, and preprocessing the MS images and PAN images in the remote sensing images, the preprocessing includes Atmospheric correction and spatial registration.

Step S2, according to the Wald criterion, the preprocessed MS image and the PAN image are reduced in resolution, and a simulation training set, a simulation test set and a real test set are constructed, wherein the simulation training set and the simulation test set include down-sampling The multispectral image DUMS, reduced resolution panchromatic image LPAN and multispectral image MS, the real test set includes upsampled multispectral image UMS image and panchromatic image PAN image, the specific steps include:

Step S22: Upsampling the DMS image using a bicubic interpolation method according to the Wald criterion, and obtaining a DUMS image, wherein the size of the DUMS image is the same as that of the LPAN image;

Step S23: Upsampling the MS image using the bicubic interpolation method according to the Wald criterion, and obtaining a UMS image, the size of the UMS image is the same as the size of the PAN image;

The present invention uses Landsat-8 satellite's DUMS image, LPAN image, MS image as simulation training set, in order to verify the performance of the present invention better, use the DUMS of Landsat-8, Landsat-7, Quickbird and GF-2 four satellites Images, LPAN images, and MS images are used as simulation test sets, and MS images and PAN images are used as real test sets.

Step S3, using AIHS transformation on the DUMS image in the simulation training set to obtain the brightness I component image, and using the I component image to carry out histogram equalization processing on the LPAN image to obtain the MLPAN image, wherein the expression of the AIHS transformation to obtain the I component image is:

Step S4: Filter the MLPAN image by using a guided filter to obtain multi-scale high-frequency component MLPAN _Hn and low-frequency component MLPAN _Ln , the specific steps are:

Use guide filter to filter MLPAN image, the input image of guide filter is MLPAN image, guide image is I component image, after filtering, obtain low frequency component MLPAN _i =GF(MLPAN _i-1 , I), wherein GF is guide Filter, MLPAN _i-1 is the output image of the i-1th filter, when i=1, MLPAN _i-1 is MLPAN, then the i-th low-frequency component MLPAN _Li =MLPAN _i , the i-th high-frequency component MLPAN _Hi =MLPAN _Li−1 −MLPAN _Li , n high frequency components MLPAN _Hn and n low frequency components MLPAN _Ln are obtained after n times of filtering.

Step S5, use NSCT to filter the I component image to obtain multi-scale and multi-directional high-frequency direction sub-band image I _Hn and low-frequency sub-band image I _Ln , wherein NSCT includes non-downsampling pyramid filter bank NSPFB and non-downsampling direction Filter bank NSDFB, as shown in Figure 2, the low-pass filter of NSPFB includes low-pass decomposition filter and low-pass reconstruction filter {D ₀ (X), D ₁ (X)}, the high-pass filter of NSDFB includes High-pass decomposition filter and high-pass reconstruction filter {G ₀ (X), G ₁ (X)}, NSPFB satisfies the Bezout identity 1D polynomial function:

The sector filter of the NSDFB includes a sector decomposition filter and a sector reconstruction filter {C ₀ (X), C ₁ (X)}, and the checkerboard filter of the NSDFB comprises a checkerboard decomposition filter and a checkerboard reconstruction filter {Q ₀ (X),Q ₁ (X)}, NSDFB satisfies the Bezout identity 1D polynomial function:

The specific steps of step S5 include:

Step S6, construct a detail extraction network ResCNN according to the DUMS image, MLPAN image, high-frequency component MLPAN _Hn , low-frequency component MLPAN _Ln , high-frequency direction sub-band image I _Hn and low-frequency sub-band image I _Ln , and obtain injection details In-details, Specific steps include:

Step S61, with DUMS image, MLPAN image, high-frequency component MLPAN _Hn , low-frequency component MLPAN _Ln , high-frequency direction sub-band image I _Hn and low-frequency sub-band image I _Ln as the input of ResCNN network, as shown in Figure 3, ResCNN network Consists of 2 layers of convolution, each layer is first normalized BN operation, then use ReLu function for nonlinear activation, and then perform convolution operation, the size of the convolution kernel is 3×3, and the convolution size of the direct connection part is 1×1;

Step S64, according to the optimal model to obtain richer detailed features, that is, to inject details In-details.

Step S7, use the injected details In-details and DUMS images as the input of the shallow CNN network, and the MS image as the output, establish a nonlinear model NLCNN network, fully train the NLCNN network, obtain the optimal nonlinear model, and use the optimal non-linear The parameters of the linear model are frozen, and the pan-sharpened image is obtained using the optimal nonlinear model. In this embodiment, the NLCNN network is composed of a single-layer CNN, which performs convolution operation first, then BN processing, and finally uses the ReLu activation function for activation, where a 1×1×n convolution kernel is used, and n is the output MS image The number of channels, 3 channels are used in this embodiment, the convolution kernel is 1×1×3, and 1×1 is the size of the convolution kernel.

The NLCNN network convolutional layer is expressed as:

MS=max(0,W _i *(DUMS,InD)+B _i );

Where W _i is the convolution kernel, _InD is the injected detail, and Bi is the bias.

The specific steps of step S7 include:

Step S72, using the MS image as a label;

Step S74, using the optimal nonlinear model to obtain a pan-sharpened image.

The present invention provides an embodiment to discuss the effectiveness, using the remote sensing image acquired by the Landsat-8 satellite sensor, wherein the spatial resolution of the multispectral image is 30 meters, and the pixel size is 600×600; the corresponding resolution of the panchromatic image is 15 meters , the pixel size is 1200×1200, according to the Wald criterion, the panchromatic image with a spatial resolution of 15 meters and the multispectral image with a spatial resolution of 30 meters are down-sampled by a factor of 2 to obtain a panchromatic image of 30 meters and a multispectral image of 60 meters. Use 7 methods (Indusion, NSCT, SFIM, MTF_GLP, PNN, DRPNN, PanNet) to compare with a two-stage lightweight network panchromatic sharpening method combined with guided filtering and NSCT of the present invention, whether it is to reduce the resolution The experimental results also at full resolution can show that the fusion effect of the method proposed by the present invention is better.

The above descriptions are only preferred embodiments of the present invention, and are not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention shall be included in the scope of the present invention. within the scope of protection.

Claims

A two-order lightweight network panchromatic sharpening method combining guided filtering and NSCT, characterized in that it comprises the following steps:

Step S1, obtaining satellite remote sensing images, and preprocessing the MS images and PAN images in the remote sensing images;

Step S2, according to the Wald criterion, the preprocessed MS image and PAN image are subjected to resolution reduction processing, and a simulation training set, a simulation test set and a real test set are constructed, wherein the simulation training set and the simulation test set include DUMS images and LPAN images As well as MS images, the real test set includes UMS images and PAN images;

Step S3, using AIHS transformation on the DUMS image in the simulation training set to obtain the brightness I component image, and using the I component image to perform histogram equalization processing on the LPAN image to obtain the MLPAN image;

Step S4, filtering the MLPAN image by using a guided filter to obtain multi-scale high-frequency component MLPAN Hn and low-frequency component MLPAN Ln ;

Step S5, using NSCT to filter the I component image to obtain multi-scale and multi-directional high-frequency direction sub-band images I Hn and low-frequency sub-band images I Ln ;

Step S6, constructing a detail extraction network ResCNN according to the DUMS image, the MLPAN image, the high-frequency component MLPAN Hn , the low-frequency component MLPAN Ln , the high-frequency direction sub-band image I Hn and the low-frequency sub-band image I Ln , and obtain injection details In-details;

Step S7, use the injected details In-details and DUMS images as the input of the shallow CNN network, and the MS image as the output, establish a nonlinear model NLCNN network, fully train the NLCNN network, obtain the optimal nonlinear model, and use the optimal non-linear The parameters of the linear model are frozen, and the pan-sharpened image is obtained using the optimal nonlinear model.
A two-stage lightweight network panchromatic sharpening method combined with guided filtering and NSCT according to claim 1, wherein the preprocessing in step S1 includes: atmospheric correction and spatial registration.
A two-stage lightweight network panchromatic sharpening method combining guided filtering and NSCT according to claim 1, wherein the specific steps of the step S2 include:

Step S21, according to the Wald criterion and the spatial resolution ratio between the panchromatic image and the multispectral image, the MS image and the PAN image are down-sampled using the bicubic interpolation method, and the reduced-resolution LPAN image and the DMS image are obtained;

Step S22. Upsampling the DMS image using a bicubic interpolation method according to the Wald criterion, and obtaining a DUMS image;

Step S23, using bicubic interpolation method to upsample the MS image according to the Wald criterion, and obtain the UMS image;

Step S24, constructing a simulation training set and a simulation test set from DUMS images, LPAN images and MS images, and constructing a real test set from UMS images and PAN images.
A two-stage lightweight network panchromatic sharpening method combined with guided filtering and NSCT according to claim 1, wherein the expression of the AIHS transformation in the step S3 to obtain the I component image is:

Where i is the i-th channel, a i is the adaptive coefficient, and N is the total number of channels.
A two-stage lightweight network panchromatic sharpening method combining guided filtering and NSCT according to claim 1, wherein the specific step of step S4 is: using a guided filter to filter the MLPAN image, The input image of the guide filter is an MLPAN image, and the guide image is an I component image. After filtering, the low-frequency component MLPAN i =GF(MLPAN i-1 , I) is obtained, wherein GF is a guide filter, and MLPAN i-1 is the i-th The output image of -1 filter, when i=1, is the MLPAN image, then the i-th low-frequency component MLPAN Li =MLPAN i , the i-th high-frequency component MLPAN Hi =MLPAN Li-1- MLPAN Li , in the process After n times of filtering, n high frequency components MLPAN Hn and n low frequency components MLPAN Ln are obtained.
A two-stage lightweight network panchromatic sharpening method combining guided filtering and NSCT according to claim 1, wherein the NSCT in step S5 includes a non-subsampling pyramid filter bank NSPFB and a non-subsampling Directional Filter Bank NSDFB.
A two-stage lightweight network panchromatic sharpening method combining guided filtering and NSCT according to claim 6, wherein the specific steps of step S5 include:

Step S51, using NSPFB to decompose the I component image to obtain the low-frequency sub-band image I Li and the high-frequency sub-band image I Hi ;

Step S52, using NSPFB to decompose the low-frequency sub-band image, and obtain the low-frequency sub-band image and the high-frequency sub-band image of the next layer;

Step S53, using NSDFB to filter the high-frequency sub-band images of each layer respectively, to obtain the high-frequency direction sub-band images of each layer.
A two-stage lightweight network panchromatic sharpening method combining guided filtering and NSCT according to claim 1, wherein the specific steps of step S6 include:

Step S61, using the DUMS image, the MLPAN image, the high frequency component MLPAN Hn , the low frequency component MLPAN Ln , the high frequency direction subband image I Hn and the low frequency subband image I Ln as the input of the ResCNN network;

Step S62, using the details of the difference between the DUMS image and the MS image as a label;

Step S63, train the ResCNN network, and after minimizing the loss function, freeze the training parameters to obtain the optimal model;

Step S64, obtaining injection details In-details according to the optimal model.
A two-stage lightweight network panchromatic sharpening method combining guided filtering and NSCT according to claim 1, wherein the specific steps of the step S7 include:

Step S71, using the In-details and DUMS images injected as the input of the nonlinear model NLCNN network;

Step S72, using the MS image as a label;

Step S73, train the above network, and after minimizing the loss function, freeze the training parameters to obtain the optimal nonlinear model;

Step S74, using the optimal nonlinear model to obtain a pan-sharpened image.