CN110148103A - EO-1 hyperion and Multispectral Image Fusion Methods, computer readable storage medium, electronic equipment based on combined optimization - Google Patents

Abstract

The present invention provides a kind of EO-1 hyperion based on combined optimization and Multispectral Image Fusion Methods, computer readable storage medium, electronic equipment, it solves existing EO-1 hyperion and Multispectral Image Fusion Methods relies on space degenerate matrix, the problem for causing high spectrum image spatial resolution lower.Method includes the following steps: step 1, input true value；Step 2 pre-processes true value, obtains observable low spatial resolution high spectrum image and observable high spatial resolution multi-spectral image；Step 3, according to line spectrum aliasing model, using Non-negative Matrix Factorization, to image I_hWith image I_mIt is mixed to carry out spectrum solution, end member matrix and abundance matrix needed for combined optimization obtain image I_hEnd member matrix E, abundance matrix A_hWith image I_mAbundance matrix A；Step 4 estimates high spatial resolution high spectrum image, is denoted as image Z.

Description

Hyperspectral and multispectral image fusion method based on joint optimization, computer readable Storage media, electronic equipment

technical field

The present invention relates to the technical field of image processing, in particular to a hyperspectral and multispectral image fusion method based on joint optimization, a computer-readable storage medium, and electronic equipment, which can be applied to fields such as environmental monitoring, target detection, target classification, and military reconnaissance .

Background technique

Hyperspectral imaging systems sample the electromagnetic spectrum in many continuous and very narrow spectral bands, resulting in hyperspectral images with high spectral resolution. In order to obtain more wavelength bands, the sensor has a splitting process before receiving light energy, that is, the beam splitter divides the light into many parts, then under the condition of a certain incident light energy, only a small part of the energy in each band reaches the sensor after splitting. The sensor can only respond when it receives a certain amount of light energy. In order to ensure a sufficient signal-to-noise ratio, the pixel size of the chip must be increased. The pixel size refers to the area of each pixel. When the pixel size increases, the number of pixels per unit area decreases, which will result in a decrease in the spatial resolution of the obtained image. Based on the above reasons, hyperspectral images obtained by hyperspectral imaging systems have low spatial resolution. Due to various hardware limitations, existing technicians have proposed to use software methods to improve the spatial resolution of hyperspectral images, that is, hyperspectral and multispectral image fusion methods, by fusing with multispectral images with high spatial resolution to improve hyperspectral Image spatial resolution.

In recent years, researchers have proposed many hyperspectral and multispectral image fusion methods. The purpose of these methods is to generate an unobservable high spatial resolution-hyperspectral image from an observable low spatial resolution-high spectral resolution hyperspectral image and an observable high spatial resolution-low spectral resolution multispectral image high-resolution hyperspectral images. Most of these methods rely on the spatial degradation matrix, which contains the spatial degradation relationship between high spectral resolution hyperspectral images and low spatial resolution multispectral images. Z.H.Nezhad et al. in the literature "Z.H.Nezhad, A.Karami, R.Heylenand P. Scheunders, "Fusion of Hyperspectral and Multispectral Images Using Spectral Unmixing and Sparse Coding," IEEE Journal of Selected Topics inApplied Earth Observations and Remote Sensing, vol.9 , no.6, pp.2377-2389, 2016." proposed a hyperspectral and multispectral image fusion method based on spectral unmixing and sparse coding. The fusion process of this method is an ill-posed inverse problem. Encoding the constructed regularization term converts it into a well-posed inverse problem, and then uses some high-spatial-resolution multispectral images or panchromatic images in uncorrelated scenes to construct a suitable dictionary, based on this dictionary and estimated by the linear spectral unmixing model The initial high-spatial-resolution hyperspectral image to estimate the sparse code, and then use the sparse code as a regularization term to calculate the abundance by solving a well-posed inverse problem, and finally obtain the required high-spatial space from the obtained abundance and endmembers high-resolution hyperspectral images. However, in practical applications, it is very difficult to accurately estimate the spatial degradation matrix. Such methods need to estimate the spatial degradation matrix, and the resulting estimation error will propagate during the fusion process, affecting the performance of the fusion method.

In summary, existing hyperspectral and multispectral image fusion methods rely too much on spatially degenerated matrices, resulting in low spatial resolution of hyperspectral images, which makes them somewhat limited.

Contents of the invention

In order to solve the problem that existing hyperspectral and multispectral image fusion methods rely on spatial degradation matrices, resulting in low spatial resolution of hyperspectral images, the present invention proposes a hyperspectral and multispectral image fusion method based on joint optimization. Reading storage media and electronic equipment are mainly used to improve the problem of low spatial resolution of hyperspectral images, so as to improve the spatial resolution of hyperspectral images.

Technical solution of the present invention is:

A hyperspectral and multispectral image fusion method based on joint optimization, comprising the following steps:

Step 1, input real value, described real value refers to real high spatial resolution hyperspectral image;

Step 2. Preprocessing the real values to obtain observable low-spatial-resolution hyperspectral images and observable high-spatial-resolution multispectral images, which are recorded as image I _h and image I _m respectively;

Step 3. According to the linear spectral aliasing model, use non-negative matrix decomposition to perform spectral unmixing on image I _h and image I _m , and jointly optimize the required endmember matrix and abundance matrix; when the maximum number of iterations is reached, get Endmember matrix E of image I _h , abundance matrix A _h and abundance matrix A of image I _m ;

Step 4. Unmixing and reconstruction. According to the endmember matrix E of the image I _h and the abundance matrix A of the image I _m , a hyperspectral image with high spatial resolution is calculated, that is, the final fusion result, which is recorded as image Z.

Further, the step 2 is specifically:

Step 2.1) Carry out spatial degeneration processing on the real value, that is, firstly perform a blurring operation on the real value, and then perform a down-sampling operation on the obtained result to obtain an observable low-spatial-resolution hyperspectral image, denoted as image I _h ;

Step 2.2) Spectral degradation processing is performed on the real value, that is, the real value is multiplied by the spectral response matrix to obtain an observable multispectral image with high spatial resolution, denoted as image I _m .

Further, the step 3 is specifically:

Step 3.1) Input image I _h , image _Im and spectral response matrix G _m , denote the endmember matrix and abundance matrix of image I _h as E and A _h , and denote the abundance matrix of image I _m as A, According to the linear spectral aliasing model I _h ≈ EA _h and I _m ≈ G _m EA, E and A _h can be obtained by performing spectral unmixing on image I _h using non-negative matrix factorization, and spectral unmixing on image _Im by using non-negative matrix factorization Unmixing can get A;

Step 3.2) Initialize the endmember matrix E, the abundance matrix A _h and the abundance matrix A, and the number of iterations k=0;

Step 3.3) Simultaneously update the endmember matrix E, the abundance matrix A _h and the abundance matrix A:

Among them, α ^(k) is the step size of the kth iteration, and L is the loss function;

Among them, the parameter λ=8, ‖·‖ _F represents the F norm, and Respectively represent the partial derivative of L to E, A _h and A;

Among them, ( ) ^T represents matrix transposition;

Step 3.4) Determine whether the maximum number of iterations is reached, and if so, output the obtained endmember matrix E, abundance matrix A _h and abundance matrix A, otherwise continue to step 3.3).

Further, the step 4 is specifically:

Step 4.1) According to the endmember matrix E of the image I _h and the abundance matrix A of the image I _m , a hyperspectral image with high spatial resolution is calculated, that is, the final fusion result, which is recorded as image Z:

Z=EA.

Further, said step 4 also includes step 4.2):

Step 4.2) Compare the image Z with the real value, and calculate the evaluation index.

At the same time, the present invention also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the above method are realized.

In addition, the present invention also provides an electronic device, which includes:

processor;

A computer-readable storage medium, on which a computer program is stored, and the computer program executes the steps of the above method when the computer program is run by the processor.

Compared with existing methods, the present invention has the following beneficial effects:

1. The image fusion method of the present invention does not require prior knowledge of the spectral degradation matrix, adopts a method based on joint optimization to fuse low spatial resolution hyperspectral images and high spatial resolution multispectral images, and according to the linear spectral aliasing model, utilizes Non-negative matrix factorization, spectral unmixing of low spatial resolution hyperspectral images and high spatial resolution multispectral images, joint optimization of the required endmember matrix and abundance matrix, and finally unmixing and reconstruction of them, the quality can be obtained Higher high spatial resolution hyperspectral image, this method does not require the participation of spectral degradation matrix, overcomes the dependence of traditional methods on spectral degradation matrix, improves the spatial resolution of hyperspectral image, has good fusion effect and low complexity The advantages.

2. The image fusion method of the present invention has a wide range of applications, and can be applied to fields such as environmental monitoring, target detection, target classification, and military reconnaissance.

Description of drawings

Fig. 1 is a flow chart of the hyperspectral and multispectral image fusion method based on joint optimization in the present invention.

Detailed ways

Below in conjunction with accompanying drawing and specific embodiment the content of the present invention is described in further detail:

According to the linear spectral aliasing model, the method of the present invention uses non-negative matrix decomposition to perform spectral unmixing on low spatial resolution hyperspectral images and high spatial resolution multispectral images, and jointly optimizes the required endmember matrix and abundance matrix, Finally, by unmixing and reconstructing them, a higher quality hyperspectral image with high spatial resolution can be obtained. Compared with the existing method, the present invention does not require the participation of the spectral degeneration matrix, overcomes the dependence of the traditional method on the spectral degeneration matrix, and has the advantages of good fusion effect, low complexity and wide application range.

As shown in Figure 1, the hyperspectral and multispectral image fusion method based on joint optimization provided by the present invention comprises the following steps:

Step 1, input the real value, the real value refers to the real high spatial resolution hyperspectral image;

Step 2. Preprocessing the real values to obtain observable low-spatial-resolution hyperspectral images and observable high-spatial-resolution multispectral images, which are recorded as image I _h and image I _m respectively;

Step 2.1) Carry out spatial degeneration processing on the real value, that is, firstly perform a blurring operation on the real value, and then perform a down-sampling operation on the obtained result to obtain an observable low-spatial-resolution hyperspectral image, denoted as image I _h ;

Step 2.2) Perform spectral degradation processing on the real value, that is, multiply the real value by the spectral response matrix to obtain an observable high-spatial-resolution multispectral image, denoted as image I _m ;

Step 3. According to the linear spectral aliasing model, use non-negative matrix decomposition to perform spectral unmixing on the image _Ih and image _Im , and jointly optimize the required endmember matrix and abundance matrix. When the maximum number of iterations is reached, get Endmember matrix E of image I _h , abundance matrix A _h and abundance matrix A of image I _m ;

Step 3.1) Input image I _h , image _Im and spectral response matrix G _m , denote the endmember matrix and abundance matrix of image I _h as E and A _h , and denote the abundance matrix of image I _m as A, According to the linear spectral aliasing model I _h ≈ EA _h and I _m ≈ G _m EA, E and A _h can be obtained by performing spectral unmixing on image I _h using non-negative matrix factorization, and spectral unmixing on image _Im by using non-negative matrix factorization Unmixing can get A;

Step 3.2) Initialize the endmember matrix E, the abundance matrix A _h and the abundance matrix A, and the number of iterations k=0;

Step 3.3) Simultaneously update the endmember matrix E, the abundance matrix A _h and the abundance matrix A:

Among them, α ^(k) is the step size of the kth iteration, and its calculation method is in the literature "C.-J.Lin, "Projected gradient methods for nonnegative matrix factorization," Neural Computation, vol. 19, no.10, pp .2756–2779,2007." In Algorithm 4, L is the loss function,

Among them, the parameter λ=8, ‖·‖ _F represents the F norm, and Respectively represent the partial derivative of L to E, A _h and A,

Among them, ( ) ^T represents matrix transposition;

Step 3.4) Judging whether the maximum number of iterations is reached, if reached, then output the obtained endmember matrix E, abundance matrix A _h and abundance matrix A, otherwise continue to step 3.3;

Step 4. Unmixing and reconstruction and calculation of evaluation indicators

Step 4.1) According to the endmember matrix E of image I _h and the abundance matrix A of image I _m , a hyperspectral image with high spatial resolution is estimated, that is, the final fusion result, which is recorded as image Z:

Z=EA.

Step 4.2) Compare the image Z with the real value, and calculate the evaluation index. The performance of the fusion method is evaluated by comparing the similarity between the fusion result and the real value, that is, the greater the similarity between the fusion result and the real value, the better the quality of the fusion result.

The effects of the present invention will be further illustrated through specific simulation experiments below.

1. Simulation conditions

The inventive method is that on the central processing unit is Intel (R) Core (TM) i7 5930k, internal memory 64GB, Ubuntu operating system, uses the simulation that MATLAB software carries out;

2. Simulation content

The experimental data used is the Pavia University database, which contains a real hyperspectral image, the image content is Pavia University, and the spatial resolution of the image is 1.3 meters. The size of the image is 200×200×103, where 200×200 is the spatial size of the image and 103 is the number of spectral bands. The original hyperspectral image in the Pavia University database is used as the real value, and the image obtained by blurring and downsampling by 4 times is used as the low spatial resolution hyperspectral image for testing.

On the Pavia University database, the experiment of the algorithm of the present invention (a hyperspectral and multispectral image fusion method based on joint optimization) is completed. In order to prove the effectiveness of the algorithm, considering the popularity and novelty of the algorithm, four comparison methods are selected for comparison: Bicubic, FUSE, PALM and CO-CNMF. Among them, Bicubic is a classic benchmark method, which was published by Zeyde et al. in the literature "R.Zeyde, M.Elad, and M.Protter, "On single image scale-upusingsparse-representations," Curves and Surfaces, 2012, pages 711-730 proposed in "; FUSE in the literature "Q.Wei, N.Dobigeon, and J.-Y.Tourneret, "Fast fusion of multi-bandimages based on solving a sylvester equation," IEEE Transactions onImageProcessing, vol.24, no. 11, pp.4109-4121, 2015."; PALM was proposed in the literature "C. Lanaras, E. Baltsavias, and K. Schindler, "Hyperspectral superresolution by coupledspectral unmixing," in IEEE International Conference on Computer Vision, 2015, pp. 3586-3594."; CO-CNMF in the literature "C.-H.Lin, F.Ma, C.-Y. Chi, and C.-H.Hsieh, "Aconvex optimization-based coupled nonnegative matrix factorization algorithmfor hyperspectral and multispectral data fusion, "IEEE Transactions on Geoscience and Remote Sensing, vol.56, no.3, pp.1652-1667, 2018."

The performance of the fusion method was measured using PSNR, UIQI, RMSE, ERGAS, and SAM of ground-truth and super-resolution hyperspectral images. On the Pavia University database, it is compared with 4 comparison methods, and the results are shown in Table 1.

It can be seen from Table 1 that the fusion results of the present invention are better than the existing fusion methods. This is because the existing method needs to estimate the spatial degradation matrix, and the error generated in the estimation process will be transmitted during the fusion process, which will affect the performance of the fusion method. The present invention does not require the participation of the spatial degradation matrix, and eases the process of estimating the spatial degradation matrix Errors generated in the fusion process will be transmitted in the problem. Therefore, this method is more robust and effective than other methods, further verifying the advancement of the present invention.

Table 1 Comparison results of different fusion methods

Bicubic FUSE PALM CO-CNMF this invention PSNR 25.885 32.032 33.708 32.105 36.329 UIQI 0.768 0.949 0.963 0.950 0.975 RMSE 13.261 6.906 5.367 6.629 4.260 ERGAS 8.086 4.256 3.473 3.955 3.049 SAM 6.271 4.840 4.038 4.280 3.650

An embodiment of the present invention also provides a computer-readable storage medium for storing a program. When the program is executed, the steps of the hyperspectral and multispectral image fusion method based on joint optimization are implemented. In some possible implementations, various aspects of the present invention can also be implemented in the form of a program product, which includes program code, and when the program product is run on a terminal device, the program code is used to make the The terminal device executes the steps according to various exemplary embodiments of the present invention described in the above method part of this specification.

The program product for realizing the above method can adopt a portable compact disc read-only memory (CD-ROM) and include program codes, and can be run on a terminal device such as a personal computer. However, the program product of the present invention is not limited thereto. In this document, a readable storage medium may be any tangible medium containing or storing a program, and the program may be used by or in combination with an instruction execution system, apparatus or device.

A program product may take the form of any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any combination thereof. More specific examples (non-exhaustive list) of readable storage media include: electrical connection with one or more conductors, portable disk, hard disk, random access memory (RAM), read only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

Claims (7)

1. a hyperspectral and multispectral image fusion method based on joint optimization, is characterized in that, comprises the following steps: Step 1, input real value, described real value refers to real high spatial resolution hyperspectral image; Step 2. Preprocessing the real values to obtain observable low-spatial-resolution hyperspectral images and observable high-spatial-resolution multispectral images, which are recorded as image I _h and image I _m respectively; Step 3. According to the linear spectral aliasing model, use non-negative matrix decomposition to perform spectral unmixing on image I _h and image I _m , and jointly optimize the required endmember matrix and abundance matrix; when the maximum number of iterations is reached, get Endmember matrix E of image I _h , abundance matrix A _h and abundance matrix A of image I _m ; Step 4. Unmixing and reconstruction. According to the endmember matrix E of the image I _h and the abundance matrix A of the image I _m , a hyperspectral image with high spatial resolution is calculated, that is, the final fusion result, which is recorded as image Z. 2. the hyperspectral and multispectral image fusion method based on joint optimization according to claim 1, is characterized in that, described step 2 is specifically: Step 2.1) Carry out spatial degeneration processing on the real value, that is, firstly perform a blurring operation on the real value, and then perform a down-sampling operation on the obtained result to obtain an observable low-spatial-resolution hyperspectral image, denoted as image I _h ; Step 2.2) Spectral degradation processing is performed on the real value, that is, the real value is multiplied by the spectral response matrix to obtain an observable multispectral image with high spatial resolution, denoted as image I _m . 3. the hyperspectral and multispectral image fusion method based on joint optimization according to claim 1, is characterized in that, described step 3 is specifically: Step 3.1) Input image I _h , image _Im and spectral response matrix G _m , denote the endmember matrix and abundance matrix of image I _h as E and A _h , and denote the abundance matrix of image I _m as A, According to the linear spectral aliasing model I _h ≈ EA _h and I _m ≈ G _m EA, E and A _h can be obtained by performing spectral unmixing on image I _h using non-negative matrix factorization, and spectral unmixing on image _Im by using non-negative matrix factorization Unmixing can get A; Step 3.2) Initialize the endmember matrix E, the abundance matrix A _h and the abundance matrix A, and the number of iterations k=0; Step 3.3) Simultaneously update the endmember matrix E, the abundance matrix A _h and the abundance matrix A: Among them, α ^(k) is the step size of the kth iteration, and L is the loss function; Among them, the parameter λ=8, ‖·‖ _F represents the F norm, and Respectively represent the partial derivative of L to E, A _h and A; Among them, ( ) ^T represents matrix transposition; Step 3.4) Determine whether the maximum number of iterations is reached, and if so, output the obtained endmember matrix E, abundance matrix A _h and abundance matrix A, otherwise continue to step 3.3). 4. The hyperspectral and multispectral image fusion method based on joint optimization according to claim 1 or 2 or 3, wherein said step 4 is specifically: Step 4.1) According to the endmember matrix E of the image I _h and the abundance matrix A of the image I _m , a hyperspectral image with high spatial resolution is calculated, that is, the final fusion result, which is recorded as image Z: Z=EA. 5. the hyperspectral and multispectral image fusion method based on joint optimization according to claim 4, is characterized in that, described step 4 also comprises step 4.2): Step 4.2) Compare the image Z with the real value, and calculate the evaluation index. 6. A computer-readable storage medium, on which a computer program is stored, characterized in that: when the computer program is executed by a processor, the steps of any one of the methods of claims 1-4 are implemented. 7. An electronic device, characterized in that the electronic device comprises: processor; A computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by the processor, the steps of the method according to any one of claims 1 to 4 are executed.