CN115311187B - Hyperspectral fusion imaging method, system and medium based on internal and external prior - Google Patents
Hyperspectral fusion imaging method, system and medium based on internal and external prior Download PDFInfo
- Publication number
- CN115311187B CN115311187B CN202211243981.3A CN202211243981A CN115311187B CN 115311187 B CN115311187 B CN 115311187B CN 202211243981 A CN202211243981 A CN 202211243981A CN 115311187 B CN115311187 B CN 115311187B
- Authority
- CN
- China
- Prior art keywords
- image
- hyperspectral
- prior
- internal
- resolution
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 230000004927 fusion Effects 0.000 title claims abstract description 51
- 238000003384 imaging method Methods 0.000 title claims abstract description 30
- 238000000034 method Methods 0.000 claims abstract description 68
- 230000011218 segmentation Effects 0.000 claims abstract description 21
- 238000013528 artificial neural network Methods 0.000 claims abstract description 10
- 238000013135 deep learning Methods 0.000 claims abstract description 7
- 239000011159 matrix material Substances 0.000 claims description 55
- 230000006870 function Effects 0.000 claims description 28
- 238000013527 convolutional neural network Methods 0.000 claims description 25
- 238000001228 spectrum Methods 0.000 claims description 24
- 230000003595 spectral effect Effects 0.000 claims description 23
- 230000003190 augmentative effect Effects 0.000 claims description 17
- 238000004590 computer program Methods 0.000 claims description 11
- 125000004432 carbon atom Chemical group C* 0.000 claims description 8
- 239000000654 additive Substances 0.000 claims description 6
- 230000000996 additive effect Effects 0.000 claims description 6
- 238000005457 optimization Methods 0.000 claims description 6
- 238000005070 sampling Methods 0.000 claims description 6
- 238000003860 storage Methods 0.000 claims description 6
- 239000013598 vector Substances 0.000 claims description 6
- 238000004422 calculation algorithm Methods 0.000 claims description 4
- 238000000354 decomposition reaction Methods 0.000 claims description 4
- 238000009826 distribution Methods 0.000 claims description 4
- 238000010606 normalization Methods 0.000 claims description 3
- 238000000638 solvent extraction Methods 0.000 claims description 3
- 150000001875 compounds Chemical class 0.000 claims description 2
- 230000000694 effects Effects 0.000 abstract description 7
- 238000010586 diagram Methods 0.000 description 12
- 238000002474 experimental method Methods 0.000 description 8
- 230000008569 process Effects 0.000 description 8
- 238000007500 overflow downdraw method Methods 0.000 description 7
- 238000012549 training Methods 0.000 description 7
- 238000007499 fusion processing Methods 0.000 description 6
- 238000004364 calculation method Methods 0.000 description 5
- 238000011156 evaluation Methods 0.000 description 5
- 238000012545 processing Methods 0.000 description 5
- 241001584785 Anavitrinella pampinaria Species 0.000 description 2
- 101100043438 Mus musculus Srsf10 gene Proteins 0.000 description 2
- 238000013461 design Methods 0.000 description 2
- 238000011160 research Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000015572 biosynthetic process Effects 0.000 description 1
- 230000000903 blocking effect Effects 0.000 description 1
- 230000008878 coupling Effects 0.000 description 1
- 238000010168 coupling process Methods 0.000 description 1
- 238000005859 coupling reaction Methods 0.000 description 1
- 238000002059 diagnostic imaging Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 239000000463 material Substances 0.000 description 1
- 238000013178 mathematical model Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000003672 processing method Methods 0.000 description 1
- 238000003786 synthesis reaction Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/50—Image enhancement or restoration using two or more images, e.g. averaging or subtraction
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T3/4053—Scaling of whole images or parts thereof, e.g. expanding or contracting based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
- G06T3/4061—Scaling of whole images or parts thereof, e.g. expanding or contracting based on super-resolution, i.e. the output image resolution being higher than the sensor resolution by injecting details from different spectral ranges
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/12—Edge-based segmentation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20021—Dividing image into blocks, subimages or windows
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20212—Image combination
- G06T2207/20221—Image fusion; Image merging
Landscapes
- Physics & Mathematics (AREA)
- Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Spectroscopy & Molecular Physics (AREA)
- Image Analysis (AREA)
- Image Processing (AREA)
Abstract
The invention discloses a hyperspectral fused imaging method, a hyperspectral fused imaging system and a hyperspectral fused imaging medium based on internal and external priors, wherein the hyperspectral fused imaging method comprises the steps of learning local similarity of an image from an input multispectral image Z by adopting a superpixel segmentation method to serve as an internal prior; learning external prior of the image from any external gray level image set by adopting a deep learning neural network; introducing internal prior and external prior into an input low-resolution hyperspectral image Y and multispectral image Z to an output high-resolution fusion hyperspectral image X basic observation model to establish an optimized observation model, and solving the optimized observation model to obtain a high-resolution fusion hyperspectral image X. The method can effectively realize the fusion of the low-resolution hyperspectral image and the high-resolution multispectral image to obtain the high-resolution hyperspectral image, and has the advantages of good fusion effect and strong robustness.
Description
Technical Field
The invention relates to a hyperspectral image and multispectral image fusion technology, in particular to a hyperspectral fusion imaging method, a hyperspectral fusion imaging system and a hyperspectral fusion imaging medium based on internal and external prior.
Background
Compared with the traditional full-color image or RGB image, the hyperspectral image has more spectral bands, namely very high spectral resolution, and the abundant spectral information is beneficial to analyzing the physicochemical characteristics of the object. Therefore, the hyperspectral image has wide application in the fields of remote sensing, medical imaging, geological exploration and the like. The radiated light energy is constant, the energy of each spectrum is reduced due to the increase of the number of the imaged spectrums, the imaged spectrums are easily interfered by noise, and the size of the photosensitive unit needs to be increased in order to avoid the noise, so the spatial resolution of the hyperspectral image is often low. In order to solve the contradiction between the spectral resolution and the spatial resolution and improve the spatial resolution of the hyperspectral image, a method of fusing the hyperspectral image and the multispectral image with higher spatial resolution is often adopted. Therefore, the research on the fusion of the hyperspectral image and the multispectral image has important significance.
In recent years, it has become an extremely effective method to improve the resolution of a hyperspectral image by fusing a high-resolution multispectral image and a low-resolution hyperspectral image, and numerous scholars at home and abroad have proposed various fusion methods, and have conducted extensive and intensive research on how to improve the quality of the fused image. The methods design or learn a regular term by utilizing various different prior characteristics of the image to constrain the hyper-spectral image hyper-resolution reconstruction process, and are mainly divided into two types of methods utilizing image internal prior and external prior according to different utilized prior characteristics. The method based on image internal prior utilizes prior information of a spectral image, for example, the method based on sparse representation comprises Bayesian sparse representation, wherein the Bayesian sparse representation firstly analyzes a scene to infer spectral probability distribution of materials of the scene and then carries out sparse coding, and the method based on local low rank utilizes local low rank and other characteristics of the spectral image to carry out regular constraint on the hyper-spectral reconstruction process of the hyper-spectral image; the method based on image external prior learns or designs a regular term according to common prior characteristics among images, and the methods based on the depth convolution neural network which are widely applied to image fusion at present can be mainly classified into the methods based on the external prior. Obviously, the method based on image internal prior lacks external prior information commonly existing in spectral image fusion, and the method based on image external prior, such as a deep convolutional neural network, often needs a large number of high-resolution hyperspectral images which are difficult to obtain for training, and often has the problems of poor generalization performance and the like.
Disclosure of Invention
The technical problems to be solved by the invention are as follows: aiming at the problems in the prior art, the invention provides a hyperspectral fusion imaging method, a hyperspectral fusion imaging system and a hyperspectral fusion imaging medium based on internal and external prior.
In order to solve the technical problems, the invention adopts the technical scheme that:
a hyperspectral fused imaging method based on internal and external priors comprises the following steps:
s101, learning local similarity of an image from an input multispectral image Z by adopting a superpixel segmentation method to serve as internal prior; learning external prior of the image from any external gray level image set by adopting a deep learning neural network;
s102, introducing internal prior and external prior into an input low-resolution hyperspectral image Y and a multispectral image Z to an output high-resolution fusion hyperspectral image X basic observation model to establish an optimized observation model, and solving the optimized observation model to obtain a high-resolution fusion hyperspectral image X.
Optionally, the learning of the local similarity of the images from the input multispectral image Z by using the superpixel segmentation method in step S101 includes:
s201, partitioning the multispectral image Z into blockskEach image block is divided by super pixels to obtain low-rank characteristics;
s202, multispectral image ZkThe low-rank characteristics of the image blocks are constrained by a nuclear norm to obtain local low-rank characteristics of a multispectral image Z shown in the following formula as an introduced internal prior;
in the above formula, the first and second carbon atoms are,kthe number of image blocks of the multi-spectral image Z,Z i is the second of the multispectral image ZiAn imageAnd (5) blocking.
Optionally, step S102 includes:
s301, establishing a correlation representation model shown in the following formula for an input low-resolution hyperspectral image Y and a multispectral image Z to an output high-resolution fusion hyperspectral image X;
Y=XBD+N h ，Z=RX+N m ，
in the above formula, B represents the fuzzy operation, D represents the spatial down-sampling, N h Representing additive noise, R representing a spectrally downsampled matrix, N m Representing additive noise;
s302, decomposing the fused hyperspectral image X into a product of a spectrum subspace E and a coefficient matrix A according to the correlation representation model, and establishing a basic observation model from an input low-resolution hyperspectral image Y and a multispectral image Z to an output high-resolution fused hyperspectral image X shown in the following formula;
in the formula, A is a coefficient matrix, and E is a hyperspectral image Y spectrum subspace;
s303, on the basis of the basic observation model, introducing internal prior and external prior as two regular terms, and establishing an optimized observation model shown as the following formula:
in the above formula, the first and second carbon atoms are,η 1 andη 2 the weight parameters of the two regularization terms respectively,
expressed as an internal prior introduced by the coefficient matrix a,knumber of image blocks forming an image for coefficient matrix A, A i Forming an image for the coefficient matrix AiEach of the image blocks is a block of an image,
representing the introduced external prior for the coefficient matrix a;
s304, constructing an augmented Lagrangian function according to the optimized observation model;
s305, solving the augmented Lagrange function to obtain a high-resolution fusion hyperspectral image X.
Optionally, the method for acquiring the spectrum subspace E of the hyperspectral image Y in step S302 includes: taking L maximum singular values to perform singular value decomposition on the hyperspectral image Y into Y = U 1 Σ 1 V 1 And a matrix U formed by the left singular vectors 1 A spectral subspace E as the hyperspectral image Y, wherein ∑ 1 As a matrix of singular values, V 1 Is a matrix composed of right singular vectors.
Optionally, in step S304, the function expression of the augmented lagrangian function is constructed according to the optimized observation model as follows:
in the above formula, the first and second carbon atoms are,L(A,V 1 ,V 2 ,G 1 ,G 2 ) Representing an augmented Lagrangian function, where V 1 ,V 2 Respectively, introduced variables, G, equal to the coefficient matrix A to be estimated 1 ,G 2 Respectively, are lagrange multipliers, respectively,λas a function of the weight parameter(s),μin order to be a penalty factor,
is expressed as a variable V 1 The internal prior introduced is that the internal prior,knumber of image blocks, V, forming an image for coefficient matrix A 1 i Is a variable V 1 To form an imageiEach of the image blocks is a block of an image,
is expressed as a variable V 2 An external prior is introduced.
Optionally, the solving the augmented lagrangian function in step S305 to obtain the fused hyperspectral image X includes:
s401, decomposing the augmented Lagrangian function into the following subproblem solving models:
in the above formula, G * 1 ,G * 2 Respectively, the updated Lagrange multiplier, mu * Gamma is an updated penalty factor and is an update coefficient;
s402, adopting an alternative direction multiplier method ADMM to respectively carry out iterative solution on the subproblem solution model until the number of iterationskSkipping to step S403 when the threshold value is equal to a preset threshold value K;
and S403, multiplying the obtained coefficient matrix A by the spectrum subspace E to obtain the high-resolution fusion hyperspectral image X.
Optionally, when the alternative direction multiplier method ADMM is used in step S402 to respectively perform iterative solution on the subproblem solution models, solving the hilbert equation for the subproblem solution model of the coefficient matrix a to obtain an updated coefficient matrix a; for variable V 1 Sub-problem solving ofThe solution model adopts a singular value threshold algorithm to realize the optimization solution of the kernel norm on the updated variable V 1 (ii) a For variable V 2 Is modeled as a variable V 2 The model for solving the subproblems of (A) is regarded as being derived from the equation with a variance ofλ/(2μ) (ii) images of Gaussian noise (A-G) 2 /(2μ) ) using a predetermined deep convolutional neural network input image (A-G) 2 /(2μ) ) and distribution of Gaussian noiseλ/(2μ) And outputting the de-noised gray level image as an updated variable V 2 。
Optionally, the deep convolutional neural network is composed of three parts, the first part is composed of a first layer containing convolution and linear rectification units, the second part is composed of 13 layers which are formed by sequentially cascading the convolution, linear rectification units and batch normalization operation units, and the third part is composed of a layer containing only one convolution operation.
In addition, the invention also provides an internal and external apriori based hyperspectral fusion imaging system which comprises a microprocessor and a memory which are connected with each other, wherein the microprocessor is programmed or configured to execute the internal and external apriori based hyperspectral fusion imaging method.
Furthermore, the present invention also provides a computer readable storage medium having stored therein a computer program for being programmed or configured by a microprocessor to perform the internal and external priors based hyper-spectral fusion imaging method.
Compared with the prior art, the invention mainly has the following advantages: the method comprises the steps of learning the local similarity of an image from an input multispectral image Z by adopting a superpixel segmentation method to serve as internal prior; learning external prior of the image from any external gray level image set by adopting a deep learning neural network; the method comprises the steps of introducing an input low-resolution hyperspectral image Y and a multispectral image Z to an output high-resolution fusion hyperspectral image X basic observation model in a priori mode and an output external priori mode to establish an optimized observation model, and solving the optimized observation model to obtain the high-resolution fusion hyperspectral image X.
Drawings
FIG. 1 is a schematic diagram of a basic flow of a method according to an embodiment of the present invention.
FIG. 2 is a diagram illustrating an example of input and output results of superpixel splitting according to an embodiment of the present invention.
Fig. 3 is a flowchart illustrating step S102 according to an embodiment of the present invention.
Fig. 4 is a schematic structural diagram of a deep learning neural network used in the embodiment of the present invention.
Detailed Description
As shown in fig. 1, the hyperspectral fusion imaging method based on internal and external priors in this embodiment includes:
s101, learning local similarity of an image from an input multispectral image Z by adopting a superpixel segmentation method to serve as internal prior; learning external prior of the image from any external gray level image set by adopting a deep learning neural network;
s102, introducing internal prior and external prior into an input low-resolution hyperspectral image Y and a multispectral image Z to an output high-resolution fusion hyperspectral image X basic observation model to establish an optimized observation model, and solving the optimized observation model to obtain a high-resolution (high resolution relative to the hyperspectral image Y) fusion hyperspectral image X.
In order to fully utilize the local low-rank characteristic of the hyperspectral image, the local low-rank prior is added by using superpixel segmentation and kernel norm optimization, and the fusion effect and the fusion efficiency are improved. In this embodiment, learning the local similarity of the image from the input multispectral image Z by using the superpixel segmentation method in step S101 includes:
s201, partitioning the multispectral image Z into blockskEach image block and each super pixelSegmenting to obtain low rank features;
it should be noted that the super-pixel segmentation method is an existing image processing method, and this embodiment only relates to the application of the super-pixel segmentation method, and does not relate to the improvement of the super-pixel segmentation method, so the implementation details thereof are not described in detail herein.
S202, multispectral image ZkThe low-rank characteristics of the image blocks are constrained by a nuclear norm to obtain local low-rank characteristics of a multispectral image Z shown in the following formula as an introduced internal prior;
in the above formula, the first and second carbon atoms are,kthe number of image blocks of the multi-spectral image Z,Z i is the second of the multispectral image ZiAnd each image block.
Low-rank characteristics often exist inside the image block, and good prior constraints are provided for the fusion process. According to the method, the local low-rank characteristic inside the multispectral image Z is utilized, the local similarity of the image is learned from the input multispectral image Z by adopting a superpixel segmentation method to serve as an internal prior, so that good prior constraint is provided for a fusion process, and the quality of the fused hyperspectral image X with high resolution can be improved.
Fig. 2 is a schematic diagram of an example of input and output results of superpixel segmentation in the present embodiment, where (a) is an original image of the multispectral image Z, (b) is a low-rank feature obtained by superpixel segmentation, and (c) is a local low-rank feature obtained by using a kernel norm constraint. As can be seen from fig. 2, there is a low rank feature in the image block obtained by superpixel segmentation of the multispectral image Z. In image processing, the kernel norm is often used to constrain the low-rank property of an image, defined as the sum of matrix singular values, which is defined as:
in the above-mentioned formula, the compound has the following structure,trrepresenting the norm, i.e. the moment, of the trace of a matrixSum of diagonal elements of the matrix.
As shown in fig. 3, step S102 includes:
s301, establishing a correlation representation model shown in the following formula for an input low-resolution hyperspectral image Y and a multispectral image Z to an output high-resolution fusion hyperspectral image X;
Y =XBD+N h ，Z=RX+N m ，
in the above formula, B represents the fuzzy operation, D represents the spatial down-sampling, N h Representing additive noise, R representing a spectral down-sampling matrix, N m Representing additive noise; in the embodiment, the relation between the hyperspectral image and the multispectral image and the fused hyperspectral image is obtained by analyzing the observation models of the hyperspectral image and the multispectral image, and a mathematical model of the fused image is established;
s302, decomposing the fused hyperspectral image X into a product of a spectrum subspace E and a coefficient matrix A according to the correlation representation model, and establishing a basic observation model from an input low-resolution hyperspectral image Y and a multispectral image Z to an output high-resolution fused hyperspectral image X shown in the following formula;
in the formula, A is a coefficient matrix, and E is a hyperspectral image Y spectrum subspace;
the fused hyperspectral image X is expressed by adopting a subspace, and is decomposed into a product of a spectrum subspace E and a coefficient matrix A, so that the calculation efficiency is greatly improved, and the external prior learned from the gray level image can be applied to the fusion of the spectrum image;
s303, on the basis of a basic observation model, introducing an internal prior and an external prior as two regular terms, and establishing an optimized observation model shown in the following formula:
in the above formula, the first and second carbon atoms are,η 1 andη 2 the weight parameters of the two regularization terms respectively,
expressed as an internal prior introduced by the coefficient matrix a,kthe number of image blocks constituting an image for the coefficient matrix A, A i Forming an image for the coefficient matrix AiEach of the image blocks is a block of an image,
representing the introduced external prior for the coefficient matrix a; since the coefficient matrix a is obtained by decomposing the high-resolution hyperspectral image X into the spectral subspace E, it can be regarded as a special multispectral image. The multispectral image Z with high resolution contains main spatial information, so that the characteristics can be well kept by performing superpixel segmentation on the multispectral image Z, and the coefficient matrix A is inaccurate at the beginning of iteration, so that the multispectral image Z with high resolution is more suitable for segmentation;
s304, constructing an augmented Lagrange function according to the optimized observation model;
s305, solving the augmented Lagrangian function to obtain a high-resolution fusion hyperspectral image X.
In this embodiment, the method for obtaining the spectrum subspace E of the hyperspectral image Y in step S302 includes: taking L maximum singular values to decompose the singular value of the hyperspectral image Y into Y = U 1 Σ 1 V 1 And a matrix U formed by the left singular vectors 1 A spectral subspace E as the hyperspectral image Y, where ∑ 1 Is a matrix of singular values, V 1 Is a matrix composed of right singular vectors. According to the method, the main spectrum information contained in the hyperspectral image is utilized, the spectrum subspace E is obtained by performing truncated singular value decomposition on the hyperspectral image, the dimensionality is reduced, meanwhile, the hyperspectral image is well approximately expressed, and the decomposed spectrum subspace E contains the main spectrum information.
In this embodiment, in step S304, the function expression of the augmented lagrangian function constructed according to the optimized observation model is as follows:
in the above formula, the first and second carbon atoms are,L(A,V 1 ,V 2 ,G 1 ,G 2 ) Representing an augmented Lagrangian function, where V 1 ,V 2 Respectively, introduced variables, G, equal to the coefficient matrix A to be estimated 1 ,G 2 Respectively, are lagrange multipliers, respectively,λas a function of the weight parameter(s),μin order to be a penalty factor,
is expressed as a variable V 1 The internal prior introduced is that,knumber of image blocks, V, forming an image for coefficient matrix A 1 i Is a variable V 1 To form an imageiEach of the image blocks is a block of an image,
is expressed as a variable V 2 An external prior is introduced.
In this embodiment, the solving the augmented lagrangian function in step S305 to obtain the fused hyperspectral image X includes:
s401, decomposing the augmented Lagrangian function into the following subproblem solving models:
in the above formula, G * 1 ,G * 2 Respectively, the updated Lagrange multiplier, mu * The penalty factor after updating is gamma, and the updating coefficient is gamma;
s402, adopting an alternative direction multiplier method ADMM to respectively carry out iterative solution on the subproblem solution model until the number of iterationskSkipping to step S403 when the threshold value is equal to a preset threshold value K;
and S403, multiplying the obtained coefficient matrix A by the spectrum subspace E to obtain the high-resolution fusion hyperspectral image X.
In this embodiment, when the alternating direction multiplier method ADMM is used to iteratively solve the sub-problem solution model in step S402, the number of iterations k is initialized to be equal to 1, and the coefficient a to be estimated and the introduced variable V are used to estimate the coefficient k 1 、V 2 Lagrange multiplier G 1 、G 2 All are initialized to be a matrix with all 0 elements, and penalty factors are initializedμTaking the value of (a); and then, carrying out iterative updating on the previous sub-problem solving model until the iteration number K is equal to a preset threshold value K.
As an optional implementation manner, in this embodiment, when performing iterative update on the foregoing sub-problem solution model, the following steps are included: aiming at the subproblem solving model of the coefficient matrix A, solving a Hilbert equation to obtain an updated coefficient matrix A; for example, the R-FUSE method of Q.Wei et al is specifically used to solve the Hillwster equation in this example. For variable V 1 The sub-problem solution model of (a) employs a singular value threshold algorithm (the existing method proposed by j. -f.cai et al) to achieve the kernel norm optimization solution of the updated variable V 1 (ii) a For variable V 2 Is modeled as a variable V 2 Sub-problem solution model of (2)Considered as having a variance ofλ/(2μ) Of Gaussian noise (A-G) 2 /(2μ) For de-noising, using a predetermined deep convolutional neural network input image (A-G) 2 /(2μ) ) and distribution of Gaussian noiseλ/(2μ) And outputting the de-noised gray level image as an updated variable V 2 。
The deep convolutional neural network used in the embodiment is essentially to insert an image denoising method with a known noise level trained on a gray level image into the fusion process of the hyperspectral image and the multispectral image instead of training the hyperspectral image and the multispectral image with a certain size or spectral dimension, so that the deep convolutional neural network can be suitable for the fusion of the hyperspectral image and the multispectral image with various spatial and spectral dimensions, and has strong generalization performance. As shown in fig. 4, the deep convolutional neural network in this embodiment is composed of three parts, the first part is composed of a first layer including a convolution and linear rectification unit, the second part is formed by sequentially cascading 13 layers each including a convolution, linear rectification unit and batch normalization operation unit, and the third part is composed of a layer including only one convolution operation. The training sample of the deep convolutional neural network is a common gray image data set, and the training result of K.Zhang et al on a gray image is directly adopted in the experiment of the embodiment. The input of the deep convolutional neural network is an image of a two-dimensional space and a corresponding noise level, a corresponding de-noised image is obtained after the three parts are combined and superposed, and the process is expressed as follows:
in the above formula, V 2 Are variables input to the deep convolutional neural network.
Because the deep convolution neural network is obtained by training a common gray image data set, and any gray image data set can be adopted, the deep convolution neural network has the characteristic of being easier to obtain compared with a specific image sample.
In order to verify the hyperspectral fused imaging method based on internal and external priors, a hyperspectral image in a Pavia University dataset is selected for experiment. The spatial resolution of the image is 610 × 340, the spectral resolution is 115, in the experiment, for convenience of down-sampling, the image is spatially clipped to be a multiple of 16, that is, the spatial resolution is 608 × 336, and 93 spectra with high signal-to-noise ratios are selected for the experiment, so that the resolution of the high-resolution hyperspectral image adopted in the experiment is 608 × 336 in the embodiment. Meanwhile, in order to transversely compare the fusion effect of the method, 5 typical hyperspectral images and multispectral image fusion methods (GSA, NSSR, CNNMF, hySure and CSTF) are selected as comparison in the experiment. Meanwhile, in order to quantitatively compare the fusion effects of different methods, five common evaluation indexes for fusing hyperspectral images are selected in the experiment in the method, including a peak signal to noise ratio (PSNR), a relative integral synthesis Error (ERGAS), a Uniform Image Quality Index (UIQI), a spectral angular distance (SAM) and a Root Mean Square Error (RMSE). The higher the PSNR and UIQI values are, the higher the quality of the fused hyperspectral image is, and the higher the ERGAS, SAM and RMSE values are, the poorer the quality of the fused hyperspectral image is. Objective evaluation indexes of 5 typical fusion methods (GSA, NSSR, CNNMF, hySure, CSTF) and the method of this example (CNN _ LLR) in the fusion experiment on the Pavia University dataset are shown in table 1, and the results of the optimal method for each evaluation index are shown in bold.
Table 1: the objective evaluation indexes of the method and five typical hyperspectral and multispectral fusion methods are provided.
Table 1 shows that the method (CNN _ LLR) of the present embodiment has an effect superior to that of other methods in the evaluation indexes of the selected 5 kinds of fused hyperspectral images, and the main reason is that the method (CNN _ LLR) of the present embodiment fully learns the prior information inside and outside the image, and plays a role in constraining the fusion process of the image by using the local low-rank characteristic inside the image and the prior knowledge learned on the grayscale image outside the image through the deep convolutional neural network.
In summary, the hyperspectral fused imaging method based on the internal and external priors of the embodiment sufficiently learns the internal and external priors of the image, the multispectral image is divided into a plurality of image blocks by superpixel division, then the local low-rank characteristic is represented by applying the kernel norm in each image block, and the kernel norm summation can represent the local low-rank characteristic of the whole image because the superpixel blocks are not overlapped with each other. In addition, because the deep convolutional neural network used in the embodiment is only trained on the gray level image without the need of the hyperspectral data which is difficult to acquire, and the training result on the external gray level image can be directly applied to the fusion process, the application of the deep convolutional neural network in the fusion of the hyperspectral image and the multispectral image is greatly improved, the deep convolutional neural network can be trained more conveniently, the application range is wide, and the deep convolutional neural network is not limited to the fusion of the hyperspectral image and the multispectral image of a certain spectrum or space dimension. The method utilizes the fact that the low-resolution hyperspectral image contains main spectral information, obtains a spectral subspace from the low-resolution hyperspectral image by using truncated singular value decomposition, greatly reduces dimensionality while retaining most image information, and improves calculation efficiency; meanwhile, the decomposed coefficient matrix almost does not contain spectral information, so that the deep convolution neural network trained on the introduced external gray level image is more applicable. In the embodiment, the estimation process of the coefficient matrix is constrained by adopting the super-pixel segmentation and the kernel norm as the local low-rank prior in the image, and the optimization of the kernel norm adopts a singular value threshold algorithm, so that the optimization process is completed with higher calculation efficiency. For the problem of combination of internal prior and external prior, in the embodiment, firstly, a hyperspectral image is decomposed into a spectrum subspace and a coefficient matrix, a basic model is constructed according to an observation model of the hyperspectral image and a multispectral image, the internal prior and the external prior are introduced into the basic model, an alternating direction multiplier method is adopted in the overall solving process, two variables and lagrange multipliers are introduced to construct a new lagrange function, the values of the variables are alternately updated in the iteration process until an iteration number threshold is reached, and a high-resolution hyperspectral image is obtained. The method has the advantages that the prior characteristics of the inside and the outside of the image are learned at the same time, and the calculation efficiency is improved while the fusion effect is improved. For the deep convolutional neural network trained from the outside of the image, the deep convolutional neural network is also used as a denoising component unit to be inserted into the fusion process, loose coupling is realized, the training is simpler and more convenient, the generalization performance of the deep convolutional neural network is expanded, and the deep convolutional neural network is suitable for fusing hyperspectral images and multispectral images with different space and spectral dimensions. Compared with other typical hyperspectral images and multispectral image fusion methods, the hyperspectral image fusion method has the advantages that the hyperspectral image fused by the hyperspectral image and multispectral image fusion method is better in quality and better in generalization performance, the structure and parameters of the deep convolutional neural network do not need to be modified when the hyperspectral images and the multispectral images with different space and spectral dimensions are fused, meanwhile, the calculation time is short, and the efficiency is high. In summary, the method of the present embodiment learns the local similarity of the images from the multispectral image Z by using the superpixel segmentation method; learning external prior of the image from any external gray level image by adopting a deep learning method; the method can effectively realize the fusion of the low-resolution hyperspectral image and the high-resolution multispectral image to obtain the high-resolution hyperspectral image, and has the advantages of good fusion effect and strong robustness.
In addition, the present embodiment also provides an internal and external prior based hyperspectral fused imaging system, which includes a microprocessor and a memory connected to each other, wherein the microprocessor is programmed or configured to execute the internal and external prior based hyperspectral fused imaging method. Furthermore, the present embodiment also provides a computer-readable storage medium, in which a computer program is stored, the computer program being programmed or configured by a microprocessor to execute the aforementioned hyper-spectral fusion imaging method based on internal and external priors.
As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-readable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein. The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks. These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks. These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
The above description is only a preferred embodiment of the present invention, and the scope of the present invention is not limited to the above embodiments, and all technical solutions that belong to the idea of the present invention belong to the scope of the present invention. It should be noted that modifications and embellishments within the scope of the invention may occur to those skilled in the art without departing from the principle of the invention, and are considered to be within the scope of the invention.
Claims (7)
1. A hyperspectral fused imaging method based on internal and external priors is characterized by comprising the following steps:
s101, learning local similarity of an image from an input multispectral image Z by adopting a superpixel segmentation method to serve as internal prior; learning external prior of the image from any external gray level image set by adopting a deep learning neural network;
s102, introducing an internal prior and an external prior into an input low-resolution hyperspectral image Y and a multispectral image Z to an output high-resolution fusion hyperspectral image X basic observation model to establish an optimized observation model, and solving the optimized observation model to obtain a high-resolution fusion hyperspectral image X;
the step S102 includes:
s301, establishing a correlation representation model shown in the following formula for an input low-resolution hyperspectral image Y and a multispectral image Z to an output high-resolution fused hyperspectral image X;
Y=XBD+N h ，Z=RX+N m ，
in the above formula, B represents fuzzy operation, D represents spatial down-sampling, and N represents h Representing additive noise, R representing a spectral down-sampling matrix, N m Representing additive noise;
s302, decomposing the fused hyperspectral image X into a product of a spectrum subspace E and a coefficient matrix A according to the correlation representation model, and establishing a basic observation model from an input low-resolution hyperspectral image Y and a multispectral image Z to an output high-resolution fused hyperspectral image X shown in the following formula;
in the formula, A is a coefficient matrix, and E is a hyperspectral image Y spectrum subspace;
s303, on the basis of a basic observation model, introducing an internal prior and an external prior as two regular terms, and establishing an optimized observation model shown in the following formula:
in the above formula, the first and second carbon atoms are,η 1 andη 2 the weight parameters of the two regularization terms respectively,
expressed as an internal prior introduced by the coefficient matrix a,kthe number of image blocks constituting an image for the coefficient matrix A, A i Forming an image for the coefficient matrix AiEach of the image blocks is a block of an image,
representing the introduced external prior for the coefficient matrix a;
s304, constructing an augmented Lagrange function according to the optimized observation model, wherein the function expression is as follows:
in the above-mentioned formula, the compound has the following structure,L(A,V 1 ,V 2 ,G 1 ,G 2 ) Representing an augmented Lagrangian function, where V 1 ,V 2 Are respectively the introduced variables, and V 1 ,V 2 Is equal to the coefficient matrix A, G to be estimated 1 ,G 2 Respectively, are lagrange multipliers, respectively,λas a function of the weight parameter(s),μin order to be a penalty factor,
is expressed as a variable V 1 The internal prior introduced is that the internal prior,knumber of image blocks, V, forming an image for coefficient matrix A 1 i Is a variable V 1 To form an imageiEach of the image blocks is a block of an image,
is expressed as a variable V 2 An introduced external prior;
s305, solving the augmented Lagrange function to obtain a high-resolution fusion hyperspectral image X;
the step S305 of solving the augmented lagrangian function to obtain the fused hyperspectral image X includes:
s401, decomposing the augmented Lagrangian function into the following subproblem solving model:
in the above formula, G * 1 ,G * 2 Are respectively updated Lagrange multipliers, mu * The penalty factor after updating is gamma, and the updating coefficient is gamma;
s402, adopting an alternative direction multiplier method ADMM to respectively carry out iterative solution on the subproblem solution model until the number of iterationsk 1 Skipping to step S403 when the threshold value is equal to a preset threshold value K;
and S403, multiplying the obtained coefficient matrix A by the spectrum subspace E to obtain the high-resolution fusion hyperspectral image X.
2. The internal and external prior based hyperspectral fused imaging method according to claim 1, wherein the learning of local similarity of images from the input multispectral image Z using the superpixel segmentation method in step S101 comprises:
s201, partitioning the multispectral image Z into blockskEach image block is subjected to super-pixel segmentation to obtain low-rank characteristics;
s202, multispectral image ZkThe low-rank characteristics of the image blocks are constrained by a nuclear norm to obtain local low-rank characteristics of a multispectral image Z shown by the following formula as introduced internal prior;
in the above formula, the first and second carbon atoms are,knumber of image blocks, Z, for multispectral image Z i Is the second of the multispectral image ZiAnd each image block.
3. The internal and external apriori based hyperspectral fusion imaging method according to claim 1, wherein the method for acquiring the spectrum subspace E of the hyperspectral image Y in the step S302 comprises the following steps: taking L maximum singular values to perform singular value decomposition on the hyperspectral image Y into Y = U 1 Σ 1 V 0 And a matrix U formed by left singular vectors in the matrix U 1 A spectral subspace E as the hyperspectral image Y, where ∑ 1 As a matrix of singular values, V 0 Is a matrix composed of right singular vectors.
4. The internal and external apriori-based hyperspectral fused imaging method according to claim 1, wherein in step S402, the alternating direction multiplier method ADMM is used to solve the sub-problems respectivelyWhen the model is subjected to iterative solution, solving a Hilbert equation aiming at a subproblem solution model of the coefficient matrix A to obtain an updated coefficient matrix A; for variable V 1 The sub-problem solving model adopts a singular value threshold algorithm to realize the optimization of the nuclear norm to solve the updated variable V 1 (ii) a For variable V 2 Is modeled as a variable V 2 The model for solving the subproblems of (A) is regarded as being derived from the equation with a variance ofλ/(2μ) Of Gaussian noise (A-G) 2 /(2μ) For de-noising, using a predetermined deep convolutional neural network input image (A-G) 2 /(2μ) ) and distribution of Gaussian noiseλ/(2μ) And outputting the de-noised gray level image as an updated variable V 2 。
5. The hyperspectral fused imaging method based on interior and exterior priors according to claim 4, wherein the deep convolutional neural network consists of three parts, the first part consists of a first layer containing convolutional and linear rectifying units, the second part consists of 13 layers which are sequentially cascaded and each consists of convolutional, linear rectifying units and batch normalization operating units, and the third part consists of a layer containing only one convolutional operation.
6. An internal and external prior based hyperspectral fused imaging system comprising a microprocessor and a memory connected to each other, characterized in that the microprocessor is programmed or configured to execute the internal and external prior based hyperspectral fused imaging method according to any one of claims 1 to 5.
7. A computer-readable storage medium having a computer program stored thereon for being programmed or configured by a microprocessor to perform the internal and external prior based hyperspectral fusion imaging method of any of claims 1 to 5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211243981.3A CN115311187B (en) | 2022-10-12 | 2022-10-12 | Hyperspectral fusion imaging method, system and medium based on internal and external prior |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211243981.3A CN115311187B (en) | 2022-10-12 | 2022-10-12 | Hyperspectral fusion imaging method, system and medium based on internal and external prior |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115311187A CN115311187A (en) | 2022-11-08 |
| CN115311187B true CN115311187B (en) | 2023-01-24 |
Family
ID=83868298
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211243981.3A Active CN115311187B (en) | 2022-10-12 | 2022-10-12 | Hyperspectral fusion imaging method, system and medium based on internal and external prior |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115311187B (en) |
Families Citing this family (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN118154422A (en) * | 2022-12-05 | 2024-06-07 | 杭州堃博生物科技有限公司 | Image processing method, device, electronic equipment and storage medium |
| CN115760670B (en) * | 2023-01-06 | 2023-05-23 | 之江实验室 | Unsupervised hyperspectral fusion method and device based on network implicit priori |
| CN116245779B (en) * | 2023-05-11 | 2023-08-22 | 四川工程职业技术学院 | Image fusion method and device, storage medium and electronic equipment |
| CN116433551B (en) * | 2023-06-13 | 2023-08-22 | 湖南大学 | High-resolution hyperspectral imaging method and device based on double-light-path RGB fusion |
| CN118096529B (en) * | 2023-10-19 | 2024-09-17 | 成都大学 | Datamation fusion algorithm for improving spatial resolution of hyperspectral image |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111369487A (en) * | 2020-05-26 | 2020-07-03 | 湖南大学 | Hyperspectral and multispectral image fusion method, system and medium |
| WO2020188998A1 (en) * | 2019-03-19 | 2020-09-24 | Mitsubishi Electric Corporation | Systems and methods for multi-spectral image fusion using unrolled projected gradient descent and convolutinoal neural network |
| CN111882512A (en) * | 2020-07-20 | 2020-11-03 | 北京理工大学 | Image fusion method, device and equipment based on deep learning and storage medium |
| CN114998167A (en) * | 2022-05-16 | 2022-09-02 | 电子科技大学 | Hyperspectral and multispectral image fusion method based on space-spectrum combined low rank |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP3510558A4 (en) * | 2016-09-06 | 2020-05-06 | B.G. Negev Technologies and Applications Ltd., at Ben-Gurion University | Recovery of hyperspectral data from image |
-
2022
- 2022-10-12 CN CN202211243981.3A patent/CN115311187B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2020188998A1 (en) * | 2019-03-19 | 2020-09-24 | Mitsubishi Electric Corporation | Systems and methods for multi-spectral image fusion using unrolled projected gradient descent and convolutinoal neural network |
| CN111369487A (en) * | 2020-05-26 | 2020-07-03 | 湖南大学 | Hyperspectral and multispectral image fusion method, system and medium |
| CN111882512A (en) * | 2020-07-20 | 2020-11-03 | 北京理工大学 | Image fusion method, device and equipment based on deep learning and storage medium |
| CN114998167A (en) * | 2022-05-16 | 2022-09-02 | 电子科技大学 | Hyperspectral and multispectral image fusion method based on space-spectrum combined low rank |
Non-Patent Citations (2)
| Title |
|---|
| HYPERSPECTRAL IMAGE SUPER-RESOLUTION VIA LOCAL LOW-RANK AND SPARSE REPRESENTATIONS;Renwei Dian等;《 IGARSS 2018 - 2018 IEEE International Geoscience and Remote Sensing Symposium》;20181104;4003-4005 * |
| 多尺度超像素纹理特征保持与融合的高光谱图像分类;涂兵等;《电子与信息学报》;20220630;第44卷(第6期);2207-2215 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN115311187A (en) | 2022-11-08 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN115311187B (en) | Hyperspectral fusion imaging method, system and medium based on internal and external prior | |
| Borsoi et al. | Deep generative endmember modeling: An application to unsupervised spectral unmixing | |
| Wei et al. | 3-D quasi-recurrent neural network for hyperspectral image denoising | |
| Lin et al. | Hyperspectral image denoising via matrix factorization and deep prior regularization | |
| Sobral et al. | Online stochastic tensor decomposition for background subtraction in multispectral video sequences | |
| CN111369487B (en) | Hyperspectral and multispectral image fusion method, system and medium | |
| CN112801881B (en) | High-resolution hyperspectral calculation imaging method, system and medium | |
| Tang et al. | Regularized simultaneous forward–backward greedy algorithm for sparse unmixing of hyperspectral data | |
| Wei et al. | Intracluster structured low-rank matrix analysis method for hyperspectral denoising | |
| Arun et al. | CNN based spectral super-resolution of remote sensing images | |
| Wen et al. | VIDOSAT: High-dimensional sparsifying transform learning for online video denoising | |
| Shen et al. | Coupling model-and data-driven methods for remote sensing image restoration and fusion: Improving physical interpretability | |
| Long et al. | Dual self-attention Swin transformer for hyperspectral image super-resolution | |
| Zhang et al. | Framelet-based sparse unmixing of hyperspectral images | |
| Jebadurai et al. | SK-SVR: Sigmoid kernel support vector regression based in-scale single image super-resolution | |
| CN112270650B (en) | Image processing method, system, medium, and apparatus based on sparse autoencoder | |
| CN115861076A (en) | Unsupervised hyperspectral image super-resolution method based on matrix decomposition network | |
| CN115457311A (en) | Hyperspectral remote sensing image band selection method based on self-expression transfer learning | |
| Liu et al. | Deep self-supervised hyperspectral image reconstruction | |
| Ma et al. | Gaussian pyramid of conditional generative adversarial network for real-world noisy image denoising | |
| Wen et al. | The power of complementary regularizers: Image recovery via transform learning and low-rank modeling | |
| Weligampola et al. | A retinex based gan pipeline to utilize paired and unpaired datasets for enhancing low light images | |
| Tirer et al. | Deep internal learning: Deep learning from a single input | |
| Xu et al. | A Coupled Tensor Double-Factor Method for Hyperspectral and Multispectral Image Fusion | |
| Qu et al. | A Fast Sparse NMF Optimization Algorithm for Hyperspectral Unmixing |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |