CN113554112A - Remote sensing image fusion method, system, equipment and medium - Google Patents
Remote sensing image fusion method, system, equipment and medium Download PDFInfo
- Publication number
- CN113554112A CN113554112A CN202110874137.XA CN202110874137A CN113554112A CN 113554112 A CN113554112 A CN 113554112A CN 202110874137 A CN202110874137 A CN 202110874137A CN 113554112 A CN113554112 A CN 113554112A
- Authority
- CN
- China
- Prior art keywords
- frequency coefficient
- image
- low
- remote sensing
- fusion
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/25—Fusion techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/213—Feature extraction, e.g. by transforming the feature space; Summarisation; Mappings, e.g. subspace methods
- G06F18/2135—Feature extraction, e.g. by transforming the feature space; Summarisation; Mappings, e.g. subspace methods based on approximation criteria, e.g. principal component analysis
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/50—Image enhancement or restoration by the use of more than one image, e.g. averaging, subtraction
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10032—Satellite or aerial image; Remote sensing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20212—Image combination
- G06T2207/20221—Image fusion; Image merging
Abstract
The invention discloses a method, a system, equipment and a medium for fusing remote sensing images, wherein the method comprises the following steps: acquiring brightness, chroma and saturation components of a source multispectral image; obtaining a full-color image after linear stretching; performing multi-resolution decomposition on the brightness component of the source multispectral image to obtain a low-frequency coefficient M and a high-frequency coefficient M; performing multi-resolution decomposition on the linearly stretched full-color image to obtain a low-frequency coefficient P and a high-frequency coefficient P; fusing the high-frequency coefficient M and the high-frequency coefficient P to obtain a high-frequency coefficient F; fusing the low-frequency coefficient M and the low-frequency coefficient P to obtain a low-frequency coefficient F; performing inverse transformation on the basis of the high-frequency coefficient F and the low-frequency coefficient F to obtain an inverse-transformed luminance component; and carrying out IHS inverse transformation on the chromaticity and saturation components of the source multispectral image and the inversely transformed brightness component to obtain a remote sensing fusion image. The method of the invention can better reserve the space and spectrum information of the image.
Description
Technical Field
The invention belongs to the technical field of remote sensing image processing, and particularly relates to a remote sensing image fusion method, system, equipment and medium.
Background
Image fusion is a process of combining a plurality of source images from different sensors into one image, and the image obtained after fusion has more comprehensive and richer information than the source image, so the image fusion is often used as a preorder step of image processing and analysis. Generally, image fusion can be divided into three levels: pixel level fusion, feature level fusion and decision level fusion; the pixel level fusion aims at improving image definition and spectral expressive force, and provides preparation for subsequent level fusion.
In recent years, image fusion technology is widely applied in the fields of aerospace remote sensing, machine vision, medicine and the like. Massive multispectral images and full-color images acquired by a plurality of aerospace platforms provide data sources for remote sensing image fusion; wherein the multispectral image comprises a plurality of spectral bands, has higher spectral resolution, but has lower spatial resolution; whereas full color images have high spatial resolution but only one band, with lower spectral resolution. In order to better retain the spectral information and definition of the original remote sensing image, a plurality of pixel-level fusion methods of multispectral images and full-color images are provided.
At present, the conventional pixel-level image fusion methods commonly used can be divided into methods based on a spatial domain and methods based on a transform domain. The classical spatial domain-based methods mainly include a weighted average method, a Principal Component Analysis (PCA) method, an Intensity-Hue-Saturation (IHS) method, and a Brooey Transform (BT) method.
The PCA-based image fusion method analyzes an image from the statistical angle, independent components extracted by principal component analysis can well represent the richest information part in a multispectral image, but the method directly uses a full-color image to replace a first principal component, which often causes the loss of spectral information. The IHS-based method can significantly improve spatial resolution, enhance features, enhance color, but it can only process three bands of multispectral images. Spectral information of each wave band of the multispectral image is distributed to the fusion image according to a certain proportional relation based on a Brovey transformation method, and the simple proportional relation causes loss of the spectral information and distortion of brightness. The classical methods based on the change domain mainly include the pyramid transform method, the wavelet transform method and other multi-resolution methods. The laplacian pyramid transform-based method proposed by p.j.burt et al in 1983 effectively improves the fusion effect, but the redundancy of the image is high. Ranchin and l.wald proposed a wavelet transform method in 1993 to solve the redundancy problem between sub-images after decomposition, which has good time-frequency characteristics but no anisotropy.
Disclosure of Invention
The invention aims to provide a remote sensing image fusion method, a remote sensing image fusion system, remote sensing image fusion equipment and a remote sensing image fusion medium, so as to solve one or more technical problems. The method of the invention can better reserve the space and spectrum information of the image.
In order to achieve the purpose, the invention adopts the following technical scheme:
the invention provides a remote sensing image fusion method in the 1 st aspect, which comprises the following steps:
IHS transformation is carried out on the source multispectral image, and the brightness, the chroma and the saturation component of the source multispectral image are obtained; linearly stretching the source full-color image according to the obtained brightness component to obtain a linearly stretched full-color image;
performing multi-resolution decomposition on the brightness component of the source multispectral image to obtain a low-frequency coefficient M and a high-frequency coefficient M; performing multi-resolution decomposition on the linearly stretched full-color image to obtain a low-frequency coefficient P and a high-frequency coefficient P;
fusing the high-frequency coefficient M and the high-frequency coefficient P to obtain a high-frequency coefficient F; fusing the low-frequency coefficient M and the low-frequency coefficient P to obtain a low-frequency coefficient F;
performing inverse transformation on the basis of the high-frequency coefficient F and the low-frequency coefficient F to obtain an inverse-transformed luminance component; and carrying out IHS inverse transformation on the chromaticity and saturation components of the source multispectral image and the inversely transformed brightness component to obtain a remote sensing fusion image.
The method is further improved in that the luminance component of the source multispectral image is subjected to multiresolution decomposition to obtain a low-frequency coefficient M and a high-frequency coefficient M; the multi-resolution decomposition of the linearly stretched full-color image to obtain the low-frequency coefficient P and the high-frequency coefficient P specifically comprises the following steps:
performing multi-resolution decomposition on the brightness component of the source multispectral image by using NSST to obtain a low-frequency coefficient M and a high-frequency coefficient M; performing multi-resolution decomposition on the linearly stretched full-color image by using NSST to obtain a low-frequency coefficient P and a high-frequency coefficient P;
the step of performing inverse transformation based on the high-frequency coefficient F and the low-frequency coefficient F to obtain an inverse transformed luminance component specifically includes: NSST inverse transformation is performed based on the high-frequency coefficient F and the low-frequency coefficient F, and an inverse-transformed luminance component is obtained.
The method of the present invention is further improved in that the step of fusing the high frequency coefficient M and the high frequency coefficient P to obtain the high frequency coefficient F specifically comprises:
and fusing the high-frequency coefficient M and the high-frequency coefficient P by adopting a fusion rule based on the large regional energy of the double thresholds to obtain the high-frequency coefficient F.
The method of the present invention is further improved in that the fusion rule based on the double-threshold region energy measure is specifically:
in the formula, HF(i, j) is the high frequency coefficient value corresponding to the fused image at point (i, j), T1 represents a strong threshold, T2 represents a weak threshold;
rule represents the selection rule of the coefficient value of the point when the ratio value alpha is between the strong threshold and the weak threshold, and the formula is as follows:
in the formula, C1 and C2 are respectively H in the pixel point sets adjacent to the eight points (i, j)P(i, j) and HM(ii) the number of (i, j);
defining the ratio of the two as alpha:
the area energy calculation formula is as follows: eP(i,j)＝(HP(i,j)*ω)2，EM(i,j)＝(HM(i,j)*ω)2；
In the formula: hP(i,j)、HM(i, j) are the corresponding high frequency coefficient values of the panchromatic image and multispectral image at point (i, j), respectively, representing the convolution operation, EP(i,j)、EM(i, j) are the corresponding regional energy values, respectively;
the energy operator of 3 × 3 is defined as:
the method of the present invention is further improved in that the step of fusing the low frequency coefficient M and the low frequency coefficient P to obtain the low frequency coefficient F specifically includes:
and fusing the low-frequency coefficient M and the low-frequency coefficient P by adopting a dictionary learning method to obtain a low-frequency coefficient F.
The method of the present invention is further improved in that the method of learning with a dictionary comprises the steps of:
firstly, sliding block processing is carried out on low-frequency coefficients of two source images by using an 8 x 8 window to obtain a corresponding ith image block
Respectively carrying out sparse representation on overcomplete dictionaries D obtained by pre-training to obtain sparse coefficients
The calculation formula is as follows:
fusing the two groups of sparse coefficients by adopting a rule of a modulus maximum value to obtain fused sparse coefficients
The ith fused image block is obtained after the inverse transformation of the coefficient
The fusion rule is as follows:
the method is further improved in that the dictionary training method adopted by the overcomplete dictionary D obtained by pre-training is MOD or K-SVD.
The invention provides a remote sensing image fusion system in the 2 nd aspect, including:
the preprocessing module is used for carrying out IHS transformation on the source multispectral image to obtain the brightness, the chroma and the saturation components of the source multispectral image; the linear stretching device is used for performing linear stretching on the source full-color image according to the obtained brightness component to obtain a full-color image after linear stretching;
the coefficient acquisition module is used for carrying out multi-resolution decomposition on the brightness component of the source multispectral image to obtain a low-frequency coefficient M and a high-frequency coefficient M; the system is used for carrying out multi-resolution decomposition on the linearly stretched full-color image to obtain a low-frequency coefficient P and a high-frequency coefficient P;
the coefficient fusion module is used for fusing the high-frequency coefficient M and the high-frequency coefficient P to obtain a high-frequency coefficient F; fusing the low-frequency coefficient M and the low-frequency coefficient P to obtain a low-frequency coefficient F;
an inverse transform module for performing NSST inverse transform based on the high frequency coefficient F and the low frequency coefficient F to obtain an inverse transformed luminance component; and performing IHS inverse transformation on the chromaticity and saturation components of the source multispectral image and the inversely transformed brightness component to obtain a remote sensing fusion image.
The invention provides a computer device in the 3 rd aspect, which comprises a memory, a processor and a computer program stored in the memory and capable of running on the processor, wherein the processor executes the computer program to realize the steps of the remote sensing image fusion method according to any one of the above aspects of the invention.
The invention provides, in a 4 th aspect, a computer-readable storage medium, which stores a computer program that, when executed by a processor, implements the steps of the remote sensing image fusion method according to any one of the above aspects of the invention.
Compared with the prior art, the invention has the following beneficial effects:
compared with other traditional methods, the method provided by the invention can better retain the spatial and spectral information of the image. Specifically, (1) the invention improves the spatial resolution of the fused image and enhances the texture detail information of the fused image; (2) the spectrum distortion degree of the fused image is reduced, and the spectrum information of the original multispectral image is better reserved.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art are briefly introduced below; it is obvious that the drawings in the following description are some embodiments of the invention, and that for a person skilled in the art, other drawings can be derived from them without inventive effort.
FIG. 1 is a schematic flow chart of a remote sensing image fusion method according to an embodiment of the present invention;
FIG. 2 is a schematic diagram of an NSST exploded in accordance with an embodiment of the present invention;
FIG. 3 is a source image of an experiment in an embodiment of the present invention; wherein, fig. 3(a) is a first multispectral chart, fig. 3(b) is a first full-color chart, fig. 3(c) is a second multispectral chart, and fig. 3(d) is a second full-color chart;
FIG. 4 is a graph showing experimental results of a first set of data according to an embodiment of the present invention; fig. 4(a) shows a PCA method, fig. 4(b) shows a BT method, fig. 4(c) shows a HIS method, fig. 4(d) shows an NSST method, fig. 4(e) shows a DL method, and fig. 4(f) shows a method according to an embodiment of the present invention;
FIG. 5 is a graph showing experimental results of a second set of data according to an embodiment of the present invention; fig. 5(a) shows a PCA method, fig. 5(b) shows a BT method, fig. 5(c) shows a HIS method, fig. 5(d) shows an NSST method, fig. 5(e) shows a DL method, and fig. 5(f) shows a method according to an embodiment of the present invention.
Detailed Description
In order to make the purpose, technical effect and technical solution of the embodiments of the present invention clearer, the following clearly and completely describes the technical solution of the embodiments of the present invention with reference to the drawings in the embodiments of the present invention; it is to be understood that the described embodiments are only some of the embodiments of the present invention. Other embodiments, which can be derived by one of ordinary skill in the art from the disclosed embodiments without inventive faculty, are intended to be within the scope of the invention.
Referring to fig. 1, a remote sensing image fusion method according to an embodiment of the present invention is a remote sensing image fusion method based on Non-subsampled shear wave Transform (NSST) and Dictionary Learning (DL), and specifically includes the following steps:
the first step is as follows: the method comprises the steps of extracting brightness, chroma and saturation components of a multispectral image with lower spatial resolution, namely performing IHS transformation on the multispectral image, and effectively separating a standard RGB image formed by three wave bands of the multispectral image into an I component representing image spatial features and H and S components representing spectral features so as to facilitate subsequent fusion, wherein an IHS color model is more in line with the description and explanation of human eyes on colors.
The second step is that: the full-color image is linearly stretched according to the I component to reduce spectral distortion.
The third step: the luminance component of the multispectral image and the linearly stretched panchromatic image are each subjected to multiresolution decomposition using NSST. NSST not only retains the good approximation effect of ST on the image edge, but also has translation invariance, and realizes the most effective representation of the image edge.
Referring to fig. 2, NSST first combines classical affine system theory with multi-resolution analysis to construct shear waves. When the dimension n is 2, the resulting dilatant system expression is:
MAB(ψ)＝{ψi,j,k(x)＝|detA|i/2ψ(BjAix-k):
x,y∈Z,k∈Z2} (1)
in the formula: psi ∈ L2(R2) A and B are two-dimensional invertible square matrices and | detB | ═ 1, if M isAB(Ψ) is a tight frame, then M is saidABThe elements in (Ψ) are composite wavelets. i, j, k represent scale, direction and shift parameters, respectively. A is called an anisotropic expansion matrix, AiAssociated with scale transformations, B being called shear matrices, BjAssociated with the region-preserving geometric transformation. When A ═ a 00 a1/2]，B＝[1 s 0 1]The composite wavelet becomes a shear wave. Typically, a is 4, s is 1, so a is 4002]，B＝[1 0 0 1]。
Next, NSST discretization typically includes two parts, multi-resolution decomposition and directional localization.
1) Multi-resolution decomposition
NSST adopts a Non-subsampled Pyramid Filter Bank (NSP) to realize a multi-resolution decomposition process, and the process ensures translation invariance of transformation because the step of downsampling is not included, thereby inhibiting a pseudo Gibbs phenomenon. And decomposing the source image for k times to obtain k high-frequency sub-bands and a low-frequency sub-band which have the same size as the source image.
2) Directional localization
The process is done by a shear wave Filter (SF), also avoiding the down-sampling process, and thus having translational invariance. In the process, each sub-band is decomposed along l different directions to obtain l direction sub-bands with the same size as the source image.
The fourth step: and fusing the high-frequency coefficients.
The high frequency coefficients contain details and edge feature information of the source image. At the high frequency level, the image composed of these coefficients highlights the directional edges. Considering that adjacent pixels have certain structural correlation, when a certain pixel and the adjacent pixels thereof are considered as a region, the image semantic is better met, the influence of interference noise can be reduced, and a better visual effect can be obtained. Therefore, the invention designs a new fusion rule based on the large regional energy of the dual threshold.
The energy operator of 3 × 3 is defined as follows:
the area energy calculation formula is as follows:
EP(i,j)＝(HP(i,j)*ω)2 (3)
EM(i,j)＝(HM(i,j)*ω)2 (4)
in the formula: hP(i,j)、HM(i, j) are the corresponding high frequency coefficient values of the panchromatic image and multispectral image at point (i, j), respectively, representing the convolution operation, EP(i,j)、EM(i, j) are the corresponding regional energy values, respectively. Defining the ratio of the two as alpha:
common fusion rules based on regional energy basically adopt a method of taking a large absolute value or a large regional energy to improve the spatial resolution of an image. But when the energy value E of a certain region of the high-frequency coefficient of the full-color imageP(i, j) and the corresponding regional energy value E of the high-frequency coefficients of the multispectral imageMWhen the difference between the (i, j) is small, the effect of increasing the simple absolute value on improving the definition of the fused image is very limited, but the correlation between the regions can be damaged, so that the distortion and the distortion of the spectrum are caused.
The fusion rule for taking large regional energy based on the double thresholds provided by the invention is as follows:
in the formula: hF(i, j) is the high-frequency coefficient value corresponding to the fused image at the point (i, j), T1 represents a strong threshold, T2 represents a weak threshold, rule represents the rule of selecting the coefficient value of the point when the ratio α is between the strong threshold and the weak threshold, and the formula is as follows:
in the formula: c1 and C2 are respectively H in the pixel point set adjacent to the eight points (i, j)P(i, j) and HM(i, j) number.
The fifth step: and fusing the low-frequency coefficients.
The low-frequency coefficient can be regarded as an approximate image of the source image, mainly reflects the general appearance information of the source image, and can be regarded as an image after filtering and denoising or blurring. The traditional fusion rules include a weighted average method and the like, and the methods are simple in rules and high in running speed, but the problems of information loss, edge distortion, spectrum distortion and the like of fused images are often caused. The low-frequency coefficient has dense sample characteristics, the redundancy of the image can be reduced through sparse representation, and effective information can be represented better, so that the low-frequency coefficient is fused by using a dictionary learning method.
Sparse representations are effective tools for characterizing the human visual system, and have been successfully applied in the fields of pattern recognition, machine age, image analysis, and the like. The over-complete sparse representation is more in line with the sparse coding mechanism of the visual cortical neurons of the mammals, and simultaneously, the model complexity is reduced, so that the related tasks of image processing are simplified. One of the main lines of current image sparse representation system research is: a redundant system called an overcomplete dictionary is constructed to sparsely represent the image (dense samples), i.e. dictionary learning. The sparse signals obtained based on dictionary learning reduce the redundancy of the original image, so that effective information can be extracted and retained.
The representation model of dictionary learning relies on the assumption that: many natural signals may be represented or approximated as linear combinations of a small number of dictionary atoms. I.e. given an overcomplete dictionary of M n-dimensional signals D e Rn×M(n < M), the signal y ∈ R can be setnExpressed as y ═ Dx or y ≈ Dx, where the vector x is called the sparse coefficient. The solution to vector x is not unique based on the redundancy of an overcomplete dictionary, and therefore, a sparse representation model is proposed as a method for determining the solution to vector x with a minimum of non-zero components. Mathematically, assuming that the optimization problem ignores noise effects, the optimization model for solving the problem is:
min||x||0s.t.y ═ Dx or
In the formula: | x | non-conducting phosphor0Denotes the number of non-zero components in x, and ε denotes the allowable deviation.
The above optimization problem is an NP-hard problem, and therefore requires the use of Matching Pursuits (MP), Orthogonal Matching Pursuits (OMP) or Simultaneous Orthogonal Matching Pursuits (SOMP) algorithms to obtain a low complexity solution, and is generally referred to as sparse coding.
The selection of the over-complete dictionary determines the capability of sparse coding of the image signal, and plays a key role in sparse representation. The dictionary is designed by using a training-based method, atoms in the dictionary can be trained according to the characteristics of the image, so that the image can be represented more sparsely, and the performance is more excellent. The current common dictionary training methods include MOD, K-SVD and the like.
Firstly, sliding block processing is carried out on low-frequency coefficients of two source images by using an 8 x 8 window to obtain a corresponding ith image block
Respectively carrying out sparse representation on the overcomplete dictionaries D obtained by training to obtain sparse coefficients
The calculation formula is as follows:
the maximum value in the sparse coefficients represents the most singular region (such as edge) in the source image, so that the two groups of sparse coefficients are selected by adopting the rule of modulus maximum value to obtain the fused sparse coefficients
The ith fused image block is obtained after the inverse transformation of the coefficient
The fusion rule is as follows:
and a sixth step: and performing NSST inverse transformation and IHS inverse transformation to obtain a fused image.
The method provided by the embodiment of the invention combines the advantages that NSST has multi-resolution analysis and dictionary learning sparse representation and can reduce the image redundancy, is better compared with other traditional methods, and can better reserve the spatial and spectral information of the image.
Referring to fig. 3 to 5, to verify the effectiveness of the method according to the embodiment of the present invention, two sets of three-band multispectral images and full-color images are fused for the experiment. The first set of images was acquired by LANDSAT satellites and were sampled for pre-processing, both 256 x 256 in size. The second set of images was acquired by a QuickBird satellite and sample pre-processed, both images were 227 x 227 in size and their grayscale values were quantized to 8 bits. The class 5 comparison algorithm includes: PCA-based method (PCA), BT-based method (BT), IHS-based method (IHS), NSST-based method (NSST), DL-based method (DL).
The full-color image has higher spatial resolution and richer detail characteristics, and the dictionary obtained by training the full-color image as a sample has stronger representation capability, so that the full-color image in the two groups of data is used as the training sample. Dividing the full-color image 1 into 8 × 8 image blocks by using a slider method, sequencing the image blocks by columns to obtain a 64 × 85849 matrix, training by using a K-SVD algorithm to obtain a 64 × 256 overcomplete dictionary D1, and obtaining an overcomplete dictionary D2 in the same way.
The experimental operating system was Windows 10 and the programming environment was MATLAB R2018 b. According to the past experimental experience, the multi-resolution decomposition filter is a 9-7 wavelet filter, the number of decomposition layers k of NSST is 4, and the direction numbers corresponding to the layers are respectively L ═ 23，23，24，25]T1 is 1.5 and T2 is 1. The analysis of pixel-level image fusion is usually based on spectral features and spatial features, and in order to better evaluate the image fusion effect, the invention performs experimental analysis on the fused image from both subjective and objective aspects.
The subjective evaluations were as follows: from experimental results, all methods can extract spectral and spatial information from a multispectral image and a panchromatic image and inject the spectral and spatial information into a fused image, but images obtained by the traditional method have some defects. For example, PCA-based methods often introduce spectral distortions due to the omission of the characteristics of the various bands. BT based methods typically reduce contrast, which means shading, and are susceptible to noise. IHS-based methods tend to cause false color and distortion due to direct replacement of the luminance component. NSST-based methods, while better retaining detailed information, can cause severe spectral distortions. The DL-based method achieves a better visual effect, but ignores the multi-resolution characteristic of human vision and is susceptible to noise. Compared with the method, the method improves the visual effect of the image, the fused image is clearer, and the spectral distortion degree is further reduced.
Objective evaluation fusion results were evaluated and analyzed using four evaluation indexes, Standard Deviation (SD), Entropy (E), Average Gradient (AG), and Spectral Distortion Degree (SDD).
1) Standard deviation of
The standard deviation is an index for measuring the overall fluctuation of the gray value of the image and is calculated by the gray value of the image and the mean value of the gray value. The larger the standard deviation value, the sharper the image is. The calculation formula is as follows:
in the formula: g (i, j) represents the gray value of the fused image at point (i, j),
the mean gray value of the fused image is represented, and M × N represents the size of the fused image.
2) Entropy of the entropy
The information entropy is an index for measuring the richness of the image information. The larger the entropy value, the richer the information contained in the representation image. The calculation formula is as follows:
in the formula: piAnd representing the probability that the gray value of the pixel point in the fused image is i.
3) Mean gradient
The average gradient, also commonly referred to as sharpness, reflects the contrast of the texture in the image. The larger the average gradient value, the sharper the image. The calculation formula is as follows:
in the formula: f (i, j), f (i +1, j) and f (i, j +1) are the gray values of the fused image at points (i, j), (i +1, j) and (i, j +1), respectively, and M × N represents the size of the fused image.
4) Degree of spectral distortion
The degree of spectral distortion reflects the degree of spectral distortion that exists between the fused image and the multi-spectral image. The smaller the spectral distortion degree is, the more accurate the spectral information carried by the fused image is. The calculation formula is as follows:
in the formula: f (i, j) represents the grayscale value of the multispectral image at point (i, j), g (i, j) represents the grayscale value of the fused image at point (i, j), and M × N represents the size of the fused image.
Specific numerical values of objective evaluation indexes of the two experiments are shown in tables 1 and 2, and the specific numerical values can be known from tables 1 and 2, so that all the evaluation indexes of the method disclosed by the embodiment of the invention are optimal. I.e. the values of standard deviation, entropy and mean gradient are the largest, which means that the fused image obtained by the method is clearer and contains richer spatial information. The spectral distortion degree is minimum, which shows that the fused image obtained by the method better retains the spectral information of the multispectral image.
TABLE 1 first set of data evaluation index values
TABLE 2 evaluation index values for the second set of data
In yet another embodiment of the present invention, a computer device is provided that includes a processor and a memory for storing a computer program comprising program instructions, the processor for executing the program instructions stored by the computer storage medium. The Processor may be a Central Processing Unit (CPU), or may be other general-purpose Processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), an off-the-shelf Programmable gate array (FPGA) or other Programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, etc., which is a computing core and a control core of the terminal, and is specifically adapted to load and execute one or more instructions in a computer storage medium to implement a corresponding method flow or a corresponding function; the processor according to the embodiment of the present invention may be used for the operation of the image retrieval method based on the triangle inequality.
In yet another embodiment of the present invention, the present invention further provides a storage medium, specifically a computer-readable storage medium (Memory), which is a Memory device in a computer device and is used for storing programs and data. It is understood that the computer readable storage medium herein can include both built-in storage media in the computer device and, of course, extended storage media supported by the computer device. The computer-readable storage medium provides a storage space storing an operating system of the terminal. Also, one or more instructions, which may be one or more computer programs (including program code), are stored in the memory space and are adapted to be loaded and executed by the processor. It should be noted that the computer-readable storage medium may be a high-speed RAM memory, or may be a non-volatile memory (non-volatile memory), such as at least one disk memory. One or more instructions stored in the computer-readable storage medium may be loaded and executed by a processor to implement the corresponding steps of the image retrieval method based on the triangle inequality in the above embodiments.
In summary, the present invention relates to a remote sensing image fusion method based on Non-subsampled shear wave Transform (NSST) and Dictionary Learning (DL). Firstly, the luminance, chrominance and saturation components of the multispectral image with lower spatial resolution are extracted, and then NSST is used for carrying out multiresolution decomposition on the luminance component of the multispectral image and the panchromatic image after linear stretching. The invention designs a new double-threshold fusion rule based on the region energy aiming at the high-frequency coefficient, and the original spectrum information is better kept while the image resolution is improved; the low-frequency coefficient adopts a fusion rule based on dictionary learning, so that the redundancy is reduced, and effective information in the low-frequency coefficient is better fused. And finally, carrying out corresponding inverse transformation to obtain a final fusion image. Experimental results show that the method can better retain the spatial and spectral information of the image.
As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art can make modifications and equivalents to the embodiments of the present invention without departing from the spirit and scope of the present invention, which is set forth in the claims of the present application.
Claims (10)
1. A remote sensing image fusion method is characterized by comprising the following steps:
IHS transformation is carried out on the source multispectral image, and the brightness, the chroma and the saturation component of the source multispectral image are obtained; linearly stretching the source full-color image according to the obtained brightness component to obtain a linearly stretched full-color image;
performing multi-resolution decomposition on the brightness component of the source multispectral image to obtain a low-frequency coefficient M and a high-frequency coefficient M; performing multi-resolution decomposition on the linearly stretched full-color image to obtain a low-frequency coefficient P and a high-frequency coefficient P;
fusing the high-frequency coefficient M and the high-frequency coefficient P to obtain a high-frequency coefficient F; fusing the low-frequency coefficient M and the low-frequency coefficient P to obtain a low-frequency coefficient F;
performing inverse transformation on the basis of the high-frequency coefficient F and the low-frequency coefficient F to obtain an inverse-transformed luminance component; and carrying out IHS inverse transformation on the chromaticity and saturation components of the source multispectral image and the inversely transformed brightness component to obtain a remote sensing fusion image.
2. The remote sensing image fusion method according to claim 1, wherein the luminance component of the source multispectral image is decomposed in a multiresolution manner to obtain a low-frequency coefficient M and a high-frequency coefficient M; the multi-resolution decomposition of the linearly stretched full-color image to obtain the low-frequency coefficient P and the high-frequency coefficient P specifically comprises the following steps:
performing multi-resolution decomposition on the brightness component of the source multispectral image by using NSST to obtain a low-frequency coefficient M and a high-frequency coefficient M; performing multi-resolution decomposition on the linearly stretched full-color image by using NSST to obtain a low-frequency coefficient P and a high-frequency coefficient P;
the specific step of performing inverse transformation based on the high-frequency coefficient F and the low-frequency coefficient F to obtain the inversely transformed luminance component includes: NSST inverse transformation is performed based on the high-frequency coefficient F and the low-frequency coefficient F, and an inverse-transformed luminance component is obtained.
3. The remote sensing image fusion method according to claim 1, wherein the step of fusing the high-frequency coefficient M and the high-frequency coefficient P to obtain the high-frequency coefficient F specifically comprises:
and fusing the high-frequency coefficient M and the high-frequency coefficient P by adopting a fusion rule based on the large regional energy of the double thresholds to obtain the high-frequency coefficient F.
4. The remote sensing image fusion method according to claim 3, wherein the fusion rule for obtaining the large regional energy based on the dual threshold specifically comprises:
in the formula, HF(i, j) is fused image inThe corresponding high frequency coefficient value at point (i, j), T1 for the strong threshold, T2 for the weak threshold;
rule represents the selection rule of the coefficient value of the point when the ratio value alpha is between the strong threshold and the weak threshold, and the formula is as follows:
in the formula, C1 and C2 are respectively H in the pixel point sets adjacent to the eight points (i, j)P(i, j) and HM(ii) the number of (i, j);
defining the ratio of the two as alpha:
the area energy calculation formula is as follows: eP(i,j)＝(HP(i,j)*ω)2，EM(i,j)＝(HM(i,j)*ω)2；
In the formula: hP(i,j)、HM(i, j) are the corresponding high frequency coefficient values of the panchromatic image and multispectral image at point (i, j), respectively, representing the convolution operation, EP(i,j)、EM(i, j) are the corresponding regional energy values, respectively;
the energy operator of 3 × 3 is defined as:
5. the remote sensing image fusion method according to claim 1, wherein the step of fusing the low-frequency coefficient M and the low-frequency coefficient P to obtain the low-frequency coefficient F specifically comprises:
and fusing the low-frequency coefficient M and the low-frequency coefficient P by adopting a dictionary learning method to obtain a low-frequency coefficient F.
6. The remote sensing image fusion method according to claim 5, wherein the method using dictionary learning comprises the steps of:
firstly, sliding block processing is carried out on low-frequency coefficients of two source images by using an 8 x 8 window to obtain a corresponding ith image block
Respectively carrying out sparse representation on overcomplete dictionaries D obtained by pre-training to obtain sparse coefficients
The calculation formula is as follows:
fusing the two groups of sparse coefficients by adopting a rule of a modulus maximum value to obtain fused sparse coefficients
The ith fused image block is obtained after the inverse transformation of the coefficient
The fusion rule is as follows:
7. the remote sensing image fusion method according to claim 6, wherein the dictionary training method adopted by the overcomplete dictionary D obtained by pre-training is MOD or K-SVD.
8. A remote sensing image fusion system, comprising:
the preprocessing module is used for carrying out IHS transformation on the source multispectral image to obtain the brightness, the chroma and the saturation components of the source multispectral image; the linear stretching device is used for performing linear stretching on the source full-color image according to the obtained brightness component to obtain a full-color image after linear stretching;
the coefficient acquisition module is used for carrying out multi-resolution decomposition on the brightness component of the source multispectral image to obtain a low-frequency coefficient M and a high-frequency coefficient M; the system is used for carrying out multi-resolution decomposition on the linearly stretched full-color image to obtain a low-frequency coefficient P and a high-frequency coefficient P;
the coefficient fusion module is used for fusing the high-frequency coefficient M and the high-frequency coefficient P to obtain a high-frequency coefficient F; fusing the low-frequency coefficient M and the low-frequency coefficient P to obtain a low-frequency coefficient F;
the inverse transformation module is used for carrying out inverse transformation on the basis of the high-frequency coefficient F and the low-frequency coefficient F to obtain an inverse transformed brightness component; and performing IHS inverse transformation on the chromaticity and saturation components of the source multispectral image and the inversely transformed brightness component to obtain a remote sensing fusion image.
9. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that the processor implements the steps of the method of fusion of remote sensing images according to any of claims 1 to 7 when executing the computer program.
10. A computer-readable storage medium, in which a computer program is stored which, when being executed by a processor, carries out the steps of the method for remote sensing image fusion according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110874137.XA CN113554112B (en) | 2021-07-30 | 2021-07-30 | Remote sensing image fusion method, system, equipment and medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110874137.XA CN113554112B (en) | 2021-07-30 | 2021-07-30 | Remote sensing image fusion method, system, equipment and medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113554112A true CN113554112A (en) | 2021-10-26 |
| CN113554112B CN113554112B (en) | 2023-09-19 |
Family
ID=78105062
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110874137.XA Active CN113554112B (en) | 2021-07-30 | 2021-07-30 | Remote sensing image fusion method, system, equipment and medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113554112B (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114119443A (en) * | 2021-11-28 | 2022-03-01 | 特斯联科技集团有限公司 | Image fusion system based on multispectral camera |
| CN115019189A (en) * | 2022-04-08 | 2022-09-06 | 辽宁师范大学 | Hyperspectral image change detection method based on NSST hidden Markov forest model |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20130129201A1 (en) * | 2011-11-18 | 2013-05-23 | Dehong Liu | Method for Pan-Sharpening Panchromatic and Multispectral Images Using Wavelet Dictionaries |
| CN105761234A (en) * | 2016-01-28 | 2016-07-13 | 华南农业大学 | Structure sparse representation-based remote sensing image fusion method |
| CN107657217A (en) * | 2017-09-12 | 2018-02-02 | 电子科技大学 | The fusion method of infrared and visible light video based on moving object detection |
| WO2018047171A1 (en) * | 2016-09-06 | 2018-03-15 | B. G. Negev Technologies And Applications Ltd., At Ben-Gurion University | Recovery of hyperspectral data from image |
| CN108921809A (en) * | 2018-06-11 | 2018-11-30 | 上海海洋大学 | Multispectral and panchromatic image fusion method under integral principle based on spatial frequency |
-
2021
- 2021-07-30 CN CN202110874137.XA patent/CN113554112B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20130129201A1 (en) * | 2011-11-18 | 2013-05-23 | Dehong Liu | Method for Pan-Sharpening Panchromatic and Multispectral Images Using Wavelet Dictionaries |
| CN105761234A (en) * | 2016-01-28 | 2016-07-13 | 华南农业大学 | Structure sparse representation-based remote sensing image fusion method |
| WO2018047171A1 (en) * | 2016-09-06 | 2018-03-15 | B. G. Negev Technologies And Applications Ltd., At Ben-Gurion University | Recovery of hyperspectral data from image |
| CN107657217A (en) * | 2017-09-12 | 2018-02-02 | 电子科技大学 | The fusion method of infrared and visible light video based on moving object detection |
| CN108921809A (en) * | 2018-06-11 | 2018-11-30 | 上海海洋大学 | Multispectral and panchromatic image fusion method under integral principle based on spatial frequency |
Non-Patent Citations (4)
| Title |
|---|
| 吴一全;陶飞翔;: "改进投影梯度NMF的NSST域多光谱与全色图像融合", 光学学报, no. 04 * |
| 崔怡文;任佳佳;: "基于非下采样Shearlet变换与差异度量的遥感图像融合算法", 电子测量与仪器学报, no. 12 * |
| 王芳;陈燕;: "基于信息量制约的遥感图像融合算法", 太赫兹科学与电子信息学报, no. 03 * |
| 金益如;杨学志;董张玉;郑鑫;李国强;: "一种NSST与稀疏表示相结合的遥感图像融合算法", 地理与地理信息科学, no. 02 * |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114119443A (en) * | 2021-11-28 | 2022-03-01 | 特斯联科技集团有限公司 | Image fusion system based on multispectral camera |
| CN114119443B (en) * | 2021-11-28 | 2022-07-01 | 特斯联科技集团有限公司 | Image fusion system based on multispectral camera |
| CN115019189A (en) * | 2022-04-08 | 2022-09-06 | 辽宁师范大学 | Hyperspectral image change detection method based on NSST hidden Markov forest model |
| CN115019189B (en) * | 2022-04-08 | 2024-04-05 | 辽宁师范大学 | NSST hidden Markov forest model-based hyperspectral image change detection method |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113554112B (en) | 2023-09-19 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN114140353B (en) | Swin-Transformer image denoising method and system based on channel attention | |
| Dong et al. | Learning a deep convolutional network for image super-resolution | |
| JP6352420B2 (en) | Method and device for determining a high resolution output image | |
| Xu et al. | Image inpainting by patch propagation using patch sparsity | |
| CN103093433B (en) | Natural image denoising method based on regionalism and dictionary learning | |
| CN113554112B (en) | Remote sensing image fusion method, system, equipment and medium | |
| Davy et al. | Non-local video denoising by CNN | |
| Zhou et al. | Method of improved fuzzy contrast combined adaptive threshold in NSCT for medical image enhancement | |
| CN110598584A (en) | Convolutional neural network face recognition algorithm based on wavelet transform and DCT | |
| CN104657951A (en) | Multiplicative noise removal method for image | |
| Juefei-Xu et al. | Pokerface: partial order keeping and energy repressing method for extreme face illumination normalization | |
| Wang et al. | Multi-wavelet residual dense convolutional neural network for image denoising | |
| CN113724164A (en) | Visible light image noise removing method based on fusion reconstruction guidance filtering | |
| Abdullah-Al-Wadud | A modified histogram equalization for contrast enhancement preserving the small parts in images | |
| CN113962882A (en) | JPEG image compression artifact eliminating method based on controllable pyramid wavelet network | |
| Yadav et al. | Underwater image enhancement using convolutional neural network | |
| CN116452930A (en) | Multispectral image fusion method and multispectral image fusion system based on frequency domain enhancement in degradation environment | |
| Ngocho et al. | Single image super resolution with guided back-projection and LoG sharpening | |
| CN107563982A (en) | A kind of Enhancement Methods about Satellite Images | |
| Chitkara et al. | Comparative analysis of image fusion methods | |
| Wang et al. | Division gets better: Learning brightness-aware and detail-sensitive representations for low-light image enhancement | |
| Al-Khafaji | Image Improvement Using The Combination of Wavelet and Multiwavelet Transform | |
| Chen et al. | Image fusion based on curvelet transform and fuzzy logic | |
| Talebi et al. | Asymptotic performance of global denoising | |
| Liu et al. | Hierarchical subspace regression for compressed face image restoration |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |