CN106897987A - Image interfusion method based on translation invariant shearing wave and stack own coding - Google Patents

Abstract

The invention discloses a kind of image interfusion method based on translation invariant shearing wave and stack own coding.Implementation step is：Wave conversion is sheared by picture breakdown to be fused into low frequency sub-band coefficient and high-frequency sub-band coefficient with translation invariant first；Secondly, low frequency sub-band coefficient reflects the base profile of image, and the method being averaged using weighting is merged；High-frequency sub-band coefficient reflects the edge and texture information of image, the present invention proposes a kind of fusion method based on stack own coding feature, using sliding the method for piecemeal by high-frequency sub-band piecemeal, stack autoencoder network is trained as input using fritter, again fritter encode using the network for training obtaining feature, and utilization space frequency carries out feature enhancing and obtains activity and estimate, finally estimating numerical value using this activity and taking big fusion rule carries out the fusion of high-frequency sub-band coefficient fritter, and high-frequency sub-band is obtained using sliding window inverse transformation after all fritters fusions；Image after finally being merged using translation invariant shearing wave inverse transformation.The present invention can preferably retain edge and texture information in original image compared to traditional fusion method.

Description

Image fusion method based on translation invariant shear wave and stack type self-coding

Technical Field

The invention relates to an image fusion method based on translation invariant shear wave and stack type self-coding, which is a fusion method in the technical field of image processing and has wide application in military application, clinical medical diagnosis and the like.

Background

Because the information contained in a single image is limited, the single image often cannot meet the practical application. The image fusion is a technology for synthesizing an image of the same scene acquired by a plurality of sensors through the processing of a fusion algorithm, and the fused image can effectively combine the advantages of a plurality of images to be fused, so that the image fusion is more suitable for the visual perception of people. Image fusion has been proposed in the seventies of the twentieth century, and in recent years, due to rapid development of multi-sensor technology, image fusion has been widely applied in the fields of military reconnaissance, medical diagnosis, remote sensing and the like.

The image fusion method can be roughly divided into image fusion in the spatial domain and image fusion in the transform domain. The image fusion in the space domain refers to a method for directly fusing pixel points of an image, and mainly comprises weighted image fusion and image fusion based on principal component analysis. Image fusion in the transform domain generally refers to a method of performing multi-scale decomposition on an image and then fusing each sub-band coefficient, and tools of the multi-scale decomposition include wavelet transform (DWT), Stationary Wavelet Transform (SWT), Contourlet Transform (CT), non-downsampling contourlet transform (NSCT), shear wave transform (ST), and shift-invariant shear wave transform (SIST). DWT is proposed by Mallat in 1989, has the advantages of time-frequency localization, multi-resolution and the like, but has limited direction information which can be captured and has Gibbs oscillation phenomenon. Rockinger proposed that SWT could suppress the gibbs oscillation phenomenon of DWT, but the high frequency subbands of SWT still have only three directions, horizontal, vertical and diagonal. The CT proposed by Minh n.do has strong direction selection capability, but the CT lacks translation invariance because of the down-sampling operation during the transformation process. The NSCT proposed by Cunha uses a non-downsampled filter and has translational invariance. Guo et al propose ST, which is more efficient to compute than CT and has no limitation on the number of directions, however ST has no translational invariance due to the use of downsampling filters. SIST proposed by Easley et al adopts non-downsampling pyramid decomposition, and has the characteristic of unchanged translation, so that the SIST has a better development prospect in the field of image fusion.

The design of the fusion rule determines the image fusion effect. The main image fusion step based on the transform domain is to design a proper scheme to fuse sub-band coefficients of each layer, so as to obtain a fused image more conforming to human visual perception. This step involves two basic problems: and selecting a fusion strategy and selecting activity measure.

A common subband coefficient fusion rule includes absolute value maximization and weighted averaging. The absolute value is large, that is, the subband coefficient with a large activity measure is selected as the fused subband coefficient. The weighting and averaging rule calculates the weight of the corresponding sub-band coefficient according to the activity measure and then the sub-band coefficients are weighted and fused. The method of weighted averaging can better keep the useful information of the image to be fused, and the fusion rule of taking the larger absolute value can improve the contrast of the fused image.

The design of the activity measure can also be understood as the characteristic selection of pixel or subband coefficients, and requires the characteristics of the response data which can be accurate from a certain angle. In the fusion method based on Principal Component Analysis (PCA), He et al first obtains principal components from eigenvalues and eigenvectors of a correlation coefficient matrix between band low-resolution images, then performs gray scale stretching on the band high-resolution images and makes the band high-resolution images have the same mean and variance as the principal components, and finally replaces the principal components with the high-resolution images and utilizes PCA inverse transformation to obtain fused images. The tensor (tensor) has prominent expression on the description of high-dimensional data and has good effect in the application of image fusion. Liang et al first decomposed the image with a high order singular value decomposition tool and then constructed the fusion rule with the 1 norm of the coefficient as the measure of activity. Sparse representation can describe complex data by linear combination of dictionary elements, and has been widely applied to image fusion. Yang et al first segments the image into overlapping patches, then decomposes the image patches to obtain sparse coefficients, and finally designs a fusion rule for the sparse coefficients. In recent years, the application of a deep learning method in the field of computer vision has been greatly successful, the deep learning method is good at exploring layered characteristics from complex data sets, but because of the lack of sufficient labeled training sets in the practical application of image fusion, supervised learning methods such as Convolutional Neural Network (CNN) and the like have no precedent application in image fusion, and the stacked self-encoding (SAE) is taken as an unsupervised learning deep neural network and conforms to the scene of image fusion application. The SAE network obtained by stacking multilayer self-encoding (AE) can not only discover the layered characteristics of the image, but also make the extracted characteristics sparse if sparse constraint (SSAE) is set on hidden layer neurons, thereby better meeting the requirements of image fusion application.

Disclosure of Invention

The invention aims to provide an image fusion algorithm based on a translation invariant shear wave and a stacked self-coding network, aiming at the defects of the prior art, so as to protect the details of an image, enhance the contrast and contour edge of the image, improve the visual effect of the image and improve the quality of image fusion. The specific technical scheme of the invention is as follows:

1) firstly, two images to be fused are subjected to SIST transformation and decomposed to obtain a low-frequency sub-bandAnd high frequency sub-bands

2) Designing different fusion rules aiming at the low-frequency subband coefficients and the high-frequency subband coefficients:

2.1) the low frequency subband coefficients contain the basic information of the image, and thereforeAndusing an averaging strategy fusion:

the low-frequency sub-band coefficient obtained by SIST decomposition of the image contains the main energy information of the image, which is the approximate component of the original image, and the low-frequency sub-band coefficient is fused by averaging, that is

Wherein,andthe original image A, B and the fused image F are each represented by a corresponding low-frequency coefficient at point (x, y).

2.2) high frequency subband coefficientsIt contains the information of the details of the system,andusing a fusion rule based on SSAE features to get large: the high frequency subband coefficients contain texture information of the image. SSAE, as a deep learning method, is very good at learning high-dimensional structural features from complex data sets. Considering the factors, the invention provides a fusion rule based on SSAE characteristics, firstly, the sub-band to be fused is divided into blocks, and the small blocks are converted into vectorsThen, the two-layer SSAE network is trained as training data, and then the network is used for coding the small blocks to be fused to obtain characteristicsThen pairCalculating spatial frequencyBy usingTaking a large rule fusion small block, wherein s ∈ { A, B };

finally, all the fused vectors are subjected toFirstly converting the matrix into a matrix form, and then obtaining a fusion coefficient matrix by adopting inverse transformation of sliding window transformationAnd adopting an averaging strategy in the overlapping area.

3) And obtaining the fused image by using SIST inverse transformation.

Compared with the existing medical image fusion method, the invention has the following advantages:

1. the invention adopts the transformation (SIST) based on the translation invariant shear wave as a multi-scale decomposition tool, compared with the wavelet transformation (DWT), the SIST can capture more direction information and eliminate the pseudo Gibbs phenomenon; SIST obtains more high frequency sub-band directions compared with Stationary Wavelet Transform (SWT); compared with the problem that Contourlet Transform (CT) lacks of translation invariance, SIST does not have the operation of down-sampling, so that the method has translation invariance; compared with non-subsampled contourlet transform (NSCT), SIST has no limit of direction number and is more efficient in calculation.

2. The invention adopts a stacked encoder (SSAE) as a feature extraction tool, compared with Principal Component Analysis (PCA), the SSAE is a data-driven feature extraction tool, and can learn unique rules from input images, thereby extracting more representative features than manually designed features; compared with tensor (tensor) and Pulse Coupled Neural Network (PCNN), SSAE can extract hierarchical features as a depth network, and is more suitable for image data representation than tensor and PCNN.

3. The spatial frequency is introduced when the activity measure is constructed, compared with the method of directly using SSAE (single-phase analysis of acoustic echo) characteristics, the activity measure with the introduced spatial frequency has stronger local contrast, and the characteristics of the original image can be better represented; the edge and texture information of the image can be protected to a greater extent.

Description of the drawings:

FIG. 1 is a diagram of the overall fusion framework of the overall invention.

Fig. 2 is a block diagram of a translation invariant shear wave transformation.

Fig. 3 is a block diagram of a single-layer sparse auto-encoder.

Fig. 4 is a structural diagram of a stacked self-encoder.

Fig. 5 is a diagram illustrating a high-frequency coefficient subband fusion rule.

FIGS. 6(a) and (b) are multi-focus images to be fused according to the first embodiment of the present invention; (c) is a GP-based fused image; (d) is a DWT based fused image; (e) is a fused image based on stDWT; (f) is a PCNN-based fused image; (g) is a fusion image based on N-PCNN; (h) is a SR-based fused image; (i) is a fused image of the method of the present invention.

FIG. 7(a) is a visible light image to be fused according to one embodiment of the present invention; (b) is an infrared image to be fused according to an embodiment of the present invention; (c) is a GP-based fused image; (d) is a DWT based fused image; (e) is a fused image based on stDWT; (f) is a PCNN-based fused image; (g) is a fusion image based on N-PCNN; (h) is a SR-based fused image; (i) is a fused image of the method of the present invention.

FIG. 8(a) is a CT image to be fused according to an embodiment of the present invention; (b) is an MRI image to be fused according to an embodiment of the invention; (c) is a GP-based fused image; (d) is a DWT based fused image; (e) is a fused image based on stDWT; (f) is a PCNN-based fused image; (g) is a fusion image based on N-PCNN; (h) is a SR-based fused image; (i) is a fused image of the method of the present invention.

The specific implementation mode is as follows:

the following is a detailed description of embodiments of the present invention with reference to the drawings, which is performed on the premise of the technical solution of the present invention, as shown in fig. 1, and the detailed implementation and specific operation steps are as follows:

step 1, decomposing two multi-focus images to be fused by using SIST (Single input Single output) transformation, and obtaining a low-frequency sub-band after the images are decomposedAnd high frequency sub-bandsWherein the dimension decomposition LP is selected from 'maxflat', the direction filter bank is selected from 'pmaxflat', and the direction decomposition parameters are set to be [3,4,5 ]]；

Step 2, low-frequency sub-band coefficient fusion and high-frequency sub-band coefficient fusion:

1) for low frequency subband coefficientThe fusion strategy using averaging:

wherein,andthe original image A, B and the fused image F are each represented by a corresponding low-frequency coefficient at point (x, y).

2) For high frequency sub-band coefficientFusing by adopting a fusion rule based on SSAE and spatial frequency:

2.1) sub-bands of high frequenciesUsing a sliding window technique to divide into num tiles, where s ∈ { a, B }:

2.2) converting all patches into vectorsWherein k is more than 1 and less than num, and the first self-coding network is trained by using the k as a training set:

the training goal of the self-coding network is to obtain a group of optimal weight values and bias values, so that the reconstruction error of the self-coding network is minimized.As input x and output of the first layer auto-encoder:

solving for optimal W using gradient descent algorithm^1，1And b^1，1：

Step 1: setting l to 1;

step 2: set up W^l，1：＝0,b^l，1：＝0,ΔW^l，1：＝0,Δb^l，1：＝0；

And step 3: computing reconstruction errors for an encoder

And 4, step 4: whisleJ (W)^l，1，b^l，1)＞10^-6：

for i＝1to m:

ComputingAnd

updating W^l，1And b^l，1：

Wherein W^l，1And b^l，1For weight matrix and bias, Δ W^l，1And Δ b^l，1Are respectively W^l，1And b^l，1Increment of (A), J (W)^l，1，b^l，1) In order to reconstruct the error,is the Kullback-Leibler (KL) distance,is the average of hidden layer neuronsDegree of activation, ρ is the sparsity coefficient, β is the coefficient of the sparsity penalty term, α is the update rate, and m is the maximum number of iterations.

To obtain W^1，1And b^1，1After the optimal value of (2), forCoding is carried out to obtain a hidden layer activation value a of a first layer coder^s，1(k)：

WhereinW^1，1And b^1，1Respectively, the weight matrix and the offset of the first layer encoder.

Obtaining the hidden layer activation value a of the first layer automatic encoder^s，1Then, it is used as input and output of second layer automatic encoder, and reused to obtain W^1，1And b^1，1The same method can be used to obtain the optimal W^2，1And b^2，1. To obtain W^2，1And b^2，1Then, the hidden layer activation value a of the second layer encoder is obtained^s，2And with a^s，2As a feature of the nubs

2.3) introducing spatial frequencies for enhancing the discrimination of features. In particular, toValue of spatial frequencyAs measure of activity;

wherein,is the value of the feature vector at position (i, j), where 1 < i < M, 1 < j < N, M is the number of rows of the feature vector and N is the number of columns of the feature vector. k is the number of the small blocks, and k is more than 1 and less than num.

2.4) for each pairAndthe rule of taking the big space frequency is utilized to fuse;

2.5) pairs of fusion vectorsObtaining a fusion coefficient matrix using an inverse transform of a sliding window transformAnd adopting an averaging strategy in the overlapping area.

And step 3, obtaining the vector by using SIST inverse transformation.

Experimental conditions and methods:

the hardware platform is as follows: intel (R) processor, CPU master frequency 1.80GHz, memory 1.0 GB;

the software platform is as follows: MATLAB R2016 a; three groups of registered source images, namely a multi-focus image, an infrared-visible light image and a medical image, are adopted in the experiment, wherein the sizes of the images are all 256 multiplied by 256, and the tif format. The multi-focus source image is shown in fig. 6(a) and 6(b), with fig. 6(a) being the left-focused image and fig. 6(b) being the right-focused image. The infrared-visible image is shown in fig. 7(a) and 7(b), fig. 7(a) is a visible image, and fig. 7(b) is an infrared image. The medical images are shown in fig. 8(a) and 8(b), fig. 8(a) is a CT image, and fig. 8(b) is an MRI image.

Simulation experiment:

in order to verify the feasibility and effectiveness of the invention, three image tests of multi-focus images, infrared-visible light images and medical images are adopted, and the fusion result is shown in fig. 6, 7 and 8.

Simulation one: following the technical solution of the present invention, the multi-focus source images (see fig. 6(a) and 6(b)) are fused, as can be seen from the analysis of fig. 6(c) -6 (i): the invention realizes the goal that the dial areas of two small clocks are focused in the multi-focus image fusion experiment, the dial scales and characters are clearest, no extra noise is introduced into the fusion result, and the subjective visual effect is best on the whole.

Simulation II: following the technical solution of the present invention, the infrared-visible light source images (see fig. 7(a) and 7(b)) are fused, as can be seen from the analysis of fig. 7(c) -7 (i): the invention realizes the purpose of storing all infrared targets into the fused image in the experiment of infrared-visible light source image fusion, which is the most important task in the infrared-visible light image fusion, and in addition, the invention can provide the clearest texture detail information at the background of the image (such as branches and leaves of trees).

And (3) simulation: following the technical solution of the present invention, the medical source image (see fig. 8(a) and 8(b)) is fused, as can be seen from the analysis of fig. 8(c) -8 (i): the invention realizes the aim of retaining all skeleton outline and tissue information into the fusion result in the experiment of medical source image fusion, and the fusion result has the highest contrast at the skeleton and the richest and clear information provided at the tissue.

The objective evaluation indexes of the experimental results of the three data sets by using various fusion methods are shown in table 1, table 2 and table 3, wherein the bold data indicate that the corresponding evaluation indexes are optimal values. GP is an image fusion method based on Gaussian gradient pyramid decomposition, DWT is an image fusion method based on discrete wavelet decomposition, strDWT is an image fusion method based on structure tensor and discrete wavelet decomposition, PCNN is an image fusion method based on pulse coupling neural network, N-PCNN is an image fusion method based on non-downsampling contourlet transform and pulse coupling neural network, SR is an image fusion method based on coefficient representation, and Porposed is the fusion method provided by the invention. The information Entropy (EN), the Average Gradient (AG), the edge conversion rate (Qabf), the Edge Intensity (EI), the Mutual Information (MI) and the Standard Deviation (SD) are selected as objective evaluation indexes.

The data in tables 1, 2 and 3 show that the fused image obtained by the method of the invention is superior to other fused methods in objective evaluation indexes such as information entropy, average gradient, mutual information and standard deviation. The information entropy reflects the amount of information carried by the image, and the value of the information entropy indicates that the larger the information amount contained in the fused image is, the better the fusion effect is; the average gradient reflects the definition of an image, and the visual effect is better when the value is larger; the edge conversion rate reflects the degree of transferring the edge information of the image to be fused into the fused image, and the visual effect is better when the value is closer to 1; the edge strength is measured by the richness of the image edge details, and the subjective effect is better when the value is larger; the mutual information reaction is the correlation degree of the information between the image to be fused and the fused image, and the larger the value is, the better the visual effect is; the standard deviation reflects the degree of dispersion of image gray levels from the mean gray level, and the larger the value of the standard deviation, the more dispersed the gray level, and the better the visual effect.

TABLE 1 Objective evaluation index of multi-focus image fusion results

TABLE 2 Objective evaluation index of infrared-visible light image fusion result

TABLE 3 Objective evaluation index of medical image fusion result

As can be seen from the fusion results of the simulation experiments, the fusion image of the invention is globally clear and has rich information. The effectiveness of the invention can be proved in subjective human visual perception and objective evaluation indexes.

Claims (4)

1. The image fusion method based on the translation invariant shear wave and the stack type self-coding is characterized by comprising the following steps of:

1) decomposing two source images to be fused into a low-frequency subband coefficient and a high-frequency subband coefficient by using translation invariant shear wave transformation;

2) fusing the low-frequency sub-band coefficients by a weighting and averaging method;

3) high-frequency subband coefficients containing detailed information are fused by using a fusion rule based on sparse constraint stacked self-encoding (SSAE) features;

4) and (3) performing inverse translation invariant shear wave transformation on the fusion coefficient obtained in the steps 2) and 3) to obtain a fusion image.

2. The method for image fusion based on a translation invariant shear wave transform according to claim 1, wherein said step 1) comprises: decomposing two source images A and B to be fused into low-frequency subband coefficients by using translation invariant shear wave transformationAnd high frequency subband coefficients

3. The new image fusion method based on the translational invariant shear wave transformation according to claim 1, wherein the step 2) comprises the following steps:

$C_L^F(x,y)=\frac{1}{2}\&times;C_L^A(x,y)+\frac{1}{2}\&times;C_L^B(x,y)$

wherein,andthe original image A, B and the fused image F are each shown to have corresponding low-frequency coefficients at point (x, y).

4. The new image fusion method based on the translational invariant shear wave transformation as claimed in claim 1, wherein said step 3) comprises the steps of:

1) firstly, high-frequency sub-bandDividing into num small blocks by using sliding window technology, and converting all small blocks into vectorsWherein s ∈ { A, B }, k is more than 1 and less than num;

2)an SSAE with a two-layer structure was trained as training data:

i. setting l to 1;

set W^l，1：＝0,b^l，1：＝0,ΔW^l，1：＝0,Δb^l，1：＝0；

3) Computing reconstruction errors for an encoder

$J(W^{l,1},b^{l,1})=\frac{1}{N}\underset{i=1}{\overset{N}{\&Sigma;}}\left(\frac{1}{2}\right||W^{l,1}{x}_i+b^{l,1}-x_i|{|}^2)+\mathrm{\&beta;}\underset{j=1}{\overset{D^{,}}{\&Sigma;}}KL\left(\mathrm{\&rho;}\right|\left|\widehat{\mathrm{\&rho;}}\right)$

i.while J(W^l，1，b^l，1)＞10^-6：

for i＝1 to m:

ComputingAnd

${\mathrm{\&Delta;W}}^{l,1}:={\mathrm{\&Delta;W}}^{l,1}+\frac{\&part;J(W^{l,1},b^{l,1})}{\&part;W^{l,1}};$

${\mathrm{\&Delta;b}}^{l,1}:={\mathrm{\&Delta;b}}^{l,1}+\frac{\&part;J(W^{l,1},b^{l,1})}{\&part;b^{l,1}};$

updating W^l，1And b^l，1：

$W^{l,1}=W^{l,1}-\mathrm{\&alpha;}\&lsqb;\left(\frac{1}{m}{\mathrm{\&Delta;W}}^{l,1}\right)+{\mathrm{\&lambda;W}}^{l,1}\&rsqb;;$

$b^{l,1}=b^{l,1}-\mathrm{\&alpha;}\&lsqb;\left(\frac{1}{m}{\mathrm{\&Delta;b}}^{l,1}\right)\&rsqb;;$

Wherein W^l，1And b^l，1For weight matrix and bias, Δ W^l，1And Δ b^l，1Are respectively W^l，1And b^l，1Increment of (A), J (W)^l，1，b^l，1) In order to reconstruct the error,is the Kullback-Leibler (KL) distance,is the average activation of hidden neurons, ρ is the sparse coefficient, β is the coefficient of the sparse penalty term, α is the update rate, and m is the maximum number of iterations.

Obtaining the optimal parameter W of the first layer automatic encoder^1，1And b^1，1Then, the hidden layer neuron activation value a of the first layer automatic encoder is calculated^s，1：

a^s，1(k)＝sigmoid(W^1，1x(k)+b^1，1)

Wherein

4) A is to^s，1As input and output of the second layer automatic encoder, and then using the same to obtain W^1，1And b^1，1The same method can be used to obtain the optimal W^2，1、b^2，1And a^s，2。

5) Implicit layer activation value a using second layer autoencoder^s，2Features as input tiles

6) Then using spatial frequency enhancement featuresThen by the value of the spatial frequencyAs measure of activity:

${\mathrm{SF}}_H^s\left(k\right)=\sqrt{\frac{1}{M*N}(\underset{i=1}{\overset{M}{\&Sigma;}}\underset{j=2}{\overset{N}{\&Sigma;}}{\&lsqb;F_H^s\left(k\right)\left(i,j\right)-F_H^s\left(k\right)\left(i,j-1\right)\&rsqb;}^2+\underset{i=2}{\overset{M}{\&Sigma;}}\underset{j=1}{\overset{N}{\&Sigma;}}{\&lsqb;F_H^s\left(k\right)\left(i,j\right)-F_H^s\left(k\right)\left(i-1,j\right)\&rsqb;}^2)}$

wherein,is the value of the feature vector at position (i, j), where 1 < i < M, 1 < j < N, M is the number of rows of the feature vector and N is the number of columns of the feature vector. k is the number of the small blocks, and k is more than 1 and less than num.

7) For each pairAndand (3) fusing rules by taking large spatial frequency:

$V_H^F\left(k\right)=\left\{\begin{array}{cc}{V}_H^A\left(k\right),& if\left({\mathrm{SF}}_H^A\right(k)\&GreaterEqual;{\mathrm{SF}}_H^B(k\left)\right)\\ V_H^B\left(k\right),& if\left({\mathrm{SF}}_H^A\right(k)<{\mathrm{SF}}_H^B(k\left)\right)\end{array}\right.$

the high frequency subbands are then regenerated using the inverse of the sliding window transform, where the overlapping portions use an averaging strategy.