CN106897987A - Image interfusion method based on translation invariant shearing wave and stack own coding - Google Patents

Abstract

The invention discloses a kind of image interfusion method based on translation invariant shearing wave and stack own coding.Implementation step is：Wave conversion is sheared by picture breakdown to be fused into low frequency sub-band coefficient and high-frequency sub-band coefficient with translation invariant first；Secondly, low frequency sub-band coefficient reflects the base profile of image, and the method being averaged using weighting is merged；High-frequency sub-band coefficient reflects the edge and texture information of image, the present invention proposes a kind of fusion method based on stack own coding feature, using sliding the method for piecemeal by high-frequency sub-band piecemeal, stack autoencoder network is trained as input using fritter, again fritter encode using the network for training obtaining feature, and utilization space frequency carries out feature enhancing and obtains activity and estimate, finally estimating numerical value using this activity and taking big fusion rule carries out the fusion of high-frequency sub-band coefficient fritter, and high-frequency sub-band is obtained using sliding window inverse transformation after all fritters fusions；Image after finally being merged using translation invariant shearing wave inverse transformation.The present invention can preferably retain edge and texture information in original image compared to traditional fusion method.

Description

Image Fusion Method Based on Translation Invariant Shearlet and Stacked Autoencoder

technical field

The invention relates to an image fusion method based on translation invariant shear wave and stacked self-encoding, which is a fusion method in the technical field of image processing and is widely used in military applications and clinical medical diagnosis.

Background technique

Due to the limited information contained in a single image, it often cannot meet the practical application. Image fusion is a technology that synthesizes images of the same scene collected by multiple sensors into one image through fusion algorithm processing. The fused image can effectively combine the advantages of multiple images to be fused, which is more suitable for human visual perception. Image fusion was proposed in the 1970s. In recent years, due to the rapid development of multi-sensor technology, image fusion has been widely used in the fields of military reconnaissance, medical diagnosis and remote sensing.

Image fusion methods can be roughly divided into image fusion in the spatial domain and image fusion in the transform domain. Image fusion in the air domain refers to the method of directly fusing the pixels of the image, mainly including weighted image fusion and image fusion based on principal component analysis. This type of fusion method has no complicated transformation and generally has high real-time performance, but They have a common shortcoming, that is, there is distortion in the fused image. Image fusion in the transform domain generally refers to the method of multi-scale decomposition of the image, and then the fusion of each sub-band coefficient. The multi-scale decomposition tools include wavelet transform (DWT), stationary wavelet transformation (SWT), contourlet Transform (CT), Non-Subsampled Contourlet Transform (NSCT), Shearlet Transform (ST), Translation Invariant Shearlet Transform (SIST), etc. DWT was proposed by Mallat in 1989. It has the advantages of time-frequency localization and multi-resolution, but it can capture limited direction information and has Gibbs oscillation phenomenon. Rockinger proposed that SWT can suppress the Gibbs oscillation phenomenon of DWT, but the high-frequency sub-band of SWT still only has three directions: horizontal, vertical and diagonal. The CT proposed by Minh N.Do has a strong direction selection ability, but CT lacks translation invariance because of the downsampling operation in the transformation process. The NSCT proposed by Cunha uses a non-subsampling filter with translation invariance. Guo et al. proposed ST. Compared with CT, ST is more efficient in calculation and has no limitation on the number of directions. However, ST does not have translation invariance due to the use of downsampling filters. The SIST proposed by Easley et al. uses a non-subsampling pyramid decomposition, which has the characteristics of translation invariance, which makes it have a better development prospect in the field of image fusion.

The design of fusion rules determines the quality of image fusion. The main step of image fusion based on transform domain is to design a suitable scheme to fuse the sub-band coefficients of each layer, so as to obtain a fusion image that is more in line with human visual perception. This step involves two basic issues: selection of fusion strategy and selection of activity measure.

Commonly used subband coefficient fusion rules include taking the largest absolute value and weighting the average. Taking a large absolute value refers to selecting the subband coefficient with a large activity measure as the fused subband coefficient. The weighted averaging rule calculates the weight of the corresponding sub-band coefficients according to the activity measure and then fuses the sub-band coefficients weightedly. The weighted average method can better retain the useful information of the image to be fused, and the fusion rule with a large absolute value can improve the contrast of the fused image.

The design of the activity measure can also be understood as the feature selection of the pixel or sub-band coefficients, and it is required to accurately reflect the characteristics of the data from a certain angle. In the fusion method based on principal component analysis (PCA), He et al. first obtained the principal components from the eigenvalues and eigenvectors of the correlation coefficient matrix between the low-resolution images of the bands, and then stretched the gray scale of the high-resolution images of the bands And make it have the same mean and variance as the principal component, and finally use this high-resolution image to replace the principal component and use PCA inverse transformation to obtain the fused image. Tensor has outstanding performance in describing high-dimensional data, and it also has a good effect in the application of image fusion. Liang et al. first decomposed the image with a high-order singular value decomposition tool, and then used the 1-norm of the coefficients as an activity measure to construct a fusion rule. Sparse representation can describe complex data with a linear combination of dictionary elements, and has been widely used in image fusion. Yang et al. first divided the image into overlapping small blocks, then decomposed the image small blocks to obtain sparse coefficients, and finally designed fusion rules for the sparse coefficients. In recent years, the application of deep learning methods in the field of computer vision has achieved great success. Deep learning methods are good at discovering hierarchical features from complex data sets, but due to the lack of sufficient labeled features in the practical application of image fusion Supervised learning methods such as convolutional neural network (CNN) have not yet been applied in image fusion, while stacked autoencoder (SAE), as a deep neural network for unsupervised learning, meets the requirements of image fusion applications. scene. The SAE network obtained by stacking multi-layer self-encoders (AE) can not only explore the hierarchical features of the image, but also make the extracted features sparse if the sparse constraints (SSAE) are set on the hidden layer neurons. , which is more in line with the needs of image fusion applications.

Contents of the invention

The purpose of the present invention is to address the above-mentioned deficiencies in the prior art, and proposes an image fusion algorithm based on translation-invariant shearlets and stacked autoencoder networks, to protect image details, enhance image contrast and contour edges, and improve its Visual effects that improve the quality of image fusion. Concrete technical scheme of the present invention is as follows:

1) First, perform SIST transformation on the two images to be fused, and decompose to obtain the low-frequency subband and high frequency subband

2) Different fusion rules are designed for low-frequency sub-band coefficients and high-frequency sub-band coefficients:

2.1) The low-frequency sub-band coefficients contain the basic information of the image, so with Use averaging strategy fusion:

The low-frequency sub-band coefficients obtained by SIST decomposition of the image contain the main energy information of the image, which is an approximate component of the original image. This paper uses the average method to fuse the low-frequency sub-band coefficients, namely

in, with Respectively represent the corresponding low-frequency coefficients of the original images A, B and the fused image F at point (x, y).

2.2) High frequency subband coefficients contains details, with Use the fusion rule based on SSAE feature selection: the high-frequency sub-band coefficients contain the texture information of the image. As a deep learning method, SSAE is very good at learning high-dimensional structural features from complex data sets. Considering these factors, the present invention proposes a fusion rule based on SSAE features, first divide the sub-bands to be fused, and convert the small blocks into vectors Then train a two-layer SSAE network as training data, and then use this network to encode the fusion small blocks to obtain features next to Find the spatial frequency use Take large rules to fuse small blocks, where s ∈ {A, B};

Finally for all fusion vectors First convert to the form of matrix, and then use the inverse transformation of sliding window transformation to obtain the fusion coefficient matrix A strategy of averaging is used in overlapping areas.

3) Use the SIST inverse transformation to obtain the fused image.

Compared with the existing medical image fusion method, the present invention has the following advantages:

1. The present invention uses translation-invariant shearlet transform (SIST) as a multi-scale decomposition tool. Compared with wavelet transform (DWT), SIST can capture more direction information and eliminate pseudo-Gibbs phenomenon; Compared with the stationary wavelet transform (SWT), SIST obtains more high-frequency subband directions; compared with the lack of translation invariance in the contourlet transform (CT), SIST does not have downsampling operations, so it has translation invariance ; Compared with the non-subsampled contourlet transform (NSCT), SIST has no limitation on the number of directions, and the calculation efficiency is higher.

2. The present invention adopts a stacked encoder (SSAE) as a feature extraction tool. Compared with Principal Component Analysis (PCA), SSAE is a data-driven feature extraction tool that can learn unique rules from an input image, thereby extracting Features that are more representative than artificially designed features; compared to tensor and pulse-coupled neural network (PCNN), SSAE, as a deep network, can extract hierarchical features, which is more suitable for image data representation than tensor and PCNN .

3. The present invention introduces spatial frequency when constructing the activity measurement. Compared with directly using SSAE features, the local contrast of the activity measurement after introducing the spatial frequency is stronger, which can better represent the characteristics of the original image; it can protect the image to a greater extent edge and texture information.

Description of drawings:

Fig. 1 is an overall fusion framework diagram of the present invention.

Figure 2 is a structural diagram of the translation invariant shearlet transform.

Figure 3 is a block diagram of a single-layer sparse autoencoder.

Figure 4 is a structural diagram of a stacked autoencoder.

Fig. 5 is a schematic diagram of a high-frequency coefficient sub-band fusion rule.

Fig. 6 (a) and (b) are multi-focus images to be fused in the first embodiment of the present invention; (c) is a fusion image based on GP; (d) is a fusion image based on DWT; (e) is a fusion image based on stDWT (f) is a fusion image based on PCNN; (g) is a fusion image based on N-PCNN; (h) is a fusion image based on SR; (i) is a fusion image of the inventive method.

Figure 7 (a) is a visible light image to be fused according to an embodiment of the present invention; (b) is an infrared image to be fused according to an embodiment of the present invention; (c) is a fusion image based on GP; (d) is a fusion based on DWT Image; (e) is a fusion image based on stDWT; (f) is a fusion image based on PCNN; (g) is a fusion image based on N-PCNN; (h) is a fusion image based on SR; (i) is the fusion image of the present invention Method for fused images.

Figure 8 (a) is a CT image to be fused according to an embodiment of the present invention; (b) is an MRI image to be fused according to an embodiment of the present invention; (c) is a fusion image based on GP; (d) is a fusion based on DWT Image; (e) is a fusion image based on stDWT; (f) is a fusion image based on PCNN; (g) is a fusion image based on N-PCNN; (h) is a fusion image based on SR; (i) is the fusion image of the present invention Method for fused images.

detailed description:

Embodiments of the present invention are described in detail below in conjunction with the accompanying drawings. The present embodiment is carried out under the premise of the technical solution of the present invention, as shown in Figure 1, the detailed implementation and concrete operation steps are as follows:

Step 1, use SIST transformation to decompose the two multi-focus images to be fused, and obtain low-frequency subbands after image decomposition and high frequency subband Among them, "maxflat" is selected for the scale decomposition LP, "pmaxflat" is selected for the direction filter bank, and the direction decomposition parameter is set to [3,4,5];

Step 2, low-frequency sub-band coefficient fusion and high-frequency sub-band coefficient fusion:

1) For low frequency subband coefficients Using the average fusion strategy:

in, with Respectively represent the corresponding low-frequency coefficients of the original images A, B and the fused image F at point (x, y).

2) For high frequency subband coefficients Fusion is performed using fusion rules based on SSAE and spatial frequency:

2.1) The high frequency sub-band Divide into num small blocks using sliding window technology, where s∈{A, B}:

2.2) Convert all small blocks into vectors where 1<k<num, and use them as the training set to train the first autoencoder network:

The goal of autoencoder network training is to obtain a set of optimal weights and bias values to minimize the reconstruction error of the autoencoder network. As the input x and output of the first layer autoencoder:

Use the gradient descent algorithm to find the optimal W ^1,1 and b ^1,1 :

Step 1: set l=1;

Step 2: set W ^l,1 :=0,b ^l,1 :=0,ΔW ^l,1 :=0,Δb ^l,1 :=0;

Step 3: Calculate the reconstruction error of the encoder

Step 4: whileJ(W ^{l, 1} , b ^{l, 1} )>10 ^-6 :

for i＝1to m:

calculate with

Update W ^l,1 and b ^l,1 :

Where W ^l,1 and b ^l,1 are the weight matrix and bias, ΔW ^l,1 and Δb ^l,1 are the increments of W ^l,1 and b ^l,1 respectively, J(W ^l,1 ,b ^{l, 1} ) is the reconstruction error, is the Kullback-Leibler (KL) distance, is the average activation degree of neurons in the hidden layer, ρ is the sparsity coefficient, β is the coefficient of the sparsity penalty term, α is the update rate, and m is the maximum number of iterations.

After obtaining the optimal values of W ^1,1 and b ^1,1 , for Encoding is performed to obtain the hidden layer activation value a ^{s, 1} (k) of the first layer encoder:

in W ^1,1 and b ^1,1 are the weight matrix and bias of the first-layer encoder, respectively.

After obtaining the hidden layer activation value a ^s,1 of the first layer of autoencoder, use it as the input and output of the second layer of autoencoder, and then use the same method to find W ^1,1 and b ^1,1 to solve Optimal W ^2,1 and b ^2,1 . After obtaining W ^2,1 and b ^2,1 , obtain the hidden layer activation value a ^s,2 of the second layer encoder, and use a ^s,2 as the feature of the small block

2.3) In order to enhance the discrimination of features, spatial frequency is introduced. Specifically, with The value of the spatial frequency as an activity measure;

in, is the value of the feature vector at position (i, j), where 1<i<M, 1<j<N, M is the number of rows of the feature vector, and N is the number of columns of the feature vector. k is the serial number of the small block, 1<k<num.

2.4) For each pair with Use the rule fusion with the largest spatial frequency;

2.5) For fusion vector Using the inverse transformation of the sliding window transformation to obtain the fusion coefficient matrix A strategy of averaging is used in overlapping areas.

Step 3, obtained by using SIST inverse transformation.

Experimental conditions and methods:

The hardware platform is: Intel(R) processor, CPU main frequency 1.80GHz, memory 1.0GB;

The software platform is: MATLAB R2016a; three sets of registered source images are used in the experiment, namely multi-focus images, infrared-visible light images and medical images, and the image size is 256×256 in tif format. Multi-focus source images are shown in Figure 6(a) and Figure 6(b), Figure 6(a) is the left-focused image, and Figure 6(b) is the right-focused image. Infrared-visible light images are shown in Figure 7(a) and Figure 7(b), Figure 7(a) is a visible light image, and Figure 7(b) is an infrared image. See Figure 8(a) and Figure 8(b) for medical images, Figure 8(a) is a CT image, and Figure 8(b) is an MRI image.

Simulation:

In order to verify the feasibility and effectiveness of the present invention, three sets of image tests including multi-focus images, infrared-visible light images and medical images were used, and the fusion results are shown in Fig. 6, Fig. 7 and Fig. 8 .

Simulation 1: Following the technical solution of the present invention, the multi-focus source images (see Figure 6(a) and Figure 6(b)) are fused, as can be seen from the analysis of Figure 6(c)-Figure 6(i): In the experiment of multi-focus image fusion, the present invention achieves the goal of focusing the dial areas of the two small clocks, and the dial scale and text are the clearest. The fusion result does not introduce additional noise, and the overall subjective visual effect is the best.

Simulation 2: Following the technical solution of the present invention, the infrared-visible light source images (see Figure 7(a) and Figure 7(b)) are fused, as can be seen from the analysis of Figure 7(c)-Figure 7(i) : the present invention has realized in the experiment of infrared-visible light source image fusion that all infrared targets have been preserved to the target of fusion image, and this is the most important task in infrared-visible light image fusion. branches and leaves), the present invention can provide the clearest texture detail information.

Simulation three: following the technical solution of the present invention, the medical source images (see Fig. 8(a) and Fig. 8(b)) are fused, and it can be seen from the analysis of Fig. 8(c)-Fig. 8(i): The invention achieves the goal of retaining all bone contours and tissue information in the fusion result in the experiment of medical source image fusion. On the whole, the fusion result of the present invention has the highest contrast at the bone and the most information at the tissue. Rich and clear.

Table 1, Table 2 and Table 3 give the objective evaluation indicators of the experimental results of the three data sets using various fusion methods, where the bold data indicates that the corresponding evaluation indicators are the optimal values. GP is an image fusion method based on Gaussian gradient pyramid decomposition, DWT is an image fusion method based on discrete wavelet decomposition, strDWT is an image fusion method based on structural tensor and discrete wavelet decomposition, PCNN is an image fusion method based on pulse-coupled neural network, N-PCNN is an image fusion method based on non-subsampling contourlet transform and pulse-coupled neural network, SR is an image fusion method based on coefficient representation, and Porposed is a fusion method proposed by the present invention. The experiment uses information entropy (EN), average gradient (AG), edge transition rate (Qabf), edge intensity (EI), mutual information (MI), and standard deviation (SD) as objective evaluation indicators.

The data in Table 1, Table 2 and Table 3 show that the fusion image obtained by the method of the present invention is superior to other fusion methods in terms of objective evaluation indicators such as information entropy, average gradient, mutual information and standard deviation. Information entropy reflects the amount of information carried by the image, and its value indicates that the greater the amount of information contained in the fused image, the better the fusion effect; the average gradient reflects the clarity of the image, and the larger the value, the better the visual effect; The conversion rate reflects the degree to which the edge information of the image to be fused is transferred to the fused image. The closer the value is to 1, the better the visual effect; the edge strength measures the richness of the edge details of the image. The larger the value, the better the subjective effect ; Mutual information reflects the degree of correlation between the image to be fused and the information between the fused image, the larger the value, the better the visual effect; the standard deviation reflects the degree of dispersion of the image grayscale compared to the gray average value, the larger the value The more dispersed the gray level, the better the visual effect.

Table 1 Objective evaluation index of multi-focus image fusion results

Table 2 Objective evaluation index of infrared-visible light image fusion results

Table 3 Objective evaluation indicators of medical image fusion results

From the fusion results of various simulation experiments, it can be seen that the fusion image of the present invention is globally clear, and the fusion image information is rich. The effectiveness of the present invention can be proved no matter from subjective human visual perception or objective evaluation index.

Claims (4)

1. The image fusion method based on translation-invariant shearlet and stacked self-encoding, is characterized in that, comprises the steps: 1) The two source images to be fused are decomposed into low-frequency sub-band coefficients and high-frequency sub-band coefficients using translation invariant shearlet transform; 2) The low-frequency sub-band coefficients are fused by means of weighted averaging; 3) The high-frequency sub-band coefficients containing detailed information are fused using a fusion rule based on sparse constrained stacked autoencoder (SSAE) feature extraction; 4) Perform translation-invariant shearlet inverse transformation on the fusion coefficients obtained in steps 2) and 3) to obtain a fusion image. 2. the image fusion method based on translation invariant shearlet transform according to claim 1, is characterized in that, described step 1) comprises: two pieces of source images A and B to be fused, utilize translation invariant shearlet transform The wavelet transform decomposes the two images into low-frequency subband coefficients respectively and high frequency subband coefficients 3. the new method of image fusion based on translation-invariant shearlet transform according to claim 1, is characterized in that, described step 2) comprises the steps: ${\mathrm{<span\; class="notranslate">\; C</span>}}_{\mathrm{<span\; class="notranslate">\; L</span>}}^{\mathrm{<span\; class="notranslate">\; f</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\frac{\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; \&times;</span>{\mathrm{<span\; class="notranslate">\; C</span>}}_{\mathrm{<span\; class="notranslate">\; L</span>}}^{\mathrm{<span\; class="notranslate">\; A</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; +</span>\frac{\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; \&times;</span>{\mathrm{<span\; class="notranslate">\; C</span>}}_{\mathrm{<span\; class="notranslate">\; L</span>}}^{\mathrm{<span\; class="notranslate">\; B</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; )</span>$ in, with Respectively represent the corresponding low-frequency coefficients of the original images A, B and the fused image F at point (x, y). 4. the new method of image fusion based on translation invariant shearlet transform according to claim 1, is characterized in that, described step 3) comprises the steps: 1) First, the high-frequency sub-band Divide into num small blocks using sliding window technology, and convert all small blocks into vectors where s∈{A,B}, 1<k<num; 2) As the training data train a SSAE with a two-layer structure: i. Set l=1; ii. Set W ^l,1 :=0,b ^l,1 :=0,ΔW ^l,1 :=0,Δb ^l,1 :=0; 3) Calculate the reconstruction error of the encoder $\mathrm{<span\; class="notranslate">\; J</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; W</span>}}^{\mathrm{<span\; class="notranslate">\; l</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; b</span>}}^{\mathrm{<span\; class="notranslate">\; l</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\frac{\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; N</span>}}\underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; N</span>}}{<span\; class="notranslate">\; \&Sigma;</span>}}<span\; class="notranslate">\; (</span>\frac{\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; W</span>}}^{\mathrm{<span\; class="notranslate">\; l</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; b</span>}}^{\mathrm{<span\; class="notranslate">\; l</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; |</span>{<span\; class="notranslate">\; |</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; \&beta;</span>}\underset{\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{{\mathrm{<span\; class="notranslate">\; D.</span>}}^{<span\; class="notranslate">\; ,</span>}}{<span\; class="notranslate">\; \&Sigma;</span>}}\mathrm{<span\; class="notranslate">\; K</span>}\mathrm{<span\; class="notranslate">\; L</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; \&rho;</span>}<span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>\stackrel{<span\; class="notranslate">\; ^</span>}{\mathrm{<span\; class="notranslate">\; \&rho;</span>}}<span\; class="notranslate">\; )</span>$ i.while J(W ^{l, 1} , b ^{l, 1} )>10 ⁻⁶ : for i=1 to m: calculate with ${\mathrm{<span\; class="notranslate">\; \&Delta;W</span>}}^{\mathrm{<span\; class="notranslate">\; l</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; :</span><span\; class="notranslate">\; =</span>{\mathrm{<span\; class="notranslate">\; \&Delta;W</span>}}^{\mathrm{<span\; class="notranslate">\; l</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; +</span>\frac{<span\; class="notranslate">\; \&part;</span>\mathrm{<span\; class="notranslate">\; J</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; W</span>}}^{\mathrm{<span\; class="notranslate">\; l</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; b</span>}}^{\mathrm{<span\; class="notranslate">\; l</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; )</span>}{<span\; class="notranslate">\; \&part;</span>{\mathrm{<span\; class="notranslate">\; W</span>}}^{\mathrm{<span\; class="notranslate">\; l</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; 1</span>}}}<span\; class="notranslate">\; ;</span>$ ${\mathrm{<span\; class="notranslate">\; \&Delta;b</span>}}^{\mathrm{<span\; class="notranslate">\; l</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; :</span><span\; class="notranslate">\; =</span>{\mathrm{<span\; class="notranslate">\; \&Delta;b</span>}}^{\mathrm{<span\; class="notranslate">\; l</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; +</span>\frac{<span\; class="notranslate">\; \&part;</span>\mathrm{<span\; class="notranslate">\; J</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; W</span>}}^{\mathrm{<span\; class="notranslate">\; l</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; b</span>}}^{\mathrm{<span\; class="notranslate">\; l</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; )</span>}{<span\; class="notranslate">\; \&part;</span>{\mathrm{<span\; class="notranslate">\; b</span>}}^{\mathrm{<span\; class="notranslate">\; l</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; 1</span>}}}<span\; class="notranslate">\; ;</span>$ Update W ^l,1 and b ^l,1 : ${\mathrm{<span\; class="notranslate">\; W</span>}}^{\mathrm{<span\; class="notranslate">\; l</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; =</span>{\mathrm{<span\; class="notranslate">\; W</span>}}^{\mathrm{<span\; class="notranslate">\; l</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; \&alpha;</span>}<span\; class="notranslate">\; \&lsqb;</span><span\; class="notranslate">\; (</span>\frac{\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; m</span>}}{\mathrm{<span\; class="notranslate">\; \&Delta;W</span>}}^{\mathrm{<span\; class="notranslate">\; l</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; \&lambda;W</span>}}^{\mathrm{<span\; class="notranslate">\; l</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; \&rsqb;</span><span\; class="notranslate">\; ;</span>$ ${\mathrm{<span\; class="notranslate">\; b</span>}}^{\mathrm{<span\; class="notranslate">\; l</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; =</span>{\mathrm{<span\; class="notranslate">\; b</span>}}^{\mathrm{<span\; class="notranslate">\; l</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; \&alpha;</span>}<span\; class="notranslate">\; \&lsqb;</span><span\; class="notranslate">\; (</span>\frac{\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; m</span>}}{\mathrm{<span\; class="notranslate">\; \&Delta;b</span>}}^{\mathrm{<span\; class="notranslate">\; l</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; \&rsqb;</span><span\; class="notranslate">\; ;</span>$ Where W ^l,1 and b ^l,1 are the weight matrix and bias, ΔW ^l,1 and Δb ^l,1 are the increments of W ^l,1 and b ^l,1 respectively, J(W ^l,1 ,b ^{l, 1} ) is the reconstruction error, is the Kullback-Leibler (KL) distance, is the average activation degree of neurons in the hidden layer, ρ is the sparsity coefficient, β is the coefficient of the sparsity penalty term, α is the update rate, and m is the maximum number of iterations. After obtaining the optimal parameters W ^1,1 and b ^1,1 of the first layer of autoencoder, calculate the hidden layer neuron activation value a ^s,1 of the first layer of autoencoder: a ^s,1 (k)=sigmoid(W ^1,1 x(k)+b ^1,1 ) in 4) Take a ^{s, 1} as the input and output of the second-layer autoencoder, and then use the same method to find W ¹ , 1 and b ¹ , 1 to solve the optimal W ^2,1 , b 2, ¹ and a ^{s, 2} . 5) Use the hidden layer activation value a ^{s, 2} of the second layer autoencoder as the feature of the input small block 6) Then use the spatial frequency to enhance the feature The degree of discrimination, and then the value of the spatial frequency As an activity measure: ${\mathrm{<span\; class="notranslate">\; SF</span>}}_{\mathrm{<span\; class="notranslate">\; h</span>}}^{\mathrm{<span\; class="notranslate">\; the\; s</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\sqrt{\frac{\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; m</span>}<span\; class="notranslate">\; *</span>\mathrm{<span\; class="notranslate">\; N</span>}}<span\; class="notranslate">\; (</span>\underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; m</span>}}{<span\; class="notranslate">\; \&Sigma;</span>}}\underset{\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 2</span>}}{\overset{\mathrm{<span\; class="notranslate">\; N</span>}}{<span\; class="notranslate">\; \&Sigma;</span>}}{<span\; class="notranslate">\; \&lsqb;</span>{\mathrm{<span\; class="notranslate">\; f</span>}}_{\mathrm{<span\; class="notranslate">\; h</span>}}^{\mathrm{<span\; class="notranslate">\; the\; s</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; f</span>}}_{\mathrm{<span\; class="notranslate">\; h</span>}}^{\mathrm{<span\; class="notranslate">\; the\; s</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; \&rsqb;</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; +</span>\underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 2</span>}}{\overset{\mathrm{<span\; class="notranslate">\; m</span>}}{<span\; class="notranslate">\; \&Sigma;</span>}}\underset{\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; N</span>}}{<span\; class="notranslate">\; \&Sigma;</span>}}{<span\; class="notranslate">\; \&lsqb;</span>{\mathrm{<span\; class="notranslate">\; f</span>}}_{\mathrm{<span\; class="notranslate">\; h</span>}}^{\mathrm{<span\; class="notranslate">\; the\; s</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; f</span>}}_{\mathrm{<span\; class="notranslate">\; h</span>}}^{\mathrm{<span\; class="notranslate">\; the\; s</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; \&rsqb;</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; )</span>}$ in, is the value of the feature vector at position (i, j), where 1<i<M, 1<j<N, M is the number of rows of the feature vector, and N is the number of columns of the feature vector. k is the serial number of the small block, 1<k<num. 7) For each pair with Use the rule fusion with the largest spatial frequency: ${\mathrm{<span\; class="notranslate">\; V</span>}}_{\mathrm{<span\; class="notranslate">\; h</span>}}^{\mathrm{<span\; class="notranslate">\; f</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\left\{\begin{array}{cc}{\mathrm{<span\; class="notranslate">\; V</span>}}_{\mathrm{<span\; class="notranslate">\; h</span>}}^{\mathrm{<span\; class="notranslate">\; A</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; ,</span>& \mathrm{<span\; class="notranslate">\; i</span>}\mathrm{<span\; class="notranslate">\; f</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; SF</span>}}_{\mathrm{<span\; class="notranslate">\; h</span>}}^{\mathrm{<span\; class="notranslate">\; A</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; \&Greater\; Equal;</span>{\mathrm{<span\; class="notranslate">\; SF</span>}}_{\mathrm{<span\; class="notranslate">\; h</span>}}^{\mathrm{<span\; class="notranslate">\; B</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; )</span>\\ {\mathrm{<span\; class="notranslate">\; V</span>}}_{\mathrm{<span\; class="notranslate">\; h</span>}}^{\mathrm{<span\; class="notranslate">\; B</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; ,</span>& \mathrm{<span\; class="notranslate">\; i</span>}\mathrm{<span\; class="notranslate">\; f</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; SF</span>}}_{\mathrm{<span\; class="notranslate">\; h</span>}}^{\mathrm{<span\; class="notranslate">\; A</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; <</span>{\mathrm{<span\; class="notranslate">\; SF</span>}}_{\mathrm{<span\; class="notranslate">\; h</span>}}^{\mathrm{<span\; class="notranslate">\; B</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; )</span>\end{array}\right.$ Then, the inverse transform of the sliding window transform is used to regenerate the high-frequency subbands, and the overlapping part uses an averaging strategy.