CN113947554B - A multi-focus image fusion method based on NSST and salient information extraction - Google Patents

Abstract

The invention proposes a multi-focus image fusion method based on NSST and salient information extraction. It mainly involves the problem of multi-focus image fusion in the field of image fusion. First, the source image is decomposed into high-frequency and low-frequency sub-bands through multi-scale and multi-directional decomposition through NSST transformation. Secondly, the improved Laplacian energy of the local area is used for the coefficients of the low-frequency subbands and the initial fusion weights of the low-frequency subbands are constructed. The coefficient adopts the fusion rule based on the combination of spatial frequency and energy based on the correlation coefficient, and then corrects it with the phase consistency fusion rule to construct the high-frequency sub-band fusion weight; finally, the final fusion image is obtained through NSST inverse transformation. Due to the corresponding weight correction strategies for the low-frequency and high-frequency sub-bands, the discrimination error rate of the focused area is reduced. The effectiveness of the method is verified in experiments with multiple sets of different focused images.

Description

A multi-focus image fusion method based on NSST and salient information extraction

technical field

The invention relates to the multi-focus image fusion problem in the image fusion field, in particular to a multi-focus image fusion method based on NSST and salient information extraction.

Background technique

Image fusion technology is a research hotspot in the field of image processing. The information contained in an image (brightness, color, space, etc.) is limited. Therefore, it is difficult to meet specific application scenarios only through an image with limited information. By merging multiple images with different emphases of information together with certain rules, an image that contains more information and can be observed more easily is obtained. Obviously, the goal of image fusion is to retain useful information as much as possible while removing certain redundant information. Multi-focus image fusion integrates different focused images in the same scene through a certain fusion method, so that the definition of the fused image is higher and the information contained is more abundant.

Among the pixel-level fusion methods that are widely used in the field of image fusion, they include fusion methods based on spatial domain, fusion methods based on transform domain, and fusion methods based on deep learning. Among them, the fusion method based on the transform domain is more widely used than the other two methods. The so-called transformation domain is to transform the original data information of the image through a certain reversible mathematical transformation to obtain intermediate data with different characteristic information. By processing these intermediate data with corresponding fusion rules, the fused image is obtained after reversible transformation. Obviously, in this type of method, reversible transformation and fusion rules become the key. The successive reversible transforms include pyramid transform, wavelet transform, non-subsampled contourlet transform (NSCT), non-subsampled shearlet transform (NSST) and so on. There are relatively many fusion rules adopted, including fusion rules based on spatial frequency, fusion rules based on energy information, and fusion rules based on guided filtering. Scholars have proposed many fusion algorithms from different perspectives. The fused images obtained by most fusion algorithms have phenomena such as low definition, loss of focus information, and blurred focus edges.

Contents of the invention

The invention proposes a multi-focus image fusion method based on NSST and salient information extraction. The high and low frequency subband coefficients obtained by performing NSST transformation on different focused images are processed by using different fusion rules and certain correction rules, and finally a fused image is obtained. The present invention mainly realizes above-mentioned purpose through following process step:

(1) Use NSST transform to process different focused images to obtain high and low frequency subband coefficients;

(2) Preliminarily process the low-frequency sub-band coefficients obtained in (1) using an improved Laplacian energy sum (SML) initial low-frequency fusion rule to obtain an initial low-frequency fusion weight;

(3) Use the low-frequency correction fusion rule of salient information extraction to perform certain error correction on the results in (2);

(4) Preliminarily process the high-frequency subband coefficients obtained in (1) using the initial high-frequency fusion rule based on the correlation coefficient to obtain the initial high-frequency fusion weight;

(5) A series of high-frequency fusion weights are supplemented with different degrees of phase consistency (PC) correction rules to perform discriminant correction;

(6) Perform NSST inverse transformation on the processing results obtained in (3) and (5) to obtain the fusion result.

Description of drawings

Figure 1. Framework of multi-focus image fusion based on NSST and salient information extraction;

Detailed ways

The present invention introduces non-local mean filter (NLMF) combined with guided filter (GF) to carry out weighted correction on low-frequency sub-band coefficients, combines spatial frequency and energy based on correlation coefficient to form initial high-frequency weighted fusion rule, and uses phase consistency correction at the same time The strategy modifies the initial high-frequency fusion weights.

The non-local mean filter correction fusion rules are as follows:

Combined with attached drawing 2, the low-frequency sub-band correction rule block diagram;

After different focused images are transformed by NSST, the obtained low-frequency sub-band images lose detailed information, and the initial fusion weights obtained by using the initial fusion rules based on the improved sum of Laplacian energy (SML) have discriminative errors. The present invention considers the source image The detailed information of the source image is filtered by non-local mean filter, and then the source image and the filtered image are differentially calculated to obtain the saliency information D _l ,

D _l ＝|I _l -I _l ×NMLF| (1)

(1) In the formula: I _l (0<l<L) is the source image, L represents the number of source images, and × represents the filtering operation.

Then use guided filtering to get the detailed information of the focus area,

G _l ＝guidedfilter(I _l ,D _l ,r,eps) (2)

对初始融合权重进行差错修正。Finally, the large strategy is used to obtain the low-frequency sub-band correction fusion weight

Error correction is performed on the initial fusion weights.

The initial high-frequency weighted fusion rules are as follows:

The high-frequency sub-band image contains detailed information of the source image, and the present invention combines spatial frequency and energy utilization correlation coefficient to form an initial high-frequency fusion rule.

First, define the operation of the correlation coefficient (Corr):

分别表示高频子带图像以及对高频子带图像进行均值滤波后的图像，/>

分别是/>

的均值，M*N为所取的图像块大小。In the formula:

respectively represent the high-frequency sub-band image and the image after mean filtering of the high-frequency sub-band image, />

respectively />

The mean value of , M*N is the size of the image block taken.

Then define the calculation formula of spatial frequency and energy (SF_Eng_Corr) based on the correlation coefficient as follows:

In the formula: SF_Corr and Eng_Corr represent the spatial frequency correlation coefficient and energy correlation coefficient, respectively. For the focus area and non-focus area of an image, the SF_Corr value and Eng_Corr value of the focus area class are often greater than the non-focus area. Taking advantage of this, combine the weights of the two to form SF_Eng_Corr, and then use the large strategy to obtain the initial high-frequency fusion weight.

The phase consistency correction strategy is as follows:

The initial high-frequency fusion weight obtained from the correlation coefficient ignores the correlation of the high-frequency subband coefficient itself. For this reason, this paper adopts phase consistency PC to correct it, and the process is as follows:

The present invention adopts a new active measurement rule NAM to obtain the high-frequency correction fusion weight:

In the formula: PC, LSCM, and LE represent phase consistency, local sharpness transformation, and local energy, respectively; α, β, and γ are scaling factors, respectively.

For high-frequency sub-band coefficients, NAM can integrate the local energy information, detail edge information, and gradient information carried by the coefficient itself through a certain ratio, which is beneficial to the identification of the focus area.

In order to verify the effectiveness of the multi-focus image fusion method based on NSST and salient information extraction proposed by the present invention, a series of comparative experiments were carried out. In the experiment, three groups of different focused images with sizes of 512pixel×512pixel, 640pixel×480pixel, and 512pixel×512pixel were selected for fusion experiments, and compared with five existing common algorithms; at the same time, six evaluation indicators were used for quantitative evaluation. The comparison images used have been registered, and the experimental results are shown in Table 1:

Table 1 Average index results

Tab.1 The result of average index

It can be seen from the table that the method of the present invention improves the visual clarity of the fused image to 0.9009 and the structural similarity to 0.9945 on the premise of retaining sufficient fusion information. Among the other indicators, except for the slight decrease in the standard deviation STD, the rest have improved to a certain extent. The algorithm in this paper not only effectively preserves the outline, texture and other details of the source image, but also has a good visual effect in the focused edge area of the image. The contrast definition of the fused image has been improved to a certain extent, and the fusion effect is ideal. Therefore, the algorithm in this paper is a feasible multi-focus image fusion method.

Claims (5)

1. The multi-focus image fusion method based on NSST and significant information extraction is characterized by comprising the following steps:

(1) Processing different focusing images by using NSST transformation to obtain high-low frequency sub-band coefficients;

(2) Performing preliminary treatment on the low-frequency subband coefficient obtained in the step (1) by adopting an initial low-frequency fusion rule of improved Laplace energy Sum (SML) to obtain an initial low-frequency fusion weight;

(3) Performing certain error correction on the result in the step (2) by using a low-frequency correction fusion rule extracted by the saliency information;

(4) Performing preliminary treatment on the high-frequency subband coefficient obtained in the step (1) by adopting an initial high-frequency fusion rule based on a correlation coefficient to obtain an initial high-frequency fusion weight;

(5) A series of high-frequency fusion weights are assisted by Phase Consistency (PC) correction rules with different degrees, and discrimination correction is carried out;

(6) And (3) performing NSST inverse transformation on the processing results obtained in the steps (3) and (5) to obtain a fusion result.

2. The method of claim 1, wherein in step (3), a non-local mean filtering correction fusion rule is added to make the low frequency fusion weight more accurate, and the correction fusion rule is as follows:

filtering the source image by utilizing non-local mean filtering, and performing difference value operation on the source image and the filtered image to obtain significance information D _l ：

D _l ＝|I _l -I _l ×NMLF| (1)

(1) Wherein: i _l (0 < L < L) is a source image, L represents the number of the source images, and x represents the filtering operation;

and then, guiding filtering is utilized to obtain detail information of a focusing area:

G _l ＝guidedfilter(I _l ,D _l ,r,eps) (2)

and finally, obtaining the low-frequency sub-band correction fusion weight by adopting a big strategy, and carrying out error correction on the initial fusion weight.

3. The method of claim 1, wherein the spatial frequency and energy based on the correlation coefficient are extracted by applying the initial high frequency fusion weights in step (4), the extraction process is as follows:

for a focused region and an unfocused region of an image, the spatial frequency correlation coefficient value sf_corr and the energy correlation coefficient value (eng_corr) of the focused region tend to be larger than those of the unfocused region; by utilizing the point, the two are weighted and combined to form spatial frequency and energy (SF_Eng_Corr) based on a correlation coefficient, and then an initial high-frequency fusion weight is obtained by utilizing a big strategy;

wherein: sf_corr and eng_corr represent spatial frequency correlation coefficients and energy correlation coefficients, respectively.

4. The method of claim 1, wherein in step (5), phase consistency fusion correction is performed on the initial high-frequency fusion weight, and the high-frequency detail information, the local energy information and the gradient edge information are integrated together by a certain proportion, so that discrimination of a focusing area is facilitated.

5. The method of claim 1, wherein the initial fusion weights are obtained according to different fusion rules for the high-frequency and low-frequency subband coefficients obtained through NSST transformation, and the initial weights are respectively corrected by using correction fusion rules, so that the information quantity and definition of the fused image are improved to a certain extent.

Images