CN112508828A

CN112508828A - Multi-focus image fusion method based on sparse representation and guided filtering

Info

Publication number: CN112508828A
Application number: CN201910869494.XA
Authority: CN
Inventors: 李启磊; 童嘉蕙; 杨晓敏
Original assignee: Sichuan University
Current assignee: Sichuan University
Priority date: 2019-09-16
Filing date: 2019-09-16
Publication date: 2021-03-16

Abstract

The invention discloses a multifocal image fusion method, which comprises the following steps: (1) carrying out fuzzy processing on the natural image of the training sample for multiple times; (2) randomly extracting image blocks with certain sizes from the training sample images to obtain a sample group, training the image sample group and obtaining a learning dictionary; (3) inputting a test sample image, sampling pixel by pixel from the test sample image to obtain an input image set, calculating to obtain a sparse coefficient and storing the sparse coefficient; (4) combining the sparse coefficient and the learning dictionary to obtain a focus characteristic map; (5) smoothing the focus feature map to determine a score map, and judging the current pixel point according to the score map to obtain an initial decision map; (6) removing black points and small holes in a small area, and refining to obtain a final decision diagram; (7) and restoring the fused image according to the final decision diagram. The method has excellent effect on fusing two pictures with different focuses, and can show more information on the premise of keeping the definition and the resolution of the original image.

Description

Multi-focus image fusion method based on sparse representation and guided filtering

Technical Field

The invention belongs to the technical field of computer vision, and relates to a multi-focus image fusion method, in particular to fusion of images with focuses concentrated in two different areas in the same scene.

Background

Due to the limited depth of field of the vision sensor, it is difficult to obtain a fully focused image, which presents difficulties in the analysis and understanding of the image. Wherein, the image generated by the sensor with excellent focusing power is true and clear, and the images generated by other objects beyond the object determination limit have fuzzy feeling. When a clear image of the target space is taken, it is necessary to take multiple clear images of each target because there is no way to clearly image all the entities in the target space. And the focuses of the images are concentrated in different areas, and a method of fusing pictures with different focuses in the same scene can be adopted to solve the problem. In the multi-focus image fusion method which is popular at present, there are two main branches, a spatial domain method and a transform domain method. The spatial domain method directly fuses source images through a specific fusion rule, and the basic principle is to calculate the mean value of the source images pixel by pixel. However, pixel-based fusion methods tend to suffer from noise and registration errors. The fusion of the initial images is mainly at the pixel level, and is a direct process for various information data contained in the source images. Much research has focused on image fusion at the pixel level, but the amount of information processed in this way is excessive, resulting in redundancy in storage and higher requirements on processing equipment. In order to further improve the fusion performance, a fusion method based on blocks and regions is proposed. For example, selecting blocks of images according to spatial frequency, or decomposing the image into multiple gradients and fusing through a rolling guided filter. In addition to spatial frequency and image gradient, the laplacian energy method is also an important sharpness evaluation method. Although the effects of noise and registration errors are getting smaller, these methods tend to suffer from block artifacts and contrast degradation. Unlike the spatial domain method, the main idea of the transform domain method is to fuse the multi-focus image in the transform domain, such as laplacian pyramid, discrete wavelet transform, dual-tree complex wavelet transform, etc. In addition to the transform domain methods described above, some new transform domain methods such as independent component analysis and sparse representation are also used to fuse the multifocal images. The image fusion method based on sparse representation divides a source image into small blocks through a sliding window with a fixed size, converts the small blocks of the image into sparse coefficients, and then measures the activity level by applying an L1 paradigm to the sparse coefficients. Although some multifocal fusion methods achieve better results, there are still some deficiencies to be solved. For spatial domain methods, some of which are prone to noise and registration errors, blocking artifacts may be generated in the fused image. In addition, some methods also increase artifacts near the boundary, reducing contrast and sharpness. For the transform domain approach, the fusion rule is based on correlation coefficients; thus, small changes in the coefficients will result in large changes in the pixel values, which will result in unwanted artifacts.

Aiming at the problems, a multifocal image fusion method based on sparse coding and a guide filter is provided.

Disclosure of Invention

The present invention is directed to solving the above problems and providing a multi-focus image fusion method with high accuracy.

The invention realizes the purpose through the following technical scheme:

1. the multi-focus image fusion method comprises the following steps:

(1) performing multiple times of fuzzy processing on the natural image of the training sample M by using a Gaussian filter, and controlling the fuzzy degree according to actual needs, wherein the fuzzy degree comprises standard deviation, size, fuzzy iteration times and the like;

(2) randomly extracting image blocks with certain sizes from the training sample images to obtain a sample group, training the image sample group by a K-svd method and obtaining a dictionary D;

(3) inputting a test sample picture, sampling pixel by pixel from the image of the test sample to obtain an input picture set, calculating by using an OMP (object model processing) method to obtain a sparse coefficient X, and storing the sparse coefficient X;

(4) obtaining a remodeling-related effect level measurement vector f from sparse coefficients_iI belongs to {1,2} and calculating to obtain a focus characteristic map E_i,i∈{1,2}；

(5) Smoothing is carried out by a guide filter to distinguish the contour of the focus area from the contour of the defocus area, and a fractional image S is obtained_iI belongs to {1,2}, and then an initial decision diagram Q is obtained;

(6) removing small black dots and small holes in the small area on the premise of not influencing the accuracy of the boundary of the large-area object;

(7) and optimizing the boundary of the decision diagram Q by using a guide filter to obtain a final decision diagram.

(8) And obtaining the fused image according to the final decision diagram.

The basic principle of the method is as follows:

the method trains an over-complete dictionary and calculates related sparse coefficients. These coefficients are used to measure activity level and then a focus profile is derived from the activity level. A derivative filter is applied to the focused feature map to generate a score map. By comparing the score maps, an initial decision map is obtained. Then, using guided filtering

And the device refines the initial decision diagram to obtain a final decision diagram.

Specifically, in the step (1), a gaussian filter is used to perform multiple times of blurring on the natural image of the training sample M, so as to improve sample diversity, and thus, the sampled image block can be expanded into a sparse dictionary better than that of the traditional sparse expression method.

In the step (2), the dictionary is trained by using a K-svd method, firstly, the dictionary D is initialized randomly, that is, n samples are randomly selected from the training sample set Y as atoms of D, and the encoding matrix X is initialized to be a 0 matrix. And step two, fixing the dictionary, and solving the sparse code of each sample by utilizing an OMP (object-matching pursuit) method. The calculation method is as follows:

s.t.||x_i||₀≤k₀,1≤i≤N

wherein

A training sample signal is input to the training device,

in order to be an ultra-complete dictionary,

for input sample signal y_iThe sparse coefficient of (2). At this point, the sample set y is obtained_iAnd (5) sparse coding.

In the step (3), after the dictionary D is learned, it is used to calculate the sparse coefficients of the N input multifocal images. In the stage of sparse coding of test samples, the method adopts the sparse coding and training stageIs sampled pixel by pixel from the image of the test sample to the input picture set, and when these image blocks are sampled they will be expanded to a column vector X ═ X_i1,x_i2,…,x_i(n-1),x_inWhere n denotes the number of image blocks. And obtaining the sparse coefficient of the test sample by utilizing an OMP method, wherein the sparse coefficient is defined as:

s.t.||Y_i-DX_i||₂≤δ

wherein Y is_iFor inputting the test sample signal, D is the overcomplete dictionary, and X is the sparse coefficient of the input test sample signal. Calculating to obtain and store a sparse coefficient X of the test sample;

in the step (4), a remodeling-related effect level measurement vector f is obtained from the sparse coefficient_iI ∈ {1,2}, i.e., f₁＝(f₁₁,f₁₂,…,f_1(n-1),f_1n),f₂＝(f₂₁,f₂₂,…,f_2(n-1),f_2n) Where n represents the number of image blocks, namely:

0＜i≤n

and then obtaining a focus characteristic map according to calculation, wherein the focus characteristic map comprises the following steps:

E_i＝reshape(f_i),i∈{1,2}

wherein reshape (·) is a remolding sparse coefficient operator, E_iAnd i belongs to {1,2} is a focus characteristic map.

In the step (5), smoothing is performed by a guide filter. Specifying the guided filtering operation as the operator GF (-) the score map after guided filtering can be expressed as follows:

S_i＝GF(E_i,E_i,r₁,∈₁),i∈{1,2}

where GF (-) is the pilot filter operator, note that the pilot image of the pilot filter is itself the focus profile. And obtaining an initial decision diagram Q by using the following formula:

wherein S is_1、S₂Respectively, are partial images from the source test sample image.

In the step (6), the partial graphs S are compared_iThe resulting initial decision graph Q may result in some non-smooth edges and some pinholes. This is because some regions have similar visual effects on both input images, and the sparse coefficients cannot determine whether they are in focus. To remove these small holes, we use a small region removal strategy. This process can be represented by the following equation:

Q_r(x,y)＝Q_i(x,y)-smallholes

Q(x,y)＝upsample(Q_r(x,y))

in the step (7), the boundary between the focus area and the non-focus area is smooth, and the decision diagram Q is sharp at the boundary. To solve this problem, we use a guided filter to optimize the decision graph Q. And the fused image W is made the guide image of the guide filter. The following were used:

W＝Q(x,y)I₁(x,y)+(1-Q(x,y))I₂(x,y)

where GF (-) represents the pilot filter operator.

Representing the final decision graph after guided filtering. I is₁、I₂Representing the source input image.

In the step (8), the fused image is obtained by using the following formula:

the invention has the beneficial effects that:

in the invention, sparse coefficients are used for classifying a focusing region and a non-focusing region to construct an initial decision diagram instead of directly fusing the sparse coefficients. The initial decision map will be optimized in the later steps. Thus, artifacts caused by improper selection of sparse coefficients are avoided. Secondly, in order to solve the problem of spatial inconsistency, the guided filtering is used for smoothing the focusing feature map, and connection with adjacent pixels is fully considered. Therefore, the structure of the image is effectively reserved, and the problem of inconsistent space is avoided. In addition, in order to generate a decision graph related to boundary information, a guide filter is used for refining the initial decision graph. So that it has a slow transition and eventually the decision graph boundary is smoothed. Thereby effectively reducing the vignetting artifact of the fused image.

Drawings

FIG. 1-1 is an overall framework diagram in an embodiment of the invention;

FIGS. 1-2 are one of the training set sample images in an embodiment of the present invention;

FIGS. 1-3 illustrate a second example of a training set sample image in accordance with an embodiment of the present invention;

FIGS. 1-4 are one of the images of a training set sample after blurring in an embodiment of the present invention;

FIGS. 1-5 illustrate a second example of a blurred image of a training set sample;

FIG. 2-1 is one of the image blocks extracted from the training set samples in an embodiment of the present invention;

FIG. 2-2 is a learning dictionary D obtained by training image sample sets by the K-svd method according to an embodiment of the present invention;

FIG. 3-1 is one of the images of the test specimen in an embodiment of the present invention;

FIG. 3-2 is a second image of a test specimen in an embodiment of the present invention;

FIG. 4-1 is an image of the test sample image of FIG. 3-1 after thinning to record focus features;

FIG. 4-2 is an image of the test sample image of FIG. 3-2 after thinning to record focus features;

FIG. 5-1 is an image of the focal point feature image of FIG. 4-1 after a guided filtering;

FIG. 5-2 is an image of the focal point feature image of FIG. 4-2 after a guided filtering;

FIG. 5-3 is the fused image of FIGS. 5-1 and 5-2 as an initial decision map;

FIG. 6-1 is the corresponding image of FIG. 5-3 after thinning and hole removal;

FIG. 7-1 is the image of FIG. 6-1 after guided filtering as a final decision graph;

FIG. 8-1 is the final restored fused image according to the fusion principle.

Detailed Description

The invention will be further illustrated with reference to the following specific examples and the accompanying drawings:

example (b):

in order to make the image fusion method of the present invention more easily understood and approximate to the real application, the whole process is described from the beginning of dictionary learning of the original training samples to the completion of image fusion, wherein the whole process includes the core fusion method of the present invention, and the general frame diagram is shown in fig. 1-1:

(1) natural images of the training sample M are blurred multiple times using a gaussian filter, and fig. 1-2 and 1-3 are representative of the original training sample images. The standard deviation, size and number of fuzzy iterations of the gaussian filter are set to 3, 5 × 5 and 5. Fig. 1-4 and fig. 1-5 are training sample images after five times of blurring processing.

(2) Randomly extracting image blocks with a certain size from the training sample images to obtain a sample set, in this example, setting the size of the image blocks to be 8 × 8, as shown in fig. 2-1. Then training the image sample group by a K-svd method to obtain a learning dictionary D, wherein the size of the dictionary is set to be 64 multiplied by 512, as shown in figure 2-2;

(3) and (3) using the updated dictionary trained in the step (2) to calculate sparse coefficients of the N input multi-focus images. In the stage of sparse coding of test samples, the sampling and training stages are adoptedTwo test sample images, such as fig. 3-1, 3-2, 3-3, and 3-4, are selected using a sliding window with identical image blocks, pixel-by-pixel sampling is performed from the test sample images into the input picture set, and when these image blocks are sampled, they are expanded into a column vector X ═ X_i1,x_i2,…,x_i(n-1),x_inWhere n denotes the number of image blocks. And obtaining the sparse coefficient of the test sample by utilizing an OMP method, wherein the sparse coefficient is defined as:

s.t.||Y_}-DX_i||₂≤δ

(4) obtaining a remodeling-related effect level measurement vector f from sparse coefficients_iI ∈ {1,2}, i.e., f₁＝(f₁₁,f₁₂,…,f_1(n-1),f_1n),f₂＝(f₂₁,f₂₂,…,f_2(n-1),f_2n) Where n represents the number of image blocks, namely:

0＜i≤n

E_i＝reshape(f_i),i∈{1,2}

wherein reshape (·) is a reshaping sparse coefficient operator and is a focus feature map. The effect is shown in figure 4-1 and figure 4-2.

(5) Smoothing is performed by a guided filter. Now, by generalizing the above-mentioned guided filtering operation into an operator GF (-) the score map after guided filtering can be expressed as follows:

S_i＝GF(E_i,E_i,r₁,∈₁),i∈{1,2}

where GF (-) is the pilot filter operator, note that the pilot image of the pilot filter is itself the focus profile. The initial decision diagram Q is then obtained using the following formula, as shown in fig. 5-3:

wherein S is₁、S₂Respectively, are partial images from the source test sample image.

(6) By comparing the partial graphs S_iThe resulting initial decision graph Q may result in some non-smooth edges and some pinholes. This is because some regions have similar visual effects on both input images, and the sparse coefficients cannot determine whether they are in focus. To remove these small holes, we use a small region removal strategy. Is defined as:

Q_r(x,y)＝Q_i(x,y)-smallholes

Q(x,y)＝upsample(Q_r(x,y))

the image after removal of the small hole is shown in FIG. 6-1.

(7) The boundaries of the focused and unfocused regions are smooth, while the decision graph Q is sharp at the boundaries. To solve this problem, we use a guided filter to optimize the decision graph Q. The multi-focus image is fused by using a decision diagram Q, and the fused image W is used as a guide image of a guide filter. The following were used:

W＝Q(x,y)I₁(x,y)+(1-Q(x,y))I₂(x,y)

where GF (-) represents the pilot filter operator. I is₁、I₂Representing the source input image, the optimized image is shown in fig. 7-1.

(8) Define the fusion formula as:

the fused image is obtained according to the formula and is shown in figure 8-1. Comparing fig. 3-1, fig. 3-2 and fig. 8-1, it can be seen that the fused image and the source test image before processing, i.e. the effect of the multi-focus image fusion technique of the present invention is better.

Among the above steps, steps (5) to (7) are the main steps of the image fusion method of the present invention.

In this example, we will use a set of parameters (bi-directional information MI, edge preservation Q) that are common in the field of subjective vision and objective image processing as a measure of the amount of information involved^AB/FFMI, standard deviation SD, etc.), the source image and the fused image are compared, thereby verifying the reliability of the present invention.

The above embodiments are only preferred embodiments of the present invention, and are not intended to limit the technical solutions of the present invention, so long as the technical solutions can be realized on the basis of the above embodiments without creative efforts, which should be considered to fall within the protection scope of the patent of the present invention.

Claims

1. The invention relates to a multi-focus image fusion method, which is characterized in that: the method comprises the following steps:

(1) smoothing the contour of the in-focus area and the out-of-focus area by a guide filter to obtain a fractional image S_iI belongs to {1,2}, and then an initial decision diagram Q is obtained;

(2) removing small black dots and small holes in the small area on the premise of not influencing the accuracy of the boundary of the large-area object;

(3) and optimizing the boundary of the decision diagram Q by using a guide filter to obtain a final decision diagram.

2. The multi-focus image fusion method according to claim 1, wherein:

in the step (1), the guiding filter is used for smoothing, and the guiding filtering operation is defined as an operator GF (-) so that the score map after guiding filtering can be expressed as follows:

S_i＝GF(E_i,E_i,r₁,∈₁),i∈{1,2}

wherein GF (-) is a guide filter operator, and the guide image of the guide filter is a focusing characteristic diagram; and obtaining an initial decision diagram Q by using the following formula:

wherein S is₁、S₂Respectively representing partial images from a source test sample image;

in the step (2), the partial graphs S are compared_iThe obtained initial decision graph Q may generate some unsmooth edges and some small holes because some regions have similar visual effects on two input images, and the sparse coefficient cannot determine whether the two input images are focused or not, and in order to remove the small holes, a small region removal strategy is adopted; this process can be represented by the following equation:

Q_r(x,y)＝Q_i(x,y)-smallholes

Q(x,y)＝upsample(Q_r(x,y))

in the step (3), the boundary of the focus area and the non-focus area is smooth, and the decision diagram Q is sharp on the boundary; in order to solve the problem, a guide filter is adopted to optimize a decision diagram Q; and the fused image W is used as a guide image of a guide filter; the following were used:

W＝Q(x,y)I₁(x,y)+(1-Q(x,y))I₂(x,y)

wherein GF (-) represents the guided filter operator;

representing a final decision graph after the guiding filtering; i is₁、I₂Representing the source input image.