WO2014102458A1

WO2014102458A1 - Method and apparatus for image fusion

Info

Publication number: WO2014102458A1
Application number: PCT/FI2013/051212
Authority: WO
Inventors: An JIANG; Qiwei XIE; Feng CUI; Shubo ZI
Original assignee: Nokia Inc
Current assignee: Nokia Inc
Priority date: 2012-12-31
Filing date: 2013-12-30
Publication date: 2014-07-03
Anticipated expiration: 2015-06-30
Also published as: CN103914815A

Abstract

The present invention provides a method for image fusion and a corresponding apparatus. In the present invention, first, a plurality of images are acquired. Then, the plurality of images are fused based on a total variation-L norm energy function. Wherein,the total variation-L norm energy function comprises two components, with one component being a total variation component, which represents retaining the detail information of a to-be-fused image, and the other component being a L norm component, which represents preserving the spectral information of a to-be-fused image. According to the present invention, detail information and spectral information of the fused image can be balanced.

Description

Method and Apparatus for Image Fusion

FIELD OF THE INVENTION

The present invention relates to an image fusion technology and, more specifically, relates to a method and apparatus for image fusion.

BACKGROUND OF THE INVENTION

Image fusion refers to a process of combining a plurality of source images into a high quality single image by performing image processing to the plurality of source images about one same object so as to maintain as much good information as possible in respective source images.

Image fusion may be applied in fields such as remote sensing, medical imaging, quality and defect detection, and biometric, etc. Particularly in mobile communication, there is a strong desire to fuse multiple pictures to obtain a better description and explanation about the sensed scene.

Image fusion should follow some fusion rules to construct a synthetic image. Therefore, it is an issue how to establish a fusion rule/algorithm. Conventional algorithms can be classified into three types: 1) projection and substitution methods, such as Intensity- Hue- Saturation (HIS) color fusion, and Principal Component Analysis (PCA) fusion; 2) band ratio and arithmetic combination, such as multiplicative and Synthetic Variable Ratio (SVR); 3) wavelet based fusion techniques.

Nunez et al. proposed fusing a high spatial resolution image (SPOT) with a low spatial resolution multispectral image (Landsat Thematic Mapper (TM) image) using the additive wavelet (AW) algorithm in the literature "Multiresolution based image fusion with additive wavelet decomposition, IEEE Tran. Geosci. Rem. Sensing, 32 (3), 1204-121 1 (1999)." In this solution, the Atrous wavelet approximation of SPOT image is substituted by the bands of TM image. Li et al. proposed an area-based maximum selection rule (AMS) based on the wavelet transform in the literature "Multisensor image fusion using the wavelet transform, Graphical Models and Image Processing, 57, 235-245 (1995)." A novel algorithm based on multi-scale first fundamental form (MFF) method was presented by Scheunders, which used a multi-valued image wavelet representation method to fuse image, in literatures "A Multivalued Image Wavelet Representation Based on Multiscale Fundamental Forms, IEEE Transactions on Image Processing, 10 (5), 568-575 (2002)" and "Fusion and merging of multispectral images using multiscale fundamental forms, Journal of the Optical Society of America A, Optics, Image Science, and Vision. 18 (10), 2468-2477 (2001." Chen improved a weighted multi-scale first fundamental form method (WMFF) to avoid the problem that MFF enlarges the wavelet coefficients in the literature "Image fusion using weighted multiscale fundamental form, Proceedings of the 2004 International Conference on Image Processing, 5, 3319-3322 (2004)."

However, the above wavelet based fusion methods have some common limits: one is that the predefined wavelet basis has to be selected; another is how to select wavelet decomposition level properly. Moreover, AW and AMS preserve more spectral information and ignore some spatial information, while MFF and WMFF pay more attention to spatial or detail information but distort the spectral information.

In order to solve the above problems, a new image adaptive decomposition algorithm is introduced by Chen et al. in the literature "Fusion of color microscopic images based on bidimensional empirical mode decomposition. OPTICS EXPRESS, 18, 21757-21769 (2010)." However, results of such image decomposition algorithms are not stable.

SUMMARY OF THE INVENTION

According to a first aspect of the present invention, there is provided a method, comprising: acquiring a plurality of images; fusing the plurality of images based on a total variation-Li norm energy function, wherein the total variation-Li norm energy function comprises two components, with one component being a total variation component, which represents retaining the detail information of a to-be-fused image, and the other component being a Li norm component, which represents preserving the spectral information of a to-be-fused image.

According to a second aspect of the present invention, there is provided an apparatus, comprising: an acquiring unit configured to acquire a plurality of images; a fusing unit configured to fuse the plurality of images based on a total variation-Li norm energy function, wherein the total variation-Li norm energy function comprises two components, with one component being a total variation component, which represents retaining the detail information of a to-be-fused image; and the other component being a Li norm component, which represents preserving the spectral information of a to-be-fused image.

According to a third aspect of the present invention, there is provided an apparatus, comprising: means for acquiring a plurality of images; means for fusing the plurality of images based on a total variation-Li norm energy function, wherein the total variation-Li norm energy function comprises two components, with one component being a total variation component, which represents retaining the detail information of a to-be-fused image, and the other component being a Li norm component, which represents preserving the spectral information of a to-be-fused image.

According to a fourth aspect of the present invention, there is provided an apparatus, comprising: at least one processor and at least one memory including computer program code, wherein the processor and the memory are configured to cause the apparatus to performs at least the following, using the processor: acquiring a plurality of images; fusing the plurality of images based on a total variation-Li norm energy function, wherein the total variation-Li norm energy function comprises two components, with one component being a total variation component, which represents retaining the detail information of a to-be-fused image, and the other component being a Li norm component, which represents preserving the spectral information of a to-be-fused image.

According to the present invention, detail information and spectral information of the fused image can be balanced.

BRIEF DESCRIPTION OF THE DRAWINGS

Other objectives and effects of the present invention will become much more apparent and easier to understand from the following description with reference to the accompanying drawings and with more comprehensible understanding of the present invention, wherein: Fig. 1 shows a flow chart diagram/ block diagram of an image fusion method/ apparatus 100 according to one embodiment of the present invention;

Fig. 2 shows a flow chart of a value calculation solution according to one embodiment of the present invention;

Fig. 3 shows fused images derived through the solution of the present invention and according to the existing AW, AMS, MFF methods;

Fig. 4 schematically shows a block diagram of an image fusion apparatus according to one embodiment of the present invention.

In all of the above accompanying drawings, like reference numbers refer to the same, like or corresponding features or functions.

DETAILED DESCRIPTION OF THE INVENTION

Some preferred embodiments of the present disclosure will be described in more detail with reference to the accompanying drawings. Although the preferred embodiments of the present disclosure have been illustrated in the drawings, it should be understood that the present disclosure can be implemented in various manners, and thus should not be construed to be limited to the embodiments set out herein. Rather, those embodiments are provided for the thorough and complete understanding of the present disclosure, and for completely conveying the scope of the present disclosure to those skilled in the art.

The present invention provides a new image fusion solution. The basic idea of the present invention lies in fusing a plurality of images based on a total variation-Li norm energy function so as to balance detail information and spectral information of a fused image, wherein said total variation-Li norm energy function comprises two components, with one component being a total variation component, which represents retaining the detail information of a to-be-fused image, and the other component being a Li norm component, which represents preserving the spectral information of a to-be-fused image.

Fig. 1 shows a flow chart/block diagram of an image fusion method/ apparatus 100 according to one embodiment of the present invention.

As shown in Fig. 1 , the method/ apparatus 100 comprises: an acquiring step/acquiring unit 1 10 for acquiring a plurality of images; a fusing step/fusing unit 120 for fusing the plurality of images based on a total variation-Li norm energy function, wherein the total variation-Li norm energy function comprises two components, with one component being a total variation component, which represents retaining the detail information of a to-be-fused image, and the other component being a Li norm component, which represents preserving the spectral information of a to-be-fused image.

According to one embodiment of the present invention, before the fusing step/fusing unit 120, there further comprises a registration step/registration unit 1 15 for performing a registration operation to the plurality of images.

For two images shot from different angles and at different positions, due to different shooting conditions, each image can only reflect features in some aspects. In order to fuse the two images, one image thereof has to be moved and rotated so as to be aligned with the other image. This alignment process is a registration process. The image that keeps still is called a reference image, and the image that changes is called a floating image. By fusing the aligned images, a fused image that reflects the panorama may be derived.

In the following description of the embodiments of the present invention, there are two to-be-fused images, one being a high resolution image with richer detail information, and the other being a low resolution multispectral image with richer spectral information.

Of course, those skilled in the art should understand that the present invention is also applicable to the scenario where the number of to-be-fused images is greater than two.

According to one embodiment of the present invention, for a high resolution image G, the intensity part IM of a multispectral image M and the fused image R, the energy function can be presented as

wherein ||V(R - G)|| represents a total variation component, |R - / | represents a Li norm component, R is the fused image, λ is a weighted parameter, such as a positive constant, and V is a gradient operator.

It should be understood that G, IM, and R are all images (appearing in the form of matrix during the computing process). The total variation component preserves the detail information, while the Li norm component preserves the spectral information. Based on the energy function, balance is conducted between the detail information and the multispectral information through selecting an appropriate λ value. Wherein, the appropriate λ value can be empirically 5 derived.

According to one embodiment of the present invention, the fused image R may be solved by minimizing the above energy function (1).

In other words, it may be stated as:

10

+ \\R - ml } (2)

In the present invention, the total variation is an isotropic total variant defined as:

wherein u is the image of L*Q, L denotes the line number of image pixels, Q denotes the column number of image pixels, i.e., u is a matrix.

u_x, Uy denote the partial derivative of u, and (i, j) denotes the position of the image pixel point.

The function shown in equation (1) may be simplified as:

0

(6) wherein K=R-G. Since there is a linear relation between K and R, minimization of the energy function of K corresponds to minimization of the energy function of R.

Since the function of equation (6) is not smooth, it is difficult to resolve K, and therefore it is required to be subject to smoothing processing.

5 In other words, according to one embodiment of the present invention, the image fusion method/ fusion image apparatus 100 further comprises a smoothing step/ smoothing unit 130 for performing smooth processing to the equation (6) so as to derive K.

There have been some papers to solve the non- smooth problem. For example, Brox, T. et al. proposed using a smooth function to approximate the non-smooth function in the literature "High accuracy optical flow estimation based on a theory for warping, European Conference on Computer Vision, 2004." Besides, Zach, C. et al proposed a dual method based on the decoupled method in the literature "A duality based approach for realtime TV-LI optical flow, The Annual Symposium of the German Association for Pattern Recognition, 2007."

According to one embodiment of the present invention, the above equation (6) is calculated using a decoupling method and a primal-dual method.

Here, a new variant N and a new parameter p are introduced to rewrite the equation (6) as:

mm_{K N} (E(K, N)) =

(7)

where p is a big constant, such that N could be vey close to K. Wherein, || \ζ represents the square of 2 norm.

The saddle-point formulation of (7) is obtained as follows:

where (·,·) is Hilbert inner product, p is a dual variable of VK , p e P , p = (p_i ¹ , pl_J) , the set P is given by P =

< l} , which is a unit ball of the infinite norm || ||_∞ space, and is defined as:

where i and j are pixel positions of the image. p)_d and p _j are components of vector p, respectively.

The function δ_ρ (ρ) is the indicator function of the set P as follows:

In order to express the algorithm clearly, the numerical scheme is represented as what is shown in Fig. 2, and the equation (8) is represented as dual step 210, primal step 220 and threshold step 230 as follows:

Dual step

If K = K is fixed, i is the iteration step number, then solve the equation: max_p{(p,VKⁱ)-5_p(p)} (11)

Because d_p (p, ^K¹ ) = VK¹ _s d_p is a derivative of p, p is updated as:

p^M = prox^K¹ +τ^Κ^ι (12)

where prox^ is the proximal operator, τ. is the dual stepsize, and in this embodiment,

Primal step

If p = p^i+l ,N = Nⁱ is fixed, then solve

min, (E(K)) = min, {(p,VK) + p\\ K-N\\² ₂} (13)

and transform equation (13) into:

min, (E(K)) = min, {-(K, divp) + p\\K-N\\² ₂} (14)

where div is the divergence operator, so K i

where σ, is the primal stepsize.

Threshold step

If K = K^i+l is fixed, then solve the following equation

min^ {λ|| N-(/ -G)ll_j + pll K-N\\² ₂}, (16)

Then N is updated as:

K^i+l -— , if K^i+l > (IM-G)+—

2p 2p

Kⁱ⁺¹ +— , if Kⁱ⁺¹ < (IM-G)-— (17)

2p 2p

K'^+l , else

After the threshold step is completed, it is judged whether a convergence condition is satisfied; if the convergence condition is satisfied, then exit from the iterative calculation; if the convergence condition is not satisfied, then return to the dual step. It should be understood that the order of dual step, primal step, and threshold step may change, not necessarily the order shown in Fig.2.

T₁ and σ, may be selected as: τ_ί = 0.2 + 0.08/ , σ_; = (0.5 — ) /τ_ί , and the convergence

15 + k |N⁺¹ - N |²

condition may be selected as -— ^——— < ε , where ε is a small constant. If K is obtained, R is solved by R = K + G .

With the hue part HM, saturation part SM, and R, which is taken as the intensity part, the fused image can be obtained.

A series of experiments about IKONOS image are used to validate the present invention.

In order to quantitatively evaluate the fused result, the following three criteria are used. First, CM (Correlation measure) (see Nunez, et al, Multiresolution based image fusion with additive wavelet decomposition, IEEE Transactions on Geoscience and Remote Sensing., 1999, 32, (3), pp. 1204-121 1) computes correlation coefficient of red, green and blue channel between the multispectral image and fused result, which can be used to assess the preservation of the spectral information of the fused image. A larger value of CM represents that the fused image obtains less distortion of the colors. Second, M (Mutual information measure) (see Qu et al,

"Information measure for performance of image fusion, Electronics Letters, 2002, 38, (7), pp. 313-315") can reflect the total amount of information that the fused image contains about the multispectral image and the high-resolution image. A larger value of mutual information measure indicates that the fused image contains more information from original images. Third, QABIF (SQQ Xyd_eas et al, "Objective Image Fusion Performance Measure, Electronics Letters,

2000, 36, (4), pp. 308-309") considers the amount of edge information transferred from the to-be-fused images to the fused image. A larger Q^ABIF value indicates that the fused image preserves more edge information from the to-be-fused images.

The inventor performs, on IKONOS satellite images, the method proposed in the present invention, the AW method proposed by Nunez et al. in the literature "Multiresolution based image fusion with additive wavelet decomposition, IEEE Transactions on Geoscience and Remote Sensing., 1999, 32, (3), pp. 1204-121 1 ," the AMS method proposed by Li et al. in the literature "Multisensor image fusion using the wavelet transform, Graphical Models and Image Processing, 1995 , 57, pp. 235-245," and the MFF method proposed by Scheunders in the literature "A Multivalued Image Wavelet Representation Based on Multiscale Fundamental Forms, IEEE Transactions on Image Processing, 2002, 10, (5), pp. 568-575," and uses the above three metrics to compare the fused images derived through different methods.

For the solution proposed in the present invention, λ=\ and p=12.5. Besides, the wavelet decomposition level of AW, AMS and MFF solutions is 4.

Fig. 3 shows fused images derived through the solution of the present invention and through the above AW, AMS, MFF methods. Wherein reference numeral 310 denotes the fused images derived through the solution of the present invention and through the above AW, AMS, and MFF methods; reference numeral 320 represents a to-be-fused multispectral image, and reference numeral 330 represents a to-be-fused high-resolution image. Reference numeral 340 indicates comparison of spectral information in respective images, and the reference numeral 350 indicates comparison of detail information in respective images.

Table 1 provides comparison of the above three metrics of the fused images derived through the solution of the present invention and through the above AW, AMS, and MFF methods.

CMired CMigre ^■.en CM(blue AB QAB/ F

F

cha nel) channel } channel)

AW 0.934 i 0.9345 0.9402 1 ,5829 0.4109

AMS 0.9426 0.9443 0.9501 1 .5452 0.3491

MFF 0.826S 0,8250 0.8466 1.5870 0.3700

Our method 0.9503 0.9534 0.9567 1.8599 0,4225

Table 1

The first three columns of Table 1 represent that the solution of the present method preserves more spectral information than the other three existing methods. The fourth column of Table 1 represents that more information from original images is obtained in the solution of the present invention than in the other three existing methods. The fifth column of Table 1 represents that the image resulting from the solution of the present invention has more accurately edge information from original images, comparing with the other three existing methods.

Fig. 4 schematically shows a block diagram of an image fusion apparatus according to one embodiment of the present invention. As shown in Fig. 4, the image fusion apparatus 400 comprises a data processor (DP) 401 and a memory (MEM) 403 coupled to the data processor 401. The memory 403 stores a program (PROG) 402. The memory 403 may be of any appropriate type suitable for the local technical environment and may be implemented by any appropriate data storage technologies, including, but not limited to, a semiconductor-based storage device, a magnetic storage device and system, an optical storage device and system. Although merely one memory unit is shown in Fig. 4, the image fusion apparatus 400 may have a plurality of physical differently memory units. DP 401 may be of any appropriate type suitable for the local technical environment and may include, but not limited to, a general computer, a dedicated computer, a microprocessor, a digital signal processor (DSP) and one or more of the processor-based multi-core processor architectures. The image fusion apparatus 400 may comprise multiple processors.

As shown in Fig. 4, the data processor 401 and the memory 403 are configured to cause the image fusion apparatus 400 to perform at least the following, using the data processor 401 : acquiring a plurality of images; fusing the plurality of images based on a total variation-Li norm energy function, wherein the total variation-Li norm energy function comprises two components, with one component being a total variation component, which represents retaining the detail information of a to-be-fused image, and the other component being a Li norm component, which represents preserving the spectral information of a to-be-fused image.

According to one embodiment of the present invention, the plurality of images are fused by minimizing the total variation-Li norm energy function.

According to one embodiment of the present invention, the energy function assumes the following form:

wherein ||V(R - represents the total variation component, ||R -

the Li norm component, R is the fused image, G is the image with richest detail information among the to-be-fused images, IM is the intensity part of the image with richest spectral information among the to-be-fused images, λ is a weighted parameter, V is a gradient operator, and the fused image R is solved by minimizing the above energy function.

According to one embodiment of the present invention, the data processor 401 and the memory 403 are further configured to cause the image fusion apparatus 400 to perform at least the following, using the data processor 401 : performing smoothing operation to the above energy function so as to solve the fused image R.

According to one embodiment of the present invention, the above energy function is calculated using a decoupling method and a primal-dual method.

According to one embodiment of the present invention, the data processor 401 and the memory 403 are further configured to cause the image fusion apparatus 400 to perform at least the following, using the data processor 401 : performing a registration operation to the plurality of images before fusing the plurality of images.

According to one embodiment of the present invention, there are two to-be-fused images, wherein one is a high-resolution image with richer detail information, and the other is a low-resolution multispectral image with richer spectral information.

It should be noted that in order to make the present invention more comprehensible, some more specific technical details, which are known to the skilled in the art but may be essential to implement the present invention, are omitted in the above description.

Thus, the preferred embodiments are selected and described to better illustrate the principle and practical application of the present invention and to enable an ordinary skilled in the art to appreciate that all modifications and alterations fall within the protection scope of the present invention as limited by the appending claims, without departing the spirit of the present invention.

Further, those skilled in the art may understand, the steps of various methods as described above may be implemented through a programming computer. Here, some embodiments are intended to cover program storage device that is a machine or computer-readable and encoded with machine-executable or computer-executable instruction program, wherein the instruction performs some or all steps of the above methods. The program storage device may be a magnetic storage media, for example, a disc and a tape, a hard disk driver, or an optical readable digital data storage media. The embodiments are also intended to cover a computer that is programmed to perform the steps of the method.

Claims

WHAT IS CLAIMED IS:

1. A method, comprising:

acquiring a plurality of images;

fusing the plurality of images based on a total variation-Li norm energy function, wherein the total variation-Li norm energy function comprises two components, with one component being a total variation component, which represents retaining the detail information of a to-be-fused image, and the other component being a Li norm component, which represents preserving the spectral information of a to-be-fused image.

2. The method according to claim 1 , wherein:

the plurality of images are fused by minimizing the total variation-Li norm energy function.

3. The method according to claim 2, wherein the energy function assumes the following form:

norm component, R is the fused image, G is the image with richest detail information among the to-be-fused images, IM is the intensity part of the image with richest spectral information among the to-be-fused images, λ is a weighted parameter, V is a gradient operator, and the fused image R is solved by minimizing the above energy function.

4. The method according to claim 3, further comprising: performing smooth processing to the above energy function so as to solve the fused image R.

5. The method according to claim 3, wherein the above energy function is calculated using a decoupling method and a prime-dual method.

6. The method according to claim 1 , further comprising: performing a registration operation to the plurality of images before fusing the plurality of images.

7. The method according to any one of claims 1 -6, wherein the number of to-be-fused images is two, wherein one is a high resolution image with rich detail information and the other is a low resolution multispectral image with rich spectral information.

8. An apparatus, comprising:

an acquiring unit configured to acquire a plurality of images;

a fusing unit configured to fuse the plurality of images based on a total variation-Li norm energy function,

wherein the total variation-Li norm energy function comprises two components, with one component being a total variation component, which represents retaining the detail information of a to-be-fused image, and the other component being a Li norm component, which represents preserving the spectral information of a to-be-fused image.

9. The apparatus according to claim 8, wherein:

10. The apparatus according to claim 9, wherein the energy function assumes the following form:

wherein ||V(R - represents the total variation component, ||R -

1 1. The apparatus according to claim 10, further comprising:

a smoothing unit configured to perform smoothing operation to the above energy function so as to solve the fused image R.

12. The apparatus according to claim 10, wherein the above energy function is calculated using a decoupling method and a prime-dual method.

13. The apparatus according to claim 8, further comprising:

a registration unit configured to perform a registration operation to the plurality of images before fusing the plurality of images.

14. The apparatus according to any one of claims 8- 13, wherein the number of to-be-fused images is two, wherein one is a high resolution image with rich detail information and the other is a low resolution multispectral image with rich spectral information.

15. An apparatus, comprising:

means for acquiring a plurality of images;

means for fusing the plurality of images based on a total variation-Li norm energy function,

16. The apparatus according to claim 15, wherein:

17. The apparatus according to claim 16, wherein the energy function assumes the following form:

wherein ||V(R - represents the total variation component, ||R -

18. The apparatus according to claim 17, further comprising:

means for performing smoothing operation to the above energy function so as to solve the fused image R.

19. The apparatus according to claim 17, wherein the above energy function is calculated using a decoupling method and a prime-dual method.

20. The apparatus according to claim 15, further comprising:

means for performing a registration operation to the plurality of images before fusing the plurality of images.

21. The apparatus according to any one of claims 15-20, wherein the number of to-be-fused images is two, wherein one is a high resolution image with rich detail information and the other is a low resolution multispectral image with rich spectral information.

22. An apparatus, comprising:

at least one processor and at least one memory including computer program code; the processor and the memory being configured to cause the apparatus to perform at least the following, using the processor:

acquiring a plurality of images;

23. The apparatus according to claim 22, wherein:

24. The apparatus according to claim 23, wherein the energy function assumes the following form:

wherein ||V(R - represents the total variation component, ||R -

25. The apparatus according to claim 24, wherein the processor and the memory are further configured to cause the apparatus to perform at least the following, using the processor:

performing smoothing operation to the above energy function so as to solve the fused image R.

26. The apparatus according to claim 24, wherein the above energy function is calculated using a decoupling method and a prime-dual method.

27. The apparatus according to claim 22, wherein the processor and the memory are further configured to cause the apparatus to perform at least the following, using the processor:

performing a registration operation to the plurality of images before fusing the plurality of images.

28. The apparatus according to any one of claims 22-27, wherein the number of to-be-fused images is two, wherein one is a high resolution image with rich detail information and the other is a low resolution multispectral image with rich spectral information.