KR102142934B1 - Apparatus and Method for Fusing Using Weighted Least Squares Filter and Sparse Respresentation - Google Patents

Abstract

The image fusion device and method using a weighted least squares filter and sparse representation is a method of fusing medical images based on clothing decomposition and sparse representation, and is reconstructed by processing a low-frequency layer by combining the fusion rule of the Laplacian pyramid and sparse representation. There is an effect of improving the image quality of medical images.

Description

Apparatus and Method for Fusing Using Weighted Least Squares Filter and Sparse Respresentation}

The present invention relates to a method of fusing medical images, and in particular, as a method of fusing medical images based on image decomposition and sparse expression, a medical image reconstructed by processing a low frequency layer by combining a fusion rule of a Laplacian pyramid and a sparse expression. The present invention relates to an image fusion apparatus and method using a weighted least squares filter and sparse expression to improve the image quality of.

Medical image fusion can obtain high-quality images by integrating complementary information of medical images for clinical diagnosis and treatment.

Medical imaging is becoming increasingly important in clinical diagnosis with the development of medical diagnostic technology.

Medical images can be obtained for diagnosis in a variety of medical equipment such as ultrasound, X-ray, single photon emission computed tomography (SPECT), computed tomography (CT), magnetic resonance imaging (MRI), and the like.

Recent diseases cannot provide sufficient information for a doctor to diagnose with only one type of medical image, and several types of medical images must be combined. Medical image fusion helps doctors make an accurate diagnosis by combining different types of medical image information.

The current medical image fusion method has the advantage of fast calculation by directly manipulating the pixel points of the image in the spatial domain, but it always brings spatial distortion to the image fusion result and cannot provide a lot of spectral information, so the quality of the fused image is poor. have.

Korean Patent Registration No. 10-1883806

In order to solve such a problem, the present invention is a method of fusing medical images based on image decomposition and sparse expression, and the image quality of the reconstructed medical image by processing the low frequency layer by combining the fusion rule of the Laplacian pyramid and the sparse expression. An object of the present invention is to provide an image fusion apparatus and method using a weighted least squares filter and a sparse expression that improves.

An image fusion method using a weighted least squares filter and a sparse expression according to a feature of the present invention for achieving the above object,

Decomposing each of the input images into a low frequency (LF) layer and a high frequency (HF) layer using a weighted least squares (WLS) filter on one or more input images;

Configuring each of the LF layers as sparse coefficient vectors using sparse representation, and fusing them to reconstruct a final LF layer fused;

Fusing each of the HF layers using a maximum absolute fusion rule to reconstruct a final HF layer; And

And reconstructing a final image by adding the final LF layer and the final HF layer.

An image fusion device using a weighted least squares filter and a sparse expression according to a feature of the present invention,

An image decomposition unit for decomposing each of the input images into a low frequency (LF) layer and a high frequency (HF) layer by using a weighted least squares (WLS) filter on one or more input images;

A low-frequency layer fusion unit configured to each of the LF layers as sparse coefficient vectors using a sparse representation, and fusing them to reconstruct a final LF layer fused;

A high-frequency layer fusion unit configured to reconstruct a final HF layer by fusing each of the HF layers using a maximum absolute fusion rule; And

And a final image fusion unit reconstructing a final image by adding the final LF layer and the final HF layer.

According to the above-described configuration, the present invention can improve the image quality of the entire reconstructed medical image by processing the low-frequency layer by combining the fusion rule of the Laplacian pyramid and the sparse expression to improve the image quality near the edge of the fused image. It works.

1 is a diagram showing the configuration of an image fusion apparatus using a weighted least squares filter and a sparse expression according to an embodiment of the present invention.
2 is a diagram conceptually showing the configuration of an image fusion device according to an embodiment of the present invention.
3 is a diagram showing an example of multi-scale decomposition of a WLS filter according to an embodiment of the present invention.
4 is a diagram showing an exploded structure of a WLS filter according to an embodiment of the present invention.
5 is a block diagram schematically showing an internal configuration of a low-frequency layer fusion unit according to an embodiment of the present invention.
6 is a diagram conceptually showing the configuration of a low-frequency layer fusion unit according to an embodiment of the present invention.

Throughout the specification, when a part “includes” a certain component, it means that the component may further include other components, not to exclude other components, unless otherwise stated.

1 is a diagram showing a configuration of an image fusion device using a weighted least squares filter and a sparse expression according to an embodiment of the present invention, and FIG. 2 is a diagram conceptually showing the configuration of an image fusion device according to an embodiment of the present invention. , FIG. 3 is a diagram showing an example of multi-scale decomposition of a WLS filter according to an embodiment of the present invention.

The image fusion device 100 using a weighted least squares filter and sparse expression according to an embodiment of the present invention includes an image decomposition unit 110, an image layer storage unit 120, a low-frequency layer fusion unit 130, and a high-frequency layer fusion unit. (140) and a final image fusion unit 150.

The image decomposition unit 110 may decompose the input image to obtain images having a plurality of features at different scales.

The image decomposition unit 110 decomposes the input image I into a low frequency (LF) layer and a high frequency (HF) layer using a weighted least squares (WLS) filter to reduce distortion and halos.

The image decomposition unit 110 decomposes the LF layer and the HF layer for each input image using a WLS filter and stores them in the image layer storage unit 120.

As shown in FIG. 1, the image decomposition unit 110 uses a weighted least squares (WLS) filter to convert one or more input images into a low frequency (LF) layer and an HF. Decompose each into (High Frequency) layers.

More specifically, the image decomposition unit 110 generates smooth images I _k from k=1 to Kth by applying a WLS filter as shown in [Equation 1] below.

Here, a smooth image I _k represents a low frequency (LF) layer.

Here, the difference image represents a high frequency (HF) layer.

As shown in FIG. 2, the image decomposition unit 110 generates a difference image from k=1 to the K-th from two adjacent smooth images as shown in Equation 2 below.

The Weighted Least Squares (WLS) filter is a nonlinear, edge-preserving, smoothing filter that can effectively capture details across multiple scales by preserving and decomposing the edges of multiple scales.

The WLS filter can obtain a filtered image S that minimizes S using the following [Equation 3], and obtains a smooth image S as close as possible to the input image I.

은 데이터 항(Data Term),

은 정규화 항(Regularization Term), W_X 및 W_Y는 수평 및 수직의 평활도 가중치, λ는 두 항 사이의 균형을 맞추는 정규화 매개 변수로 기설정되어 있다.here,

Is the Data Term,

Is a regularization term, W _X and W _Y are the horizontal and vertical smoothness weights, and λ is a normalization parameter that balances the two terms.

The data term can make the filtered image closer to the input image, and the normalization term minimizes the partial derivative of the input image to achieve smoothness.

이 최소로 되는 S 값을 구하는 식이다.[Equation 3] is

This is an equation to find the minimum S value.

3 shows an example of multi-scale decomposition of a WLS filter in which λ is set to 15. Figure 3 (a) is the original image, Figure 3 (b) is the smooth image of Figure 3 (a), Figure 3 (c) is the smooth image of Figure 3 (b), Figure 3 (d) is a difference image of FIGS. 3A and 3B, and FIG. 3E is a difference image of FIGS. 3B and 3C.

4 is a diagram showing a decomposed structure of a WLS filter according to an embodiment of the present invention, FIG. 5 is a block diagram schematically showing an internal configuration of a low frequency layer fusion unit according to an embodiment of the present invention, and FIG. A diagram conceptually showing a configuration of a low frequency layer fusion unit according to an embodiment.

The low frequency layer fusion unit 130 according to an embodiment of the present invention includes a Laplacian pyramid decomposition unit 131, an SLF layer storage unit 132, an SHF layer storage unit 133, a sparse expression unit 134, and an SLF layer configuration unit. (135), an SHF layer configuration unit 136 and an LF reconstruction unit 137.

The Laplacian pyramid decomposition unit 131 decomposes the LF layer into a set of a sub low frequency layer (SLF layer) and a sub high frequency layer (SHF layer) using the Laplacian pyramid, and stores the SLF layer. 132 and SHF layer storage unit 133, respectively.

)로 분해한다. 여기서, N은 라플라시안 피라미드 분해의 레벨이다.As shown in FIG. 4, the Laplacian pyramid decomposition unit 131 receives one or more LF layers from the image layer storage unit 120, and applies the Laplacian pyramid to two SLF layers (sublow). _A , sublow _B ) and a set of multiple SHF layers (

). Where N is the level of Laplacian pyramid decomposition.

크기의 패치로 분해한다. 여기서, 패치는 영상 처리를 위한 최소 단위를 나타낸다.The sparse representation unit 134 uses a sliding window technology to create two SLF layers (sublow _A and sublow _B ) having a step length of S pixels from upper left to lower right.

Disassemble into sized patches. Here, the patch represents the minimum unit for image processing.

는 동일한 위치 i를 가진 각각의 sublow_A와 sublow_B로부터 얻은 패치이다. 여기서, M은 패치들의 총 개수이다.Representing an exploded patch

Is the patch from sublow _A and sublow _B respectively with the same position i. Here, M is the total number of patches.

)를 열 벡터(

)로 각각 재배열한 후, 하기의 [수학식 4]와 [수학식 5]에 의해 최종 열 벡터(

)를 생성한다.The sparse representation unit 134 is a patch decomposed at the same position i (

) To column vector (

) After rearrangement, respectively, by the following [Equation 4] and [Equation 5], the final column vector (

).

는 n×1 열 벡터(column vector)이고, 패치(

)의 평균값을 나타낸다.here,

Is an n×1 column vector, and the patch (

) Represents the average value.

는 고화질 영상 집합의 임의의 샘플에서 고정된 크기의 원도우

를 사용하여 샘플링된 패치이다. 여기서, N은 샘플링된 패치의 총 수를 나타낸다.

Is a fixed-size window from any sample of a set of high-definition images.

Is a sampled patch using. Here, N represents the total number of sampled patches.

Then, each patch is rearranged into a column vector, and the unique average value is subtracted so that the intensity of each patch is zero. Patches set a threshold of intensity variance to remove soft patches and have edge information.

The sparse expression unit 134 is interlocked with the dictionary training unit 138 to receive a dictionary.

Dictionary plays a crucial role in sparse representation.

The dictionary training unit 138 can be formulated as shown in [Equation 6] below.

는 훈련 패치(Training Patches),

는 에러 계수(Tolerance Factor),

는 희소 계수(Sparse Cofficients)를 나타낸다.here,

Is the Training Patches,

Is the Tolerance Factor,

Denotes Sparse Cofficients.

를 최소로 하는 D를 계산한다.The dictionary training unit 138 uses the least squares method

Calculate D to minimize

)를 각각 계산한다.The sparse expression unit 134 uses the following [Equation 7] and [Equation 8] as an Orthogonal Matching Pursuit (OMP) algorithm.

) Respectively.

를 최소로 하는

를 계산한다. 하기의 [수학식 7]과 [수학식 8]은 딕셔너리(D)를 전술한 [수학식 6]을 이용하여 이미 훈련된 딕셔너리 훈련부(138)를 통해 제공받고, 최종 열 벡터는 전술한 [수학식 4], [수학식 5]를 이용하여 계산된다.The sparse expression unit 134 uses the least squares method

To minimize

To calculate. The following [Equation 7] and [Equation 8] are provided through the dictionary training unit 138 already trained using the [Equation 6] described above for the dictionary (D), and the final column vector is the aforementioned [Math It is calculated using Equation 4] and [Equation 5].

)와 제2 희소 계수 벡터(

)를 희소 계수 벡터의 융합을 위해 max L1 규칙(수학식 9)을 이용하여 최종 희소 계수 벡터를 구한다. 즉, 최종 희소 계수 벡터는 제1 희소 계수 벡터와 제2 희소 계수 벡터의 대소 여부를 비교하여 파라미터가 더 큰 값을 가진 벡터를 선택한다.The SLF layer construction unit 135 includes a first sparse coefficient vector calculated by the sparse expression unit 134 (

) And the second sparse coefficient vector (

) For the fusion of the sparse coefficient vector, a final sparse coefficient vector is obtained using the max L1 rule (Equation 9). That is, as the final sparse coefficient vector, a vector having a larger parameter is selected by comparing whether the first sparse coefficient vector and the second sparse coefficient vector are large or small.

Subsequently, the SLF layer construction unit 135 calculates the final fusion result information by substituting the final sparse coefficient vector having a larger parameter value into [Equation 10] below.

Here, D is a dictionary, and m is a vector representing the average value of the patch.

)를 얻기 위해서 모든 패치에 대하여 [수학식 3] 내지 [수학식 10]의 과정을 반복한다.The SLF layer configuration unit 135 is a fused LF layer patch (

), the process of [Equation 3] to [Equation 10] is repeated for all patches.

사이즈의 패치(

)로 재구성한다. 여기서,

는 LF 레이어가 융합된 결과를 나타내고, 모든 패치를 원본 패치의 원래 위치에 배치한다. The SLF layer configuration unit 135 creates each fused LF layer patch.

The size of the patch (

). here,

Denotes the result of fusion of the LF layer, and all patches are placed at the original position of the original patch.

)를 수신하고, 수신한 복수의 SHF 레이어에서 3×3 원도우 크기를 가진 최대 절대 규칙(Max Absolute Rule)을 이용하여 픽셀 간 더 큰 값을 가진 벡터를 선택하여 SHF 레이어를 재구성한다.The SHF layer configuration unit 136 is a set of a plurality of SHF layers from the SHF layer storage unit 133 (

) Is received, and the SHF layer is reconstructed by selecting a vector having a larger value between pixels using a Max Absolute Rule with a 3×3 window size from the received SHF layers.

The LF reconfiguration unit 137 performs an inverse Laplacian pyramid on the patch in which the SHF layer received from the SHF layer construction unit 136 and the LF layer received from the SLF layer construction unit 135 are fused, and performs the fused LF layer (L _F ).

)가 된다.The high frequency layer fusion unit 140 reconstructs a plurality of HF layers received from the image layer storage unit 120 into a fused HF layer using a maximum absolute fusion rule. Here, the maximum absolute fusion rule is not a rule that selects a pixel with a maximum value between pixel-to-pixel, but the result of fusion of the HF layer based on the area of the 3×3 window size (

).

The final image fusion unit 150 receives the fused LF layer (L _F ) from the low frequency layer fusion unit 130, receives the fused HF layer from the high frequency layer fusion unit 140, and receives the fused LF layer ( L _F ) and the fused HF layer are added to restore the fused final image (F) (Equation 11).

In the above, the embodiment of the present invention is not implemented only through an apparatus and/or method, and may be implemented through a program for realizing a function corresponding to the configuration of the embodiment of the present invention, a recording medium in which the program is recorded, and the like. There is, such an implementation can be easily implemented by those skilled in the art to which the present invention belongs from the description of the above-described embodiment.

Although the embodiments of the present invention have been described in detail above, the scope of the present invention is not limited thereto, and various modifications and improvements of those skilled in the art using the basic concept of the present invention defined in the following claims are also provided. It belongs to the scope of rights.

100: image fusion device 110: image decomposition unit
120: image layer storage unit 130: low frequency layer fusion unit
131: Laplacian pyramid decomposition unit 132: SLF layer storage unit
133: SHF layer storage unit 134: sparse expression unit
135: SLF layer configuration unit 136: SHF layer configuration unit
137: LF reconstruction unit 138: dictionary training unit
140: high frequency layer fusion unit 150: final image fusion unit

Claims (12)

delete delete delete delete delete delete An image decomposition unit for decomposing each of the input images into a low frequency (LF) layer and a high frequency (HF) layer by using a weighted least squares (WLS) filter on one or more input images;
A low-frequency layer fusion unit configured to each of the LF layers as sparse coefficient vectors using a sparse representation, and fusing them to reconstruct a final LF layer fused;
A high-frequency layer fusion unit configured to reconstruct a final HF layer by fusing each of the HF layers using a maximum absolute fusion rule; And
A final image fusion unit for reconstructing a final image by adding the final LF layer and the final HF layer,
The image decomposition unit generates smooth images I _k from k=1 to Kth by applying the WLS filter according to Equation 1 below, and generates two adjacent smooth images by Equation 2 below. A difference image is generated from k=1 to the K-th, and the WLS filter uses Equation 3 below.

We get a filtered image S that makes this minimum S value, a smooth image S that is as close as possible to the input image I,
The low frequency layer fusion unit,
A set of two SLF (Sub Low Frequency) layers (sublow _A , sublow _B ) and a plurality of SHF layers by applying a Laplacian pyramid to each of the LF layers (

) And storing the SLF layer storage unit and the SHF layer storage unit respectively; And
Using sliding window technology, two SLF layers (sublow _A and sublow _B ) are divided into the smallest unit with a step length of S pixels from top left to bottom right

Decomposed into patches of size, and the decomposed patch at the same position i (

) To column vector (

) After rearrangement, respectively, by the following equations 4 and 5, the final column vector (

) Further comprising a sparse expression unit that generates,
Using the least squares method in Equation 8 below

Further comprising a dictionary training unit for calculating the dictionary (D) to minimize the,
The sparse expression unit uses an Orthogonal Matching Pursuit (OMP) algorithm, and substitutes the calculated dictionary into Equations 6 and 7 below to obtain a sparse coefficient vector (

An image fusion device using a weighted least squares filter and sparse expression, characterized in that each computes ).

Is a fixed-size window from any sample of a set of high-definition images.

Is the sampled patch using, N represents the total number of sampled patches,
[Equation 1]

Here, a smooth image I _k represents a low frequency (LF) layer.
Here, the difference image represents a high frequency (HF) layer.
[Equation 2]

[Equation 3]

here,

Is the Data Term,

Is a regularization term, W _X and W _Y are horizontal and vertical smoothness weights, λ is a normalization parameter that balances the two terms, and the data term is the filtered image added to the input image. It can be made close, and the normalization term minimizes the partial derivative of the input image to achieve smoothness.
[Equation 4]

[Equation 5]

here,

Is an n×1 column vector, and the patch (

) Represents the average value.
[Equation 6]

[Equation 7]

Where D is a dictionary,

Is the Tolerance Factor,

Denotes Sparse Cofficients.
[Equation 8]

here,

Is the Training Patches,

Is the Tolerance Factor,

Denotes Sparse Cofficients. delete delete The method of claim 7,
The first sparse coefficient vector (

) And the second sparse coefficient vector (

) For the fusion of the sparse coefficient vectors, a final sparse coefficient vector, which is a vector having a larger parameter, is obtained by using Equation 9 below for comparing the magnitude of the first sparse coefficient vector and the second sparse coefficient vector. An SLF layer construction unit that calculates final fusion result information by obtaining and substituting the final sparse coefficient vector into Equation 10 below;
An SHF layer constructing unit configured to reconstruct an SHF layer by selecting a vector having a larger value between pixels using a maximum absolute rule having a 3×3 window size from the set of the plurality of SHF layers; And
An image fusion device using a weighted least squares filter and sparse expression, comprising: an LF reconstruction unit configured to reconstruct the final fusion result information and the reconstructed SHF layer into a fused LF layer (L _F ) by performing an inverse Laplacian pyramid. .
[Equation 9]

[Equation 10]

The method of claim 10,
And the final image fusion unit reconstructs a final image by adding the fused LF layer (L _F ) and the final HF layer. The method of claim 10,
The SLF layer configuration unit is a fused LF layer patch (

), repeat the process of Equation 3 to Equation 10 for all patches, and each fused LF layer patch

The size of the patch (

), and the

Denotes a result of fusion of the LF layers, and an image fusion device using a weighted least squares filter and sparse expression, characterized in that all patches are placed at the original positions of the original patch.

Images