KR20220071554A - Medical Image Fusion System - Google Patents

Abstract

The present invention relates to a medical image fusion system using a Laplacian pyramid (LP) and convolutional representation to reduce calculation burden of convolutional sparse representation (CSR). According to the present invention, an image fusion method in an image fusion system comprises the following steps of: calculating an image pyramid including multi-scale level images through image decomposition of each image with respect to two images; fusing base layers having the smallest scale among the multi-scale level images of each image to generate a final fusion base layer (F_B); fusing one or more detail layers of the multi-scale level images of each image to generate a final fused detail layer (F_D); and reconstructing images with respect to the final fusion base layer (F_B) and the final fusion detail layer (F_D) through a reverse process of LP transformation to output a final fusion image.

Description

Medical Image Fusion System

The present invention relates to an image fusion system, and more particularly, to an image fusion system that combines a Laplacian Pyramid (LP) and a Convolutional Sparse Representation (CSR) to fuse medical images and the like.

BACKGROUND With the recent development of imaging technologies such as ultrasound, computed tomography (CT), X-ray, and magnetic resonance imaging (MRI), various imaging techniques have been widely applied to various medical practices. Due to the diversity of imaging mechanisms, these different imaging modalities allow different types of information to be captured. For example, CT images can focus on dense organs and tissues such as bones and implants, whereas MRI images can provide high-resolution soft tissue details. Complementary information of CT and MRI images may provide structural information with high resolution. Therefore, CT and MRI images are generally applied for tumor diagnosis in patients with early head and neck cancer. However, for an accurate diagnosis, a doctor must individually and sequentially analyze various types of images, which can cause inconvenience in many cases. Medical image fusion effectively solves this problem, creating a fused image to integrate complementary information contained in multiple medical images into different formats. Medical image fusion is not only low cost, but also flexible because it can fuse all types of images according to the clinical needs of doctors. Therefore, it has been widely studied in the field of medical imaging, and until recently, various image forming methods have been proposed.

The two main general methods are the spatial domain method and the transform domain method. Spatial domain methods deal directly with the source image on a pixel-by-pixel basis in the spatial domain, such as weighted average (WA), intensity-saturation-saturation (IHS) and principal component analysis (PCA). Although these methods are simple and time-efficient, they do not take much into account the important differences of multimodal medical imaging, and some information is lost. The transform domain method reconstructs a final fusion image by performing a corresponding inverse transform on a fusion coefficient obtained by a specific fusion rule of the transform domain. Due to the differences in multimodal medical imaging of imaging mechanisms, most transform domain methods apply multi-scale transform (MST) to obtain perceptually good results.

In general, the MST fusion method includes three steps: image multi-scale decomposition, image fusion, and image reconstruction. The MST method has been along with the ratio of Laplacian pyramid (LP), Low-Pass pyramid (RP), Wavelet transform (WT), Dual-tree complex wavelet transform (DTCWT), Curvelet transform (CVT), and nonsubsampled contourlet transform (NSCT). It was widely used. However, these MST methods are generally integrated into the base layer by applying a 'variance rule' or an 'averaging rule', which can completely convert the information contained in the source image into the final fusion image. none. In addition, such a base layer fusion rule has a problem of reducing the overall contrast of the fused image because a lot of energy included in the base layer is lost.

Accordingly, the present invention has been devised to solve the above problems, and an object of the present invention is to fuse medical images to improve a doctor's ability to diagnose and treat a disease, but LP (Laplacian Pyramid) and CSR (Convolutional Sparse) Representation) and fusion of medical images, etc., to provide an image fusion system that can completely extract texture (texture) detail information included in the source image and maintain the overall contrast of the final fusion image.

According to the image fusion system of the present invention, LP transformation is performed on a computed tomography image and a magnetic resonance image to obtain a base layer and a detail layer, and the detail layer is fused using a 'max-absolute' rule The base layer reconstructs the fused image by performing inverse LP transformation on the fused base layer and detail layer by fusion in a CSR-based approach to maintain the texture (texture) included in the source image.

First, to summarize the features of the present invention, the image fusion method in the image fusion system according to an aspect of the present invention for achieving the above object is a multi-scale level image through image decomposition of each image for two images. performing image transformation to calculate an image pyramid including generating a final fusion base layer (F _B ) by fusing a base layer having the smallest scale among the images of the multi-scale level of each image; generating a final fusion detail layer (F _D ) by fusing one or more detail layers among the images of the multi-scale level of each image; and outputting a final fusion image by reconstructing an image through an inverse process of image transformation with respect to the final fusion base layer (F _B ) and the final fusion detail layer (F _D ).

The two images include a CT image and an MRI image.

In the calculation of the image pyramid, a Gaussian pyramid transformation that generates an image pyramid for each scale level by using convolution and downsampling by the Gaussian kernel for each image, and the Gaussian pyramid transformation result in the Gaussian kernel LP transformation to generate an image pyramid for each scale level is performed using convolution and up-sampling by

The generating of the final fusion base layer (F _B ) includes the sparse coefficient vector (C _C,m ) of the first image and the sparseness of the second image through CSR coding coding for the base layers corresponding to each other of each image. calculating a coefficient vector (C _M,m ); For each sparse coefficient vector, each activity is calculated through L1 normalization, and the first activity average value and the second activity average value in a predetermined pixel window are respectively calculated, and the first activity average value and the first activity average value are calculated at each pixel position. determining a fusion sparse expression (C _F,M ) by selecting a sparse coefficient vector on the image side corresponding to the larger of the second activity average values; and generating the final fusion base layer (F _B ) through a convolution operation on the fusion sparse representation (C _F,M ) and a set of pre-trained pre-filters (d _m ). do.

을 만족하도록 수행되고, C_m은 C_C,m 또는 C_M,m이고, d_m은 미리 학습으로 훈련된 사전 필터, I_B 는 각 베이스 레이어, λ는 정규화 파라미터이다.In the CSR coding coding, each sparse coefficient vector is

is performed to satisfy , C _m is C _C,m or C _M,m , d _m is a pre-trained pre-filter, I _B is each base layer, and λ is a regularization parameter.

In the step of generating the final fusion detail layer (F _D ), for the detail layers corresponding to each other in each image, a larger value among the absolute values of both grayscale values of each pixel position is selected and determined as the grayscale value of each pixel position. , the final fusion detail layer F _D may be generated.

The final fusion detail layer F _D may be output by verifying whether the determined gradation value of each pixel position has a difference within a predetermined range from the gradation value of one or more adjacent pixels.

In addition, the recording medium on which the computer-readable code for performing the image fusion processing function in the image fusion system according to another aspect of the present invention is recorded is a multi-scale level of two images through image decomposition of each image. a function of performing image transformation to calculate an image pyramid including images; a function of generating a final fusion base layer (F _B ) by fusing a base layer having the smallest scale among the images of the multi-scale level of each image; a function of fusing one or more detail layers among the images of the multi-scale level of each image to generate a final fusion detail layer (F _D ); And with respect to the final fusion base layer (F _B ) and the final fusion detail layer (F _D ), a function of outputting a final fusion image is implemented by reconstructing an image through an inverse process of the image transformation.

According to the image fusion system according to the present invention, excellent fusion performance can be obtained through a new medical image fusion method combining LP and CSR called LP-CSR. LP transformation decomposes the CT and MRI source images into multi-scale representations, i.e., detail layers and base layers to extract features of the images, and converts the base layer into a CSR-based approach to fully preserve the texture detail information contained in the base layer. fuse The detail layer is fused according to the widely used 'max-absolute' rule, and suitability can be verified by comparing it with the grayscale values of neighboring pixels. Since the base layer is the minimum scale image of the LP, the computational burden of the CSR can be alleviated.

BRIEF DESCRIPTION OF THE DRAWINGS The accompanying drawings, which are included as part of the detailed description to help the understanding of the present invention, provide an embodiment of the present invention and together with the detailed description, explain the technical spirit of the present invention.
1 is a view for explaining an image fusion method in the image fusion system of the present invention.
2 is a view for explaining a method of generating an image pyramid by image decomposition in the image fusion system of the present invention.
3 is a view for explaining a method of decomposing an image by LP transformation in the image fusion system of the present invention.
4 is a view for explaining an image fusion and reconstruction method of a base layer and a detail layer in the image fusion system of the present invention.
5 shows examples of the results (J) of the image fusion method of the present invention and the results (C ~ I) of the other conventional image fusion methods with respect to the input CT image (A) and the MRI (B) image.
6 is a diagram for explaining an example of a method for implementing an image fusion system according to an embodiment of the present invention.

Hereinafter, the present invention will be described in detail with reference to the accompanying drawings. In this case, the same components in each drawing are denoted by the same reference numerals as much as possible. In addition, detailed descriptions of already known functions and/or configurations will be omitted. The content disclosed below will focus on parts necessary to understand operations according to various embodiments, and descriptions of elements that may obscure the gist of the description will be omitted. Also, some components in the drawings may be exaggerated, omitted, or schematically illustrated. The size of each component does not fully reflect the actual size, so the contents described herein are not limited by the relative size or spacing of the components drawn in each drawing.

In describing the embodiments of the present invention, if it is determined that the detailed description of the known technology related to the present invention may unnecessarily obscure the gist of the present invention, the detailed description thereof will be omitted. And, the terms to be described later are terms defined in consideration of functions in the present invention, which may vary according to intentions or customs of users and operators. Therefore, the definition should be made based on the content throughout this specification. The terminology used in the detailed description is for the purpose of describing embodiments of the present invention only, and should not be limiting in any way. Unless explicitly used otherwise, expressions in the singular include the meaning of the plural. In this description, expressions such as “comprising” or “comprising” are intended to indicate certain features, numbers, steps, acts, elements, some or a combination thereof, one or more other than those described. It should not be construed to exclude the presence or possibility of other features, numbers, steps, acts, elements, or any part or combination thereof.

In addition, terms such as first, second, etc. may be used to describe various components, but the components are not limited by the terms, and the terms are for the purpose of distinguishing one component from other components. is used only as

Recently, edge-preserving filters have been successfully used for image fusion. Li et al. used a guided filter (GF) to fuse images and achieve improved performance. Zhou et al. proposed a hybrid multi-scale decomposition (Hybrid-MSD) fusion method by combining a Gaussian filter and a bidirectional filter, but this fusion method is not suitable for medical imaging. Jian et al. obtained objective good performance by fusing multimodal medical images and applying a rolling guidance filter (RGF). However, fusion images obtained by RGF can cause many afterimages. Recently, Ma et al. proposed a new image fusion method called radiant transfer fusion (GTF), which formulated image transcription as a minimization problem. However, the GTF method can only be used for fusion of infrared and visible images and is not robust to other scene images. In recent years, sparse representation (SR) has been widely used for image processing tasks such as image classification and image recognition. SR was first introduced in the image fusion of Yang and Li. Here, the original image is transformed into the SR domain and expressed as a sparse vector, the maximum L1 standard is selected as a fusion rule, and the final fusion image is reconstructed by implementing an inverse sparse transform on the fusion sparse vector. To overcome the shortcomings of the SR-based fusion method, Liu et al. proposed an adaptive sparse representation (ASR) model that fuses images.

In particular, the ASR fusion method has high redundancy and replaces the completed dictionary with a sub-dictionary set for reconstructing the final fused image. Convolutional sparse representation (CSR) is applied to image fusion to overcome the shortcomings of the local patch method in the SR and ASR fusion methods that lose texture detail information. However, the image fusion CSR-based method in the study of Liu et al. does not achieve satisfactory fusion performance and contains few multi-scale transform (MST) structures. In addition, the processing speed of the CSR-based method is not satisfactory for all CSR tasks performed in a large-scale image space.

1 is a view for explaining an image fusion method in the image fusion system of the present invention.

1 , the image fusion system of the present invention generates an image pyramid including images of multiple scale levels through image decomposition using LP transformation for two different images (eg, CT image and MRI image). and (S110), the base layer (image) having the smallest scale among the images of the multi-scale level of each image is fused with each other (using CSR coding) to generate a final fusion base layer F _B , and the multiple The final fusion detail layer F _D is generated by fusing (using the max-absolute rule) one or more detail layers (images) among the images of the scale level (S120), and the final fusion base layer F _B and the final fusion detail layer F _D , by reconstructing the image through the reverse process of the above LP transformation, the final fusion image 510 is output (S130).

In order to solve the shortcomings of the conventional method, the present invention was inspired by the recent image fusion literature, and proposed a new medical image fusion method combining LP and CSR called LP-CSR. To extract medical image features, LP transformation is applied to decompose the CT and MRI source images into images of multiple scale levels, ie, a detail layer and a base layer. We fuse the base layer with a CSR-based approach to fully preserve the texture detail information contained in the base layer. In the existing SR, the fused image uses a pixel-by-pixel 'overlap-averaging' strategy to aggregate all reconstructed patches into the entire image, so texture detail information is easily lost. CSR can be considered as a convolutional SR, which obtains sparse coefficients of the entire image rather than a local image patch. The fusion result of CSR is optimized for the entire image without a pixel-by-pixel 'duplicate averaging' strategy, so texture detail information can be perfectly preserved. Therefore, the present invention can obtain better fusion performance than the conventional SR. In addition, since the base layer is an image of the minimum scale (the smallest number of horizontal and vertical pixels) of the LP, it is possible to alleviate the computational burden of the CSR. The suitability of the detail layer is verified by comparing it with the grayscale values of pixels around the widely used 'max-absolute' rule. Finally, the fused image can be reconstructed by performing inverse LP transformation on the fused detail and base layers.

2 is a view for explaining a method of generating an image pyramid by image decomposition in the image fusion system of the present invention.

Referring to FIG. 2 , the image pyramid generation by image decomposition using the LP transformation in step S110 is a process for maintaining the high frequency component of the image without losing, in the first step, the Gauss for the input image I _input . GP (Gaussian pyramid) transformation to generate an image pyramid by scale level (lowest 1 ^st ,.., highest Kth) using convolution (conv) and downsampling (f _↓ ) by Shan kernel, and In the second step, the LP (lowest 1 ^st ,.., highest Kth) image pyramid is generated by scale level using convolution (conv) and up-sampling (f _↑ ) by Gaussian kernel for the GP transformation result. Laplacian Pyramid (Laplacian Pyramid) transformation.

In GP transformation, as in [Equation 1], the input image (I _input ) is the layer (image) G ₁ of the lowest ^1st level, and n=2,..,K (K is a natural number greater than or equal to 2) up to the highest Kth level ), calculate the layer (image) G _n of each level. Calculation of G _n is, as in [Equation 1], convolution (conv) by a Gaussian kernel is performed on a lower-level layer (G _n-1 ) as shown in Equation 1, but down-sampling (f _↓ ) is performed to scale the image ( size) is reduced by 1/2 (see patent application KR10-2010-0036388). For example, the previous image scale (pixel size) m*n may be reduced to 0.5m*0.5n at each level of G _n .

[Equation 1]

G ₁ = I _input

G _n = f _↓ (conv(G _n-1 ))

LP conversion is, as in [Equation 2], from the highest ^Kth level layer (image) to the lowest 1st level layer (image) for each level for n = 1,2,..,K (K is a natural number greater than or equal to 2) Calculate the layer (image) LP _n of Calculation of LP _n is, as in [Equation 2], convolution (conv) by Gaussian kernel is performed on the upper-level layer (G _n+1 ), but up-sampling (f _↑ ) is processed to scale the image. Including the process of magnifying (size) twice (pixel size m*n->2m*2n), the difference between the results is calculated sequentially in G _n .

[Equation 2]

LP _n = G _n - f _↑ conv(G _n+1 )

3 is a diagram for explaining the calculation of an image pyramid using LP transformation in the image fusion system of the present invention.

Referring to FIG. 3 , the image fusion system of the present invention calculates an image pyramid including images of multiple scale levels using LP transformation in step S110 above, but with respect to a CT image and an MRI image, each of the multiple scale levels Create images (I _B , I _D ¹ ,.., I _D ⁱ ,.., I _D ^K-1 ) of Here, the base layer of the lowest scale level is denoted by I _B , and detail layers of other scale levels are denoted by I _D ⁱ .

According to the image pyramid creation by image decomposition using LP transformation of the present invention, the image pyramid LP _C (I _input , K) by image decomposition using LP transformation for CT image can be expressed as [Equation 3], , the image pyramid LP _M (I _input , K) by image decomposition using LP transformation for the MRI image can be expressed as [Equation 4].

Here, I _B ^C and I _B ^M represent a base layer of a CT image and a base layer of an MRI image, respectively. In addition, the detail layers I _DC ⁱ and I _DM ⁱ represent detail layers of the CT image and the MRI image, respectively.

[Equation 3]

LP _C (I _input , K) = (I _B ^C , I _DC ¹ ,.., I _DC ⁱ ,.., I _DC ^K-1 )

[Equation 4]

LP _M (I _input , K) = (I _B ^M , I _DM ¹ ,.., I _DM ⁱ ,.., I _DM ^K-1 )

4 is a view for explaining an image fusion and reconstruction method of a base layer and a detail layer in the image fusion system of the present invention.

Referring to FIG. 4 , the image fusion system of the present invention performs fusion of a base layer of two images and fusion of a detail layer of two images in step S120 above.

To this end, in the image fusion system of the present invention, according to the result performed in step S110 above, for fusion of the base layer, images (I _B ^C , I _DC ¹ , .., I _DC ⁱ ,.., I _DC ^K-1 ) of the base layer I _B ^C 421 of the CT image of the highest scale level, and the image (I _B ^M , I _DM ¹ ,.., I _DM ⁱ ,.., I _DM ^K-1 ) receive the base layer I _B ^M 422 of the MRI image of the highest scale level.

For each image, the image fusion system of the present invention is CSR coding (Wohlberg B, Efficient Algorithms for convolutional sparse representations. IEEE Trans Image Process. 2015;25(1)) so that the following [Equation 5] is satisfied: 301-315) through (431, 432), CSR sparse coefficient vector C _m (sparse coefficient vector (map) C _C,m of CT image, sparse coefficient vector of MRI image) for each pixel position in CT image and MRI image. (map) C _M,m ) is calculated (441, 442). Here, C _m is a column vector of the corresponding dimension with m=1,2,..,M.

[Equation 5]

The sparse coefficient C _m (C _C,m , C _M,m ) representing the m-dimensional column vector for each pixel position (x,y) of the base layer of each image is pre-learned so as to satisfy [Equation 5]. _{Convolution operation} of _{a set} of ^dictionary filters _{d m} ^trained _with *) is obtained through Here, λ is a normalization parameter. The CSR coding result for fusing the base layers maintains the texture detail information of the base layers.

와 같이 산출하되(C_1:M(x,y)은 C_m(x,y)의 m=1,2,..,M인 해당 차원 칼럼 벡터의 원소값들), [수학식6]과 같이 소정의 픽셀윈도우 r을 적용해 평균값 /A(x,y)(C_C,m에 대한 /A_C(x,y), C_M,m에 대한 /A_M(x,y))를 산출함으로써, 서로 다른 영상의 융합에 의한 부적합성을 해소할 수 있도록 한다(450). 윈도우 r이 커지면 영상의 융합에 의한 적합도가 떨어지므로 적절한 값이 선택될 수 있다(예, r=3).Next, the image fusion system of the present invention, for each CSR sparse coefficient vector C _m (C _C,m , C _M,m ), through L1 normalization (L1 Norm), the activity according to the sparse expression fusion method )(map) A(x, y)

(C _1:M (x,y) is the element values of the corresponding dimension column vector where m=1,2,..,M of C _m (x,y)), [Equation 6] and Similarly, the average value /A(x,y) (/A _C (x,y) for C _C ,m, /A _M (x,y) for C _M,m ) is calculated by applying a predetermined pixel window r By doing so, incompatibility due to the fusion of different images can be resolved ( 450 ). As the window r increases, the fit due to image fusion decreases, so an appropriate value may be selected (eg, r=3).

[Equation 6]

Next, the image fusion system of the present invention is fused as in [Equation 7] using /A _C (x,y) for C _C ,m and /A _M (x,y) for C _M,m Yields the sparse expression C _F:1:M (x,y) (460). In other words, at each pixel position (x,y), the fusion sparseness coefficient is selected by selecting the larger of the CT image activity /A _C (x,y) and the MRI image activity /A _M (x,y). Returns the element values of the expression C _F:1:M (x,y). C _C,1:M (x,y) is the element values of the M-dimensional column vector from m=1 of C _C,m (x,y), C _M,1:M (x,y) is C _M, These are the element values of the M-dimensional column vector from m=1 of _m (x,y).

[Equation 7]

The image fusion system of the present invention is trained by pre-learning and a fusion sparse expression (C _F,M ) consisting of element values of C _F:1:M (x,y) above, as in [Equation 8]. A final fusion base layer F _B is obtained (470) through a convolution operation (*) for the set d _m (461) of the pre-filters.

[Equation 8]

In addition, in the image fusion system of the present invention, images (I _B ^C , I _DC ¹ ,..., I _DC ⁱ ,.., I _DC ^K-1 ) of the detail layers I _DC ⁱ (481), and the image (I _B ^M , I _DM ¹ ,.., I _DM ⁱ , .., I _DM ^K-1 ) of detail layers I _DM ⁱ 482 are received.

In addition, the image fusion system of the present invention processes the detail layers 481 and 482 I _D ⁱ of different scales of the CT and MRI images by matching the same scale level with each other, but in the detail layer of the same scale level. According to the 'max-absolute' rule, the largest value among the absolute values of the gray scale values on both sides of the corresponding pixel position (x, y) is selected (490) and determined as the gray scale value of each pixel position (x, y). A fusion detail layer F _D is output (500).

In this case, the image fusion system of the present invention checks whether the determined grayscale value of each pixel has a similar value to the grayscale values of neighboring pixel positions in order to verify suitability according to the fusion of different images at each scale level. can In the image fusion system of the present invention, if the grayscale value of each pixel determined above is different from the grayscale value of one or more adjacent pixels (top, bottom, left, right) within a predetermined range according to the suitability verification algorithm, it is determined as the final fusion detail layers at the corresponding scale level. can be printed out. In this case, if it is not suitable, an error may be displayed through the user interface, and the predetermined range may be adjusted to be performed.

Through this process, in step S130 above, the image fusion system of the present invention reconstructs an image through the reverse process of the above LP transformation with respect to the final fusion base layer F _B and the final fusion detail layer F _D By doing so, the final fusion image 510 may be output. For example, the final fusion image 510 can be output by reconstructing the image so that the image pyramid including images of multiple scale levels becomes a synthesized image reflecting the characteristic information of each level through inverse LP transformation. have. That is, by providing a high-resolution final fusion image 510 from which complementary information of CT and MRI images is taken, information on dense organs and tissues such as bones and implants by CT image, and information on soft tissue by MRI image It is possible to provide a fused, high-resolution final fusion image 510 .

5 shows examples of the results (J) of the image fusion method of the present invention and the results (C ~ I) of the other conventional image fusion methods with respect to the input CT image (A) and the MRI (B) image.

Referring to FIG. 5 , the result (J) of the image fusion method of the present invention is obtained by fusion of CT image (A) and MRI (B) image to converge information on dense organs and tissues, and information on soft tissue. It was confirmed that high-resolution final fusion images can be provided. The result (J) of the image fusion method of the present invention is more clear by fusion of information on dense organs and tissues and information on soft tissues, compared with results (C ~ I) of other conventional image fusion methods. was found to be able to provide.

6 is a diagram for explaining an example of a method for implementing an image fusion system according to an embodiment of the present invention.

The image fusion system for image fusion processing according to an embodiment of the present invention as described above may be formed of hardware, software, or a combination thereof. For example, the image fusion system of the present invention may be implemented in the form of a computing system 1000 as shown in FIG. 5 having at least one processor for performing the above functions/steps/processes or a server on the Internet.

The computing system 1000 includes at least one processor 1100 , a memory 1300 , a user interface input device 1400 , a user interface output device 1500 , a storage 1600 connected through a bus 1200 , and a network An interface 1700 may be included. The processor 1100 may be a central processing unit (CPU) or a semiconductor device that processes instructions stored in the memory 1300 and/or the storage 1600 . The memory 1300 and the storage 1600 may include various types of volatile or nonvolatile storage media. For example, the memory 1300 may include a read only memory (ROM) 1310 and a random access memory (RAM) 1320 .

Accordingly, steps of a method or algorithm described in connection with the embodiments disclosed herein may be directly implemented in hardware, a software module executed by the processor 1100 , or a combination of the two. A software module may be a storage/recording medium (i.e., memory 1300 and/or memory 1300) readable by a device, such as a computer, such as RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, removable disk, CD-ROM. Alternatively, it may reside in storage 1600 . An exemplary storage medium is coupled to the processor 1100 , the processor 1100 capable of reading information (code) from, and writing information (code) to, the storage medium. Alternatively, the storage medium may be integrated with the processor 1100 . The processor and storage medium may reside within an application specific integrated circuit (ASIC). The ASIC may reside within the user terminal. Alternatively, the processor and storage medium may reside as separate components within the user terminal.

As described above, according to the image fusion system according to the present invention, excellent fusion performance can be obtained through a new medical image fusion method combining LP and CSR called LP-CSR. LP transform decomposes the CT and MRI source images into multi-scale representations, i.e., detail layers and base layers to extract features of the image, and converts the base layer into a CSR-based approach to fully preserve the texture detail information contained in the base layer. fuse The detail layer is fused according to the widely used 'max-absolute' rule, and suitability can be verified by comparing it with the grayscale values of neighboring pixels. Since the base layer is the minimum scale image of the LP, it is possible to alleviate the computational burden of the CSR.

As described above, in the present invention, specific matters such as specific components, etc., and limited embodiments and drawings have been described, but these are only provided to help a more general understanding of the present invention, and the present invention is not limited to the above embodiments. , various modifications and variations will be possible without departing from the essential characteristics of the present invention by those of ordinary skill in the art to which the present invention pertains. Therefore, the spirit of the present invention should not be limited to the described embodiments, and all technical ideas with equivalent or equivalent modifications to the claims as well as the claims to be described later are included in the scope of the present invention. should be interpreted as

Claims (14)

An image fusion method in an image fusion system, comprising:
performing image conversion on two images to calculate an image pyramid including images of multiple scale levels through image decomposition of each image;
generating a final fusion base layer (F _B ) by fusing a base layer having the smallest scale among the images of the multi-scale level of each image;
generating a final fusion detail layer (F _D ) by fusing one or more detail layers among the images of the multi-scale level of each image; and
For the final fusion base layer (F _B ) and the final fusion detail layer (F _D ), outputting a final fusion image by reconstructing an image through an inverse process of the image transformation
An image fusion method comprising a. The method of claim 1,
The two images are an image fusion method comprising a CT image and an MRI image. The method of claim 1,
In the calculation of the image pyramid, a Gaussian pyramid transformation that generates an image pyramid for each scale level by using convolution and downsampling by the Gaussian kernel for each image, and the Gaussian pyramid transformation result in the Gaussian kernel An image fusion method that performs LP transformation to generate an image pyramid for each scale level using convolution and up-sampling by The method of claim 1,
The step of creating the final fusion base layer (F _B ),
calculating a sparse coefficient vector (C _C,m ) of a first image and a sparse coefficient vector (C _M,m ) of a second image through CSR coding coding for the base layers corresponding to each other;
For each sparse coefficient vector, each activity is calculated through L1 normalization, and the first activity average value and the second activity average value in a predetermined pixel window are respectively calculated, and the first activity average value and the first activity average value are calculated at each pixel position. determining a fusion sparse expression (C _F,M ) by selecting a sparse coefficient vector on the image side corresponding to the larger of the second activity average values; and
generating the final fusion base layer (F _B ) through a convolution operation on the fusion sparse representation (C _F,M ) and a set of pre-trained pre-filters (d _m )
An image fusion method comprising a. 5. The method of claim 4,
In the CSR coding coding, each sparse coefficient vector is

An image fusion method performed to satisfy , C _m is C _C,m or C _M,m , d _m is a pre-trained pre-filter, I _B is each base layer, and λ is a regularization parameter. The method of claim 1,
In the step of generating the final fusion detail layer (F _D ), for the detail layers corresponding to each other in each image, a larger value among the absolute values of both grayscale values of each pixel position is selected and determined as the grayscale value of each pixel position. , an image fusion method for generating the final fusion detail layer (F _D ). 7. The method of claim 6,
The image fusion method of outputting the final fusion detail layer (F _D ) by verifying whether the determined gradation value of each pixel position has a difference within a predetermined range from the gradation value of one or more adjacent pixels. A recording medium recording a computer-readable code for performing an image fusion processing function in an image fusion system, the recording medium comprising:
a function of performing image conversion on two images to calculate an image pyramid including images of multiple scale levels through image decomposition of each image;
a function of generating a final fusion base layer (F _B ) by fusing a base layer having the smallest scale among the images of the multi-scale level of each image;
a function of fusing one or more detail layers among the images of the multi-scale level of each image to generate a final fusion detail layer (F _D ); and
With respect to the final fusion base layer (F _B ) and the final fusion detail layer (F _D ), a function of outputting a final fusion image by reconstructing an image through an inverse process of the image transformation
recording medium for implementing the 9. The method of claim 8,
The two images are a recording medium including a CT image and an MRI image. 9. The method of claim 8,
In the function of calculating the image pyramid, a Gaussian pyramid transformation for generating an image pyramid for each scale level by using convolution and downsampling by a Gaussian kernel for each image, and a Gaussian pyramid transformation for the result of the Gaussian pyramid transformation A recording medium that performs LP transformation to generate an image pyramid for each scale level using convolution by a kernel and up-sampling. 9. The method of claim 8,
The function of creating the final fusion base layer (F _B ) is,
a function of calculating a sparse coefficient vector (C _C,m ) of a first image and a sparse coefficient vector (C _M,m ) of a second image through CSR coding coding for the base layers corresponding to each other;
For each sparse coefficient vector, each activity is calculated through L1 normalization, and the first activity average value and the second activity average value in a predetermined pixel window are respectively calculated, and the first activity average value and the first activity average value are calculated at each pixel position. a function of determining a fusion sparse expression (C _F,M ) by selecting a sparse coefficient vector on the image side corresponding to the larger of the second activity average values; and
The function of generating the final fusion base layer (F _B ) through a convolution operation on the fusion sparse representation (C _F,M ) and a set of pre-trained pre-filters (d _m )
A recording medium comprising a. 12. The method of claim 11,
In the CSR coding coding, each sparse coefficient vector is

is performed to satisfy , C _m is C _C,m or C _M,m , d _m is a pre-filter trained in pre-training, I _B is each base layer,

is a normalization parameter. 9. The method of claim 8,
In the function of generating the final fusion detail layer (F _D ), for the detail layers corresponding to each other in each image, by selecting the larger value among the absolute values of the grayscale values on both sides of each pixel position and determining it as the grayscale value of each pixel position , a recording medium that creates the final fusion detail layer (F _D ). 14. The method of claim 13,
and outputting the final fusion detail layer (F _D ) by verifying whether the determined gradation value of each pixel position has a difference within a predetermined range from the gradation value of one or more adjacent pixels.

Images