CN112163994B - Multi-scale medical image fusion method based on convolutional neural network - Google Patents

Multi-scale medical image fusion method based on convolutional neural network Download PDF

Info

Publication number
CN112163994B
CN112163994B CN202010904222.1A CN202010904222A CN112163994B CN 112163994 B CN112163994 B CN 112163994B CN 202010904222 A CN202010904222 A CN 202010904222A CN 112163994 B CN112163994 B CN 112163994B
Authority
CN
China
Prior art keywords
image
scale
neural network
anatomical
fusion
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202010904222.1A
Other languages
Chinese (zh)
Other versions
CN112163994A (en
Inventor
李伟生
黄渝萍
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Chongqing University of Post and Telecommunications
Original Assignee
Chongqing University of Post and Telecommunications
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Chongqing University of Post and Telecommunications filed Critical Chongqing University of Post and Telecommunications
Priority to CN202010904222.1A priority Critical patent/CN112163994B/en
Publication of CN112163994A publication Critical patent/CN112163994A/en
Application granted granted Critical
Publication of CN112163994B publication Critical patent/CN112163994B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
    • G06T3/4053—Scaling of whole images or parts thereof, e.g. expanding or contracting based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/50—Image enhancement or restoration using two or more images, e.g. averaging or subtraction
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10072—Tomographic images
    • G06T2207/10088—Magnetic resonance imaging [MRI]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10072—Tomographic images
    • G06T2207/10104—Positron emission tomography [PET]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10072—Tomographic images
    • G06T2207/10108—Single photon emission computed tomography [SPECT]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20024—Filtering details
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20212—Image combination
    • G06T2207/20221—Image fusion; Image merging

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • General Health & Medical Sciences (AREA)
  • Molecular Biology (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • Data Mining & Analysis (AREA)
  • Evolutionary Computation (AREA)
  • Artificial Intelligence (AREA)
  • Biomedical Technology (AREA)
  • Computing Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Health & Medical Sciences (AREA)
  • Image Processing (AREA)
  • Image Analysis (AREA)

Abstract

The invention requests to protect a multi-scale medical image fusion method based on a convolutional neural network, which comprises the following steps: s1, local Laplacian filtering processing is carried out on the anatomical line image and the functional image after the registration, and the anatomical line image and the functional image are decomposed into a multi-scale approximate image with enhanced details and a residual image; s2, inputting the registered anatomical image into a depth convolution neural network to extract a super-resolution anatomical image; s3, inputting the super-resolution anatomical image and the functional image into a double-branch convolution neural network for convolution to obtain a weight map, and combining a multi-scale approximate image and a residual image to form a multi-scale fusion image; s4 reconstructs the multi-scale fusion image using the inverse of local laplace. The invention effectively solves the problems of color distortion, information loss and the like when the pseudo-color image and the gray level image are fused by the medical image fusion method.

Description

Multi-scale medical image fusion method based on convolutional neural network
Technical Field
The invention belongs to the technical field of image processing, and particularly relates to a multi-scale medical image fusion method based on a convolutional neural network.
Background
The medical image fusion method belongs to the field of computer vision and has wide application in medical image, clinical diagnosis and other fields. Medical image fusion methods are mainly classified into fusion methods based on a single scale and fusion methods based on multiple scales.
Compared with a single-scale pixel-level medical image fusion method, the pixel-level multi-scale medical image fusion method can effectively improve the quality of the fusion image by extracting the features of the image pixel values on the sub-band images with different scales. In the traditional multi-scale fusion method, part of detail information is lost in convolution and down-sampling operations based on pyramid transformation of a Gaussian pyramid, and direction information cannot be captured. The fusion is performed by using wavelet transform and complex wavelet transform, and provides direction information in the decomposition process. However, the wavelet transform-based method is limited by blurring of the fused image. Fusion methods based on parallelepiped transformation, such as contourlet transformation, non-subsampled shear wave, shear transformation, etc., have been proposed. The focus of these fusion methods is to design filters to extract more detailed information. Therefore, these methods require higher computational complexity to optimize the parameters, reducing their efficiency.
In recent years, deep learning has achieved excellent results in medical image fusion. But only rely on deep learning to carry out end-to-end image fusion, it is difficult to retain the detail information, color information and brightness information of the source image at the same time satisfactorily. And artifacts are easy to generate, so that the advantages of the traditional algorithm and deep learning are combined, the medical image fusion is completed by utilizing the advantages of the traditional algorithm and the deep learning, a satisfactory fusion effect can be achieved, and a high-quality fusion image can be obtained.
Although the fusion method is many, many challenges still exist in practical scenes, such as noise influence, quality of the image to be fused, color distortion, image artifact problem, and the like. Although the current fusion method based on the convolutional neural network is greatly improved in retaining texture and color information, the fusion method still has certain problems for real-time performance due to the participation of a local Laplace algorithm.
Disclosure of Invention
The present invention is directed to solving the above problems of the prior art. A multi-scale medical image fusion method based on a convolutional neural network is provided. The technical scheme of the invention is as follows:
a multi-scale medical image fusion method based on a convolutional neural network comprises the following steps;
s1, local Laplacian filtering processing is carried out on the anatomical image (MRI image) and the functional image (PET/SPECT image), and the images are decomposed into a detail-enhanced multi-scale approximate image and a residual image;
s2, inputting the anatomical image into a depth convolution neural network with 20 layers to extract a super-resolution anatomical image;
s3, inputting the super-resolution anatomical image and the functional image into a double-branch convolution neural network for convolution to obtain a weight map, and combining a multi-scale approximate image and a residual image to fuse the weight map into a multi-scale fusion image;
and S4, reconstructing the multi-scale fusion image by adopting the inverse operation of local Laplace.
Further, the image decomposition in step S1 uses a local laplacian filter as a tool, and outputs only an anatomical image (a) and a functional image (B) of three different scales, where (a) and (B) are respectively provided1，B1)、(A2，B2)、(A3，B3) The sizes are respectively as follows: 256 × 256, 128 × 128, 64 × 64.
Further, the super-resolution image S of S2 is directly obtained by superimposing the feature map F extracted by the depth convolution neural network VDSR and the residual image R in the convolution process, and the calculation formula is as follows:
S＝F+R。
further, the weight map of step S3 is obtained by inputting the registered super-resolution anatomical image and source functional image into a two-branch convolutional neural network, and the weight map has the same size as the source image, and its pixel point is between 0 and 1, and represents the probability of selecting the pixel value of the point; the multi-scale fusion image of step S3 is obtained by weight distribution of the decomposed multi-scale image and the weight map after gaussian decomposition.
Further, in step S4, the fused image of multiple scales is restored to the fused image of the original size by using the inverse operation of the local laplacian.
The invention has the following advantages and beneficial effects:
the invention uses the deep convolution neural network to carry out image enhancement on the medical image and inputs the medical image into the double-branch convolution neural network to generate the weight map, so that the obtained weight map has better robustness and is more beneficial to pixel distribution, then, the local Laplace filter is used for carrying out multi-scale decomposition on the image, and the filter can enhance the image in the decomposition process, so that the image is prevented from losing gradient information and brightness in the decomposition process, texture and edge information in a source image can be better reserved, a weight graph and a decomposed image are used for fusion, then local Laplace inverse operation is used for reconstruction, sub-images with multiple scales are completely restored to original scales, and finally the method can well reserve details, texture information and color information in the source image to obtain a fusion image which is more beneficial to clinical diagnosis.
Drawings
FIG. 1 is a flow chart of multi-scale medical image fusion based on a convolutional neural network according to a preferred embodiment of the present invention;
table 1 shows the results compared to other mainstream methods.
Detailed Description
The technical solutions in the embodiments of the present invention will be described in detail and clearly with reference to the accompanying drawings. The described embodiments are only some of the embodiments of the present invention.
The technical scheme for solving the technical problems is as follows:
as shown in fig. 1, a convolution neural network-based multi-scale medical image fusion method includes the following steps:
s1, local Laplacian filtering processing is carried out on the anatomical line image and the functional image after the registration, and the anatomical line image and the functional image are decomposed into a multi-scale approximate image with enhanced details and a residual image;
s2, inputting the registered anatomical image into a depth convolution neural network to extract a super-resolution anatomical image;
s3, inputting the super-resolution anatomical image and the functional image into a double-branch Convolutional Neural Network (CNN) for convolution to obtain a weight map, and combining the multi-scale approximate image and the residual image to form a multi-scale fusion image;
and S4, reconstructing the multi-scale fusion image by adopting the inverse operation of local Laplace.
Further, the image decomposition in step S1 uses a local laplacian filter as a tool, and outputs only an anatomical image (a) and a functional image (B) of three different scales, where (a) and (B) are respectively provided1，B1)、(A2，B2)、(A3，B3) The sizes are respectively as follows: 256 × 256, 128 × 128, 64 × 64.
Further, the method for fusing multi-scale medical images based on the convolutional neural network is characterized in that the super-resolution image (S) of S2 is obtained by directly superimposing a feature map (F) extracted by a deep convolutional neural network (VDSR) and a residual image (R) in a convolution process, and a calculation formula is as follows:
S＝F+R
further, the weight map of S3 is obtained by inputting the registered super-resolution anatomical image and source functional image into a two-branch Convolutional Neural Network (CNN); and the multi-scale fusion image in the third step is obtained by carrying out weight distribution on the decomposed multi-scale image and the decomposed weight map.
Further, the final fusion result of S4 is obtained by restoring the fusion images of multiple scales to the original size fusion image by using the inverse operation of local laplacian.
Further, Table 1 shows the results of comparison with other mainstream methods
TABLE 1
Figure BDA0002660819590000041
Figure BDA0002660819590000051
It should also be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.
The above examples are to be construed as merely illustrative and not limitative of the remainder of the disclosure. After reading the description of the invention, the skilled person can make various changes or modifications to the invention, and these equivalent changes and modifications also fall into the scope of the invention defined by the claims.

Claims (1)

1. A multi-scale medical image fusion method based on a convolutional neural network is characterized by comprising the following steps;
s1, local Laplacian filtering processing is carried out on the anatomical image and the functional image, and the anatomical image and the functional image are decomposed into a multi-scale approximate image with enhanced details and a residual image, wherein the anatomical image is an MRI image, and the functional image is a PET/SPECT image;
s2, inputting the anatomical image into a depth convolution neural network with 20 layers to extract a super-resolution anatomical image;
s3, inputting the super-resolution anatomical image and the functional image into a double-branch convolution neural network for convolution to obtain a weight map, and combining a multi-scale approximate image and a residual image to fuse the weight map into a multi-scale fusion image;
s4, reconstructing the multi-scale fusion image by adopting the inverse operation of local Laplace;
said step (c) isThe image decomposition at S1 uses a local laplacian filter as a tool, and outputs only an anatomical image (a) and a functional image (B) at three different scales, where (a) and (B) are the respective images1，B1)、(A2，B2)、(A3，B3) The sizes are respectively as follows: 256 × 256, 128 × 128, 64 × 64;
the super-resolution image S of S2 is directly obtained by superposing a feature image F extracted by a depth convolution neural network and a residual image R in the convolution process, and the calculation formula is as follows:
S＝F+R；
the weight map of the step S3 is obtained by inputting the registered super-resolution anatomical image and source functional image into a two-branch convolutional neural network, the weight map is the same as the source image in size, and the pixel point of the weight map is between 0 and 1, and represents the probability of selecting the pixel value of the point; the multi-scale fusion image of the step S3 is obtained by performing weight distribution on the decomposed multi-scale image and the weight map after gaussian decomposition;
in step S4, the fused image of multiple scales is restored to the original size fused image by using the inverse operation of local laplacian.
CN202010904222.1A 2020-09-01 2020-09-01 Multi-scale medical image fusion method based on convolutional neural network Active CN112163994B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202010904222.1A CN112163994B (en) 2020-09-01 2020-09-01 Multi-scale medical image fusion method based on convolutional neural network

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202010904222.1A CN112163994B (en) 2020-09-01 2020-09-01 Multi-scale medical image fusion method based on convolutional neural network

Publications (2)

Publication Number Publication Date
CN112163994A CN112163994A (en) 2021-01-01
CN112163994B true CN112163994B (en) 2022-07-01

Family

ID=73857371

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202010904222.1A Active CN112163994B (en) 2020-09-01 2020-09-01 Multi-scale medical image fusion method based on convolutional neural network

Country Status (1)

Country Link
CN (1) CN112163994B (en)

Families Citing this family (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113128583B (en) * 2021-04-15 2022-08-23 重庆邮电大学 Medical image fusion method and medium based on multi-scale mechanism and residual attention
CN113129244B (en) * 2021-04-16 2022-10-18 重庆邮电大学 Multi-scale medical image fusion method based on local Laplace decomposition
CN113610759A (en) * 2021-07-05 2021-11-05 金华电力设计院有限公司 A on-spot safe management and control system for roofbolter construction
CN115171882B (en) * 2022-07-07 2023-06-02 广东工业大学 Intelligent medical auxiliary diagnosis method and system based on Y-type network with multiple priori embedding
CN115810139B (en) * 2022-12-16 2023-09-01 西北民族大学 Target area identification method and system for SPECT image

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101425137A (en) * 2008-11-10 2009-05-06 北方工业大学 Face Image Fusion Method Based on Laplacian Pyramid
CN104318525A (en) * 2014-10-17 2015-01-28 合肥工业大学 Space guiding filtering based image detail enhancement method
CN104517264A (en) * 2013-09-30 2015-04-15 华为终端有限公司 Image processing method and device
CN106981059A (en) * 2017-03-30 2017-07-25 中国矿业大学 With reference to PCNN and the two-dimensional empirical mode decomposition image interfusion method of compressed sensing
CN110415198A (en) * 2019-07-16 2019-11-05 南京信息工程大学 A kind of Method of Medical Image Fusion based on laplacian pyramid Yu parameter adaptive Pulse Coupled Neural Network

Family Cites Families (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7970727B2 (en) * 2007-11-16 2011-06-28 Microsoft Corporation Method for modeling data structures by creating digraphs through contexual distances
US8768069B2 (en) * 2011-02-24 2014-07-01 Sony Corporation Image enhancement apparatus and method

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101425137A (en) * 2008-11-10 2009-05-06 北方工业大学 Face Image Fusion Method Based on Laplacian Pyramid
CN104517264A (en) * 2013-09-30 2015-04-15 华为终端有限公司 Image processing method and device
CN104318525A (en) * 2014-10-17 2015-01-28 合肥工业大学 Space guiding filtering based image detail enhancement method
CN106981059A (en) * 2017-03-30 2017-07-25 中国矿业大学 With reference to PCNN and the two-dimensional empirical mode decomposition image interfusion method of compressed sensing
CN110415198A (en) * 2019-07-16 2019-11-05 南京信息工程大学 A kind of Method of Medical Image Fusion based on laplacian pyramid Yu parameter adaptive Pulse Coupled Neural Network

Non-Patent Citations (6)

* Cited by examiner, † Cited by third party
Title
A medical image fusion method based on convolutional neural networks;Yu Liu等;《2017 20th International Conference on Information Fusion (Fusion)》;20170815;全文 *
Accurate Image Super-Resolution Using Very Deep Convolutional Networks;Jiwon Kim等;《2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)》;20161212;全文 *
Local Laplacian filters edge-aware image processing with a Laplacian pyramid;Sylvain Paris等;《Communication of the ACM》;20151231;全文 *
Multispectral Image Super-Resolution via RGB Image Fusion and Radiometric Calibration;Zhi-Wei Pan等;《 IEEE Transactions on Image Processing》;20181126;全文 *
基于正交元空间的多模态图像增强技术研究;梁杰;《中国优秀硕士学位论文全文数据库信息科技辑》;20200315;全文 *
基于深度残差学习的图像超分辨率重建研究;张广睿;《中国优秀硕士学位论文全文数据库信息科技辑》;20200215;全文 *

Also Published As

Publication number Publication date
CN112163994A (en) 2021-01-01

Similar Documents

Publication Publication Date Title
CN112163994B (en) Multi-scale medical image fusion method based on convolutional neural network
Suryanarayana et al. Accurate magnetic resonance image super-resolution using deep networks and Gaussian filtering in the stationary wavelet domain
CN104282007B (en) Based on the adaptive Method of Medical Image Fusion of non-sampled profile wave convert
CN107392852B (en) Super-resolution reconstruction method, device and equipment for depth image and storage medium
Huang et al. Deep hyperspectral image fusion network with iterative spatio-spectral regularization
Pan et al. Dual convolutional neural networks for low-level vision
CN109509160A (en) Hierarchical remote sensing image fusion method utilizing layer-by-layer iteration super-resolution
CN114723630B (en) Image deblurring method and system based on cavity double-residual multi-scale depth network
Yadav et al. A review on image fusion methodologies and applications
CN112785539B (en) Multi-focus image fusion method based on image adaptive decomposition and parameter adaptive
CN103400360A (en) Multi-source image fusing method based on Wedgelet and NSCT (Non Subsampled Contourlet Transform)
CN116402691B (en) Image super-resolution method and system based on rapid image feature stitching
CN107146202B (en) Image blind deblurring method based on L0 regularization and fuzzy kernel post-processing
Liu et al. Sparse representation with morphologic regularizations for single image super-resolution
CN115170410A (en) Image enhancement method and device integrating wavelet transformation and attention mechanism
Gao et al. Learning to Incorporate Texture Saliency Adaptive Attention to Image Cartoonization.
Wang et al. Medical image super-resolution analysis with sparse representation
CN106981046B (en) Single image super resolution ratio reconstruction method based on multi-gradient constrained regression
Chahi et al. MFGAN: towards a generic multi-kernel filter based adversarial generator for image restoration
CN116611995A (en) Super-resolution reconstruction method of handwritten text image based on depth expansion network
CN110895790A (en) Scene image super-resolution method based on posterior degradation information estimation
Jana et al. NSCT-DCT based Fourier analysis for fusion of multimodal images
Yi et al. Attention-model guided image enhancement for robotic vision applications
Tun et al. Joint Training of Noisy Image Patch and Impulse Response of Low-Pass Filter in CNN for Image Denoising
CN113129244B (en) Multi-scale medical image fusion method based on local Laplace decomposition

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant