CN114565816A - Multi-modal medical image fusion method based on global information fusion - Google Patents

Multi-modal medical image fusion method based on global information fusion Download PDF

Info

Publication number
CN114565816A
CN114565816A CN202210202366.1A CN202210202366A CN114565816A CN 114565816 A CN114565816 A CN 114565816A CN 202210202366 A CN202210202366 A CN 202210202366A CN 114565816 A CN114565816 A CN 114565816A
Authority
CN
China
Prior art keywords
fusion
module
convolution
nth
network
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202210202366.1A
Other languages
Chinese (zh)
Other versions
CN114565816B (en
Inventor
陈勋
张静
刘爱萍
张勇东
吴枫
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
University of Science and Technology of China USTC
Original Assignee
University of Science and Technology of China USTC
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by University of Science and Technology of China USTC filed Critical University of Science and Technology of China USTC
Priority to CN202210202366.1A priority Critical patent/CN114565816B/en
Publication of CN114565816A publication Critical patent/CN114565816A/en
Application granted granted Critical
Publication of CN114565816B publication Critical patent/CN114565816B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/25—Fusion techniques
    • G06F18/253—Fusion techniques of extracted features
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
    • G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
    • G06F18/2155—Generating training patterns; Bootstrap methods, e.g. bagging or boosting characterised by the incorporation of unlabelled data, e.g. multiple instance learning [MIL], semi-supervised techniques using expectation-maximisation [EM] or naïve labelling
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/088—Non-supervised learning, e.g. competitive learning
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
    • Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Artificial Intelligence (AREA)
  • General Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • Evolutionary Computation (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Computational Linguistics (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Health & Medical Sciences (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • Evolutionary Biology (AREA)
  • General Health & Medical Sciences (AREA)
  • Molecular Biology (AREA)
  • Computing Systems (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Image Analysis (AREA)
  • Image Processing (AREA)

Abstract

The invention discloses a multi-modal medical image fusion method based on global information fusion, which comprises the following steps: 1. carrying out color space conversion and image shearing preprocessing on the original medical images of a plurality of modalities; 2. establishing a modal branch network which interacts through a fusion module in a plurality of scales, and establishing a fusion module consisting of a Transformer to combine multi-modal characteristic information; 3. establishing a reconstruction module, and synthesizing a fusion image from multi-scale multi-modal characteristics; 4. training and evaluating the model on a public data set; 4. and realizing a medical image fusion task by using the trained model. The invention can fully fuse the multi-modal semantic information through the Transformer fusion module and the interactive modal branch network, realizes the fine-grained fusion effect, well retains the structure and texture information of the original image, and improves the mosaic phenomenon caused by the low-resolution medical image.

Description

Multi-modal medical image fusion method based on global information fusion
Technical Field
The invention relates to the technical field of image fusion, in particular to a medical image fusion technology based on deep learning.
Background
Medical images can assist doctors in better understanding human structures and tissues, and are widely used in clinical applications such as disease diagnosis, treatment planning, surgical guidance, and the like. Due to the difference of imaging mechanisms, the attention degree of medical images of different modalities to human organs and tissues is different. The medical image of single modality often can not provide comprehensive and sufficient information, and doctors often need to observe a plurality of images simultaneously to accurately judge the state of an illness, which inevitably brings certain difficulty to diagnosis. Due to the limitations of single modality medical images, multi-modality medical image fusion is a field in which research is very necessary. The multi-modal medical image fusion is to synthesize an image by synthesizing important information of medical images of different modalities in the same scene.
Generally, medical images can be divided into anatomical images and functional images. The anatomical image has high spatial resolution, can clearly image the anatomical structure of organs, but cannot display the function change of human metabolism, such as Computed Tomography (CT) and Magnetic Resonance imaging (MR); functional images, on the contrary, are very functional and metabolic displays, but have low resolution and do not accurately describe anatomical details of organs, such as Positron Emission Tomography (PET) and Single-Photon Emission Computed Tomography (Single-Photon Emission Computed Tomography). Even if CT and MR are both anatomical images and PET and SPECT are both functional images, the information they are interested in is different. CT mainly reflects the position information of human bones and implants, while MR is mainly used to provide clear detailed information to soft tissues and other parts. MR also contains multiple modalities, focusing on sub-regions of different properties, and the commonly used modalities are T1 weighted (denoted as T1), contrast enhanced T1 weighted (denoted as T1c), T2 weighted (denoted as T2), and liquid decay inversion recovery pulse (denoted as FLAIR). PET primarily reflects tumor function and metabolic information, whereas SPECT primarily provides organ and tissue blood flow information.
Most multi-modality medical image fusion methods can be summarized in three processes: and (5) extracting, fusing and reconstructing the characteristics. In order to realize medical image fusion, various algorithms have been proposed by scholars at home and abroad for over three decades, and generally speaking, the methods can be divided into two major categories: a conventional fusion method and a fusion method based on deep learning.
In the traditional medical image fusion framework, researchers propose a plurality of decomposition modes or transformation modes to extract the characteristics of a source image, then select a certain fusion strategy to fuse the characteristics, and finally, perform inverse transformation on the fused characteristics to reconstruct a fused image. Conventional methods can be divided into four categories according to the way of feature extraction: (1) sparse representation-based methods; (2) methods based on multi-scale decomposition, such as pyramids and wavelets; (3) subspace-based methods, such as independent component analysis; (4) a method based on salient features. The traditional medical image fusion method has achieved good fusion effect, but has some defects, and further improvement of fusion performance is limited. First, the fusion performance of the conventional approach relies heavily on artificially defined features, which may limit the generalization of the approach to other fusion tasks. Second, different features may require different fusion strategies to function. Third, for the fusion method based on sparse representation, the dictionary learning is relatively time-consuming, so that it takes much time to synthesize a fusion image.
In recent years, a method based on deep learning becomes a new research hotspot in the field of image fusion, a deep learning model represented by a Convolutional Neural Network (CNN) and a Generative adaptive Network (generic adaptive Network) is successfully applied to image fusion problems of multi-focus, infrared, visible light and the like, characteristics and fusion strategies do not need to be defined artificially, and the method has the advantages over the traditional fusion method. However, because a reference image of a fusion result cannot be constructed for supervised learning and the complex diversity of human structures and tissues causes that the imaging characteristics of each modality are not easy to quantitatively describe and other factors, the current medical image fusion method based on deep learning has relatively few researches and is still in a starting stage.
Investigations have found that existing medical image fusion methods typically use either a human-defined fusion strategy or a convolution-based network to fuse multimodal image features. However, these fusion strategies do not efficiently extract global semantic information of multimodal images. In addition, the current medical image fusion method based on deep learning has the problems of insufficient and imprecise utilization of multi-modal image information. Most approaches use multimodal images in a simplistic way, the most common way being to stack the original different modality images (or the respective extracted underlying features) in channel dimensions and then input them directly into the network model for fusion.
Disclosure of Invention
The invention provides a medical image fusion method based on global information fusion to overcome the defects of the prior art, so that global information with multi-modal characteristics can be combined through a self-attention mechanism, and information of different modes can be utilized to the maximum extent through an interactive mode branch network, and a high-quality medical image fusion effect is realized.
In order to achieve the purpose, the invention adopts the following technical scheme:
the invention relates to a multi-modal medical image fusion method based on global information fusion, which is characterized by comprising the following steps:
step one, obtaining original medical images of M different modes and carrying out conversion of YCbCr color space to obtain Y channel images { I) of all modes1，...，Im...，IM}; wherein, ImA Y-channel image representing the mth modality, M ∈ {1, 2.., M }; y-channel image for all modalities { I1，...，Im...，IMCutting the image to obtain the image block set S of all the modes1，...，Sm...，SMIn which S ismA set of image blocks representing an mth modality;
step two, constructing a fusion network Transfusion, comprising the following steps: m modal branch networks, N fusion modules and a reconstruction module; and sets the image blocks of all the modalities { S1，...，Sm...，SMInputting into the fusion network TransFusion:
step 2.1, constructing the M modal branch networks and the N fusion modules:
step 2.1.1, M modal branch networks are constructed:
the mth modal branch network in the M modal branch networks is composed of N convolution modules, and the N convolution modules are respectively marked as ConvBlockm1，...，ConvBlockmn，...，ConvBlockmNWherein, ConvBlockmnAn nth convolution module representing an mth modal branching network, N ∈ {1, 2.., N };
when n is 1, the nth convolution module convBlock of the mth modal branching networkmnFrom XmnTwo-dimensional convolution layers;
when N is 2, 3.. times.n, the nth convolution module ConvBlock of the mth modal branching networkmnFrom a maximum pooling layer and XmnTwo-dimensional convolution layers;
the convolution kernel size of the x two-dimensional convolution layer of the n convolution module of the m modal branching network is ksmnx×ksmnxThe number of convolution kernels is knmnx，x∈{1，2，...，Xmn}；
Step 2.1.2, constructing N fusion modules:
any nth fusion module of the N fusion modules is a Transformer network and consists of L self-attention mechanism modules; the first of the L self-attention mechanism modules comprises: 1 multi-head attention layer, 2 layers of normalization and 1 full-connection layer;
step 2.2, gathering image blocks of all modes { S1，...，Sm...，SMInputting into M modal branch networks, and performing information fusion through N fusion modules:
when n is 1, the set S of image blocks of the mth modalitymAn nth convolution module ConvBlock input to the mth modal branching networkmnX of (2)mnOutputting characteristic diagram after two-dimensional convolution layer
Figure BDA0003529948640000031
Wherein Hn、Wn、DnRespectively representing the output characteristic diagram of the mth modal branch network in the nth convolution module
Figure BDA0003529948640000032
High, wide, number of channels; thereby obtaining the output characteristic diagram of the M modal branch networks in the nth convolution module
Figure BDA0003529948640000033
Output characteristic diagram of the M modal branch networks in the nth convolution module
Figure BDA0003529948640000034
After the n-th fusion module processes, outputting a characteristic diagram
Figure BDA0003529948640000035
Wherein the content of the first and second substances,
Figure BDA0003529948640000036
friend the mth characteristic diagram output by the nth fusion module;
the mth feature map output by the nth fusion module
Figure BDA0003529948640000037
Feature map output by the nth convolution module of the mth modal branching network
Figure BDA0003529948640000038
Adding to obtain the characteristic diagram of the mth modal branch network after the nth convolution module interaction
Figure BDA0003529948640000039
Thereby obtaining the characteristic diagram of the M modal branch networks after the n convolution module interaction
Figure BDA00035299486400000310
When N is 2,3, the characteristic diagram of the mth modal branch network after the interaction of the nth-1 convolution modules
Figure BDA00035299486400000323
Inputting the data into the maximum pooling layer of the nth convolution module of the mth modal branch network for down-sampling to obtain a feature map of the mth modal branch network after down-sampling in the nth convolution module
Figure BDA00035299486400000312
The feature map after down sampling
Figure BDA00035299486400000324
Inputting the data into the 1 st two-dimensional convolution layer of the nth convolution module of the mth modal branch network, and sequentially passing through XmAfter the two-dimensional convolution layer is processed, a characteristic diagram is output
Figure BDA00035299486400000314
Thereby obtaining a characteristic diagram output by the nth convolution module of the M modal branch networks
Figure BDA00035299486400000315
The characteristic diagram is combined
Figure BDA00035299486400000316
Through the n fusionAfter the module is processed, the characteristic diagram is output
Figure BDA00035299486400000317
The mth feature map output by the nth fusion module
Figure BDA00035299486400000318
Characteristic diagram output by nth convolution module of mth modal branch network
Figure BDA00035299486400000319
Adding to obtain an added characteristic diagram
Figure BDA00035299486400000325
Thereby obtaining M added feature maps
Figure BDA00035299486400000321
Further obtaining a characteristic diagram output by the Nth fusion module
Figure BDA00035299486400000322
Step 2.3, the reconstruction module is composed of an N-level convolution network; and outputting the feature maps of the N fusion modules
Figure BDA0003529948640000041
Inputting the image data into the reconstruction module to obtain a preliminary fusion image F':
step 2.3.1, all characteristic graphs output by the n fusion module
Figure BDA0003529948640000042
Adding to obtain a fused feature map phin(ii) a Thereby obtaining a fusion characteristic diagram { phi ] of N fusion modules1，...，Φn，...，ΦN}；
Step 2.3.2, a reconstruction module formed by N-level convolution networks is constructed, and the fusion characteristic diagram phi of the N-th fusion module is usednThe nth stage convolution network input to the reconstruction module:
when n is 1The nth stage convolutional network comprises: b isnConvolution module RConvBlockn1，...，RConvBlocknb，...，
Figure BDA0003529948640000043
When N is 2, 3.. times.n, the nth stage convolutional network includes: b isnConvolution module RConvBlockn1，...，RConvBlocknb，...，
Figure BDA0003529948640000044
And Bn+1 upsampling layer UpSamplen1，...，UpSamplenb，...，
Figure BDA0003529948640000045
The b convolution module RConvBlock of the nth stage convolution networknbConsists of Y two-dimensional convolution layers, B is in the form of {1,2n}；
When n is 1 and b is 1, fusing the feature map phi of the n-th fusion modulenA b-th convolution module RConvBlock input to the n-th stage convolution networknbAnd outputting a characteristic diagram phi Rnb；
When N is 2,3,.., N and b is 1, the fusion feature map Φ of the N-th fusion module is comparednUpSample of the b-th upsampling layer input to the n-th convolutional networknbAfter up-sampling processing is carried out, an up-sampled characteristic diagram phi U is outputnb(ii) a Thereby obtaining the characteristic diagram { phi U after the up-sampling of the convolution networks from the 2 nd level to the N-1 th level2b，...，ΦUnb，...，ΦUNb}；
When N is 2,3, thenThen, the fusion characteristic diagram phi of the n-th fusion module is combinednOutput characteristic diagram { phi R of first b-1 convolution modules of nth-level convolution networkn1，...，ΦRn(b-1)B characteristic graphs after up-sampling of the first b of the (n + 1) th level convolution network (phi U)(n+1)1，...，ΦU(n+1)bSplicing to obtain a spliced characteristic diagram; inputting the spliced feature map to the nth stageThe b-th convolution module RConvBlock of the convolution networknbAnd outputting the output characteristic diagram phi R of the b convolution module of the nth-level convolution networknb(ii) a Thereby obtaining the B-th of the 1 st convolution network1Output characteristic diagram of convolution module
Figure BDA0003529948640000046
The B-th of the 1 st level convolution network1Output characteristic diagram of convolution module
Figure BDA0003529948640000047
After processing of a convolution layer, obtaining a primary fusion image F';
step three, constructing a loss function and training a network to obtain an optimal fusion model:
step 3.1, respectively calculating image block sets { S of all modes1，...，Sm...，SMEntropy of each image block set in the image block set is obtained, and a corresponding entropy value { e }is obtained1，...，em...，eMIn which emEntropy values of a set of image blocks representing an mth modality;
step 3.2, for the entropy value { e1，...，em...，eMRespectively carrying out normalization processing to obtain image block sets (S) of all modes1，...，Sm...，SMWeight of [ omega ]1，...，ωm，...，ωMIn which ω ismWeights representing a set of image blocks of the mth modality;
step 3.3, constructing a total Loss function Loss by using the formula (1):
Figure BDA0003529948640000051
in the formula (1), Lssim(SmF') represents the set S of image blocks of the m-th modalitymA structural similarity loss function with the preliminary fused image F';
step 3.4, an optimizer is used for carrying out minimum solving on the total Loss function Loss, so that all parameters in the fusion network Transfusion are optimized, and an optimal fusion model is obtained;
step four, utilizing the optimal fusion model to perform Y-channel image { I) of all modes1，I2，...，IMProcessing and outputting a primary fusion image F'; the preliminary fused image F' is converted into an RGB color space, thereby obtaining a final fused image F.
The multi-modal medical image fusion method based on the global information fusion is also characterized in that the nth fusion module in the step 2.2 is processed according to the following processes:
step 2.2.1, the characteristic diagram of the N-th convolution module output by the n-th fusion module to the M modal branch networks
Figure BDA0003529948640000052
Splicing and leveling to obtain the size of (M H)n*Wn)×DnThe flattened feature vectors of (a); adding the flattened feature vector and a trainable vector with the same size to obtain the feature vector containing the position information of the nth fusion module
Figure BDA0003529948640000053
Step 2.2.2, when l is 1, the 1 st self-attention mechanism module of the nth fusion module uses the feature vector
Figure BDA00035299486400000511
After linear mapping, three matrixes Q are obtainednl，Knl，Vnl(ii) a Recalculate Qnl，Knl，VnlMultiple head attention results Z in betweennl(ii) a The multiple head attention result ZnlInputting the data into a full connection layer of the ith self-attention mechanism module of the nth fusion module, and obtaining an output sequence vector of the ith self-attention mechanism module of the nth fusion module
Figure BDA00035299486400000512
When l is 2,3,.. times.N, the output sequence vector of the l-1 self-attention mechanism module of the N-th fusion module is used
Figure BDA0003529948640000056
Inputting the input into the first self-attention mechanism module of the nth fusion module, and obtaining the output sequence vector of the first self-attention mechanism module of the nth fusion module
Figure BDA0003529948640000057
Thereby obtaining the output sequence vector of the L-th self-attention mechanism module of the n-th fusion module
Figure BDA0003529948640000058
Step 2.2.3, outputting the output sequence vector
Figure BDA0003529948640000059
Dividing into M modes, and deforming the size of each mode into Hn×Wn×DnTo obtain an output characteristic diagram
Figure BDA00035299486400000510
Compared with the prior art, the invention has the beneficial effects that:
1. the invention provides an unsupervised anatomical and functional image fusion method. The method introduces a Transformer structure as a fusion strategy, and the Transformer combines the global information of the multi-modal medical image and sufficiently fuses the semantic information of the multi-modal image by utilizing a self-attention mechanism contained in the Transformer, so that a fine-grained fusion effect is realized. The invention not only well retains the structure and texture information in the anatomical image, but also improves the mosaic phenomenon caused by low resolution in the functional image.
2. The present invention proposes modal branching networks that interact on multiple scales. The network can extract multi-scale complementary features of each modal image, and fully utilizes multi-modal image information. The interactive branched network enhances the anatomical and functional image fusion effect.
Drawings
FIG. 1 is a schematic flow chart of the method of the present invention, wherein "ks × ks, kn" represents a convolution layer with a kernel size of ks × ks and a number of kernels of kn;
fig. 2 is a structural diagram of a modal branching network and a convergence module provided in an embodiment of the present invention;
fig. 3 is a structural diagram of a reconstruction module according to an embodiment of the present invention.
Detailed Description
In this embodiment, a multi-modal image fusion method based on global information fusion, as shown in fig. 1, includes the following steps:
step one, obtaining M original medical images of different modalities and carrying out preprocessing of color space conversion and image shearing to obtain a preprocessed image block set { S) of all modalities1，S2，...，SMIn which S ismA set of image blocks representing the mth modality, M ∈ {1, 2.., M }:
step 1.1, acquiring original medical images of a plurality of modalities required by an experiment from a Harvard medical image data set website (http:// www.med.harvard.edu/AANLIB/home. html); the present embodiment collects medical images of M-2 modalities from this public dataset, including 279 pairs of MR-T1 and PET images and 318 pairs of MR-T2 and SPECT images, where MR-T1 and MR-T2 are anatomical images in grayscale, the number of channels is 1, PET and SPECT are functional images of RGB color space, and the number of channels is 3;
step 1.2, converting the image of the RGB color space into the YCbCr space according to the formula (1):
Figure BDA0003529948640000061
in formula (1), R, G, B are three channels of RGB color space, Y is a luminance channel, and Cb and Cr are two color channels, respectively;
step (ii) of1.3, in order to expand the number of samples, cutting the gray level image and the image of the Y channel into image blocks to obtain an image block set { S) of all modes1，S2，...，SM}; in this embodiment, the size of the clipped image block is 64 × 64;
step two, constructing a fusion network Transfusion, comprising the following steps: m modal branch networks, N fusion modules and a reconstruction module; and sets the image blocks of all the modalities { S1，...，Sm...，SMInputting into a fusion network TransFusion:
step 2.1, constructing M modal branch networks and N fusion modules:
step 2.1.1, constructing M modal branch networks:
the mth modal branch network in the M modal branch networks is composed of N convolution modules, and the N convolution modules are respectively marked as ConvBlockm1，...，ConvBlockmn，...，ConvBlockmNWherein, ConvBlockmnAn nth convolution module representing an mth modal branching network, N ∈ {1, 2.., N };
when n is equal to 1, the nth convolution module convBlock of the mth modal branching networkmnFrom XmnTwo-dimensional convolution layers;
when N is 2, 3.. times.n, the nth convolution module ConvBlock of the mth modal branching networkmnFrom a maximum pooling layer and XmnTwo-dimensional convolution layers;
the convolution kernel size of the x two-dimensional convolution layer of the n convolution module of the m modal branching network is ksmnx×ksmnxThe number of convolution kernels is knmnx，x∈{1，2，...，Xmn}；
In this embodiment, N is 4, the kernel size of all the largest pooling layers is 2 × 2, the step size is 2, and X ismn、ksmnx、knmnxAs shown in fig. 2;
step 2.1.2, constructing N fusion modules:
any nth fusion module of the N fusion modules is a Transformer network and consists of L self-attention mechanism modules; the first of the L self-attention mechanism modules comprises: 1 multi-head attention layer, 2 layers of normalization and 1 full-connection layer; in the present embodiment, L ═ 1;
step 2.2, gathering image blocks of all modes { S1，...，Sm...，SMInputting into M modal branch networks, and performing information fusion through N fusion modules:
when n is 1, the set S of image blocks of the mth modalitymThe nth convolution module ConvBlock input to the mth modal branching networkmnX of (2)mnOutputting characteristic diagram after two-dimensional convolution layer
Figure BDA0003529948640000071
Wherein Hn、Wn、DnRespectively representing the output characteristic diagram of the mth modal branch network at the nth convolution module
Figure BDA0003529948640000072
High, nest, channel count; thereby obtaining the output characteristic diagram of the M modal branch networks in the nth convolution module
Figure BDA0003529948640000073
In this example, (H)1，W1，D1)＝(64，64，64)，(H2，W2，D2)＝(32，32，128)，(H3，W3，D3)＝(16，16，256)，(H4，W4，D4)＝(8，8，512)；
Output characteristic diagram of M modal branch networks in nth convolution module
Figure BDA0003529948640000074
After the n-th fusion module is processed according to the formula (2), a characteristic diagram is output
Figure BDA0003529948640000075
Wherein, the first and the second end of the pipe are connected with each other,
Figure BDA0003529948640000076
an mth feature map representing an output of the nth fusion module:
Figure BDA0003529948640000077
in the formula (2), a TransformernRepresenting the nth fusion module and realizing the following steps:
step 2.2.1, the characteristic diagram of the n-th convolution module output by the n-th fusion module to the M modal branch networks
Figure BDA0003529948640000078
Splicing and leveling to obtain the size of (M H)n*Wn)×DnThe flattened feature vectors of (a); adding the flattened feature vector with a trainable vector with the same size to obtain the feature vector containing the position information of the nth fusion module
Figure BDA0003529948640000079
Step 2.2.2, the feature vector is processed by the self-attention mechanism module of the n fusion module
Figure BDA0003529948640000081
Linear mapping to three matrices, Qn，Kn，Vn：
Figure BDA0003529948640000082
In the formula (3), the reaction mixture is,
Figure BDA0003529948640000083
is a trainable matrix with dimensions of Dn×Dn；
Step 2.2.3, mixing Qn，Kn，VnAre respectively divided into h heads to obtain
Figure BDA0003529948640000084
i ∈ {1, 2., h }, and then the multi-head attention is calculated according to equation (4) -equation (6), resulting in a result Z:
Figure BDA0003529948640000085
Figure BDA0003529948640000086
Figure BDA0003529948640000087
in the formula (5), Concat represents a splicing operation,
Figure BDA0003529948640000088
is a trainable matrix; in formula (6), LayerNorm indicates layer normalization;
step 2.2.4, result Z of multi-head attentionnInputting the data into a full connection layer to obtain an output sequence vector of the nth fusion module
Figure BDA0003529948640000089
Figure BDA00035299486400000810
In formula (7), MLP represents a fully connected layer;
step 2.2.5, mixing
Figure BDA00035299486400000811
Dividing into M modes, and deforming the size of each mode into Hn×Wn×DnObtaining an output characteristic diagram
Figure BDA00035299486400000812
Wherein the content of the first and second substances,
Figure BDA00035299486400000813
an mth feature map representing an output of the nth fusion module;
the mth feature map output by the nth fusion module
Figure BDA00035299486400000814
Characteristic diagram output by nth convolution module of mth modal branch network
Figure BDA00035299486400000815
Adding to obtain a characteristic diagram of the mth modal branch network after the nth convolution module is interacted
Figure BDA00035299486400000816
Thereby obtaining the characteristic diagram of the M modal branch networks after the n convolution module interaction
Figure BDA00035299486400000817
When N is 2,3,.. times, N, the characteristic diagram of the mth modal branch network after the interaction of the (N-1) th convolution module is obtained
Figure BDA00035299486400000818
Inputting the data into the maximum pooling layer of the nth convolution module of the mth modal branch network for down-sampling to obtain a feature map of the mth modal branch network after down-sampling in the nth convolution module
Figure BDA00035299486400000819
The feature map after down sampling
Figure BDA00035299486400000820
Transporting into the 1 st two-dimensional convolution layer of the nth convolution module of the mth modal branch network, and sequentially passing through XmnAfter the two-dimensional convolution layer is processed, a characteristic diagram is output
Figure BDA00035299486400000821
Thereby obtaining M modal branched netsCharacteristic diagram output by nth convolution module of network
Figure BDA00035299486400000822
Will feature map
Figure BDA00035299486400000823
After the processing of the n-th fusion module, a characteristic diagram is output
Figure BDA00035299486400000824
The mth feature map output by the nth fusion module
Figure BDA00035299486400000825
Characteristic diagram output by nth convolution module of mth modal branch network
Figure BDA00035299486400000826
Adding to obtain an added characteristic diagram
Figure BDA00035299486400000827
Thereby obtaining M added feature maps
Figure BDA0003529948640000091
Further obtaining a characteristic diagram output by the Nth fusion module
Figure BDA0003529948640000092
Step 2.3, constructing a reconstruction module consisting of N-level convolution networks, and fusing output characteristic graphs of the N fusion modules
Figure BDA0003529948640000093
Inputting the image into a reconstruction module to obtain a preliminary fusion image F':
step 2.3.1, all characteristic graphs output by the n fusion module
Figure BDA0003529948640000094
Adding to obtain a fused feature map phin(ii) a Thereby obtaining N fusion modulesFused feature map of blocks Φ1，...，Φn，...，ΦN}；
Step 2.3.2, a reconstruction module formed by N-level convolution networks is constructed, and the fusion characteristic diagram phi of the N-th fusion module is usednThe nth level convolution network input to the reconstruction module:
when n is 1, the nth stage convolutional network includes: b isnConvolution module RConvBlockn1，RConvBlockn2，...，
Figure BDA0003529948640000095
When N is 2, 3.. times.n, the nth stage convolutional network includes: b isnConvolution module RConvBlockn1，RConvBlockn2，...，
Figure BDA0003529948640000096
And Bn+1 upsampling layer UpSamplen1，UpSamplen2，...，
Figure BDA0003529948640000097
The b convolution module RConvBlock of the nth stage convolution networknbConsists of Y two-dimensional convolution layers, B is in the form of {1,2n}；
When n is 1 and b is 1, fusing characteristic diagram phi of the n-th fusing modulenThe b convolution module RConvBlock input to the nth stage convolution networknbAnd outputting a characteristic diagram phi Rnb；
When N is 2,3, the. N, and b is 1, the fusion feature map Φ of the N-th fusion module is comparednInput to the b-th upsampling layer UpSample of the nth stage convolutional networknbAfter up-sampling processing is carried out, an up-sampled characteristic diagram phi U is outputnb(ii) a Thereby obtaining the characteristic diagram { phi U after the up-sampling of the convolution networks from the 2 nd level to the N-1 th level2b，...，ΦUnb，...，ΦUNb}；
When N is 2,3, thenThen, the fusion characteristic diagram phi of the n-th fusion module is combinednN-th order convolutionOutput characteristic diagram { phi R of first b-1 convolution modules of networkn1，...，ΦRn(b-1)B characteristic graphs after up-sampling of the first b of the (n + 1) th level convolution network (phi U)(n+1)1，...，ΦU(n+1)bSplicing to obtain a spliced characteristic diagram; inputting the spliced feature map into a b-th convolution module RConvBlock of an nth-stage convolution networknbAnd outputting the output characteristic diagram phi R of the b convolution module of the nth-level convolution networknb(ii) a Thereby obtaining the B-th of the 1 st convolution network1Output characteristic diagram of convolution module
Figure BDA0003529948640000098
Convolution network B of level 11Output characteristic diagram of convolution module
Figure BDA0003529948640000099
After processing of a convolution layer, obtaining a primary fusion image F';
in this embodiment, the reconstruction module is shown in fig. 3, where Y is 2 and B is1＝3，B2＝2，B3＝1，B4＝0；
Step three, constructing a loss function and training a network to obtain an optimal fusion model:
step 3.1, calculating image block sets { S of all the modalities according to the formulas (8) to (9)1，S2，...，SMEntropy of each image block set in the image block set is obtained, and a corresponding entropy value { e }is obtained1，e2，...，eMIn which emEntropy values of a set of image blocks representing the mth modality:
em＝Entropy(Sm) (8)
Figure BDA0003529948640000101
in the formula (9), plProbability of being the l-th gray value;
step 3.2, entropy value { e }1，e2，...，eMRespectively carrying out normalization processing to obtain image block sets (S) of all modes1，S2，...，SMWeight of { omega } ω1，ω2，...，ωMIn which ω ismWeight of set of image blocks representing mth modality:
Figure BDA0003529948640000102
in the formula (10), η is a temperature parameter; in the present embodiment, η ═ 1;
step 3.3, constructing a total Loss function Loss by using the formula (11):
Loss＝ω1Lssim(S1，F′)+ω2Lssim(S2，F′) (11)
Lssim(Sj，F′)＝1-SSIM(Sj，F′) (12)
in the formula (11), Lssim(SmF') represents the set S of image blocks of the m-th modalitymA structural similarity loss function with the preliminary fused image F'; in formula (12), SSIM is a structural similarity function;
step 3.4, adopting an AdamW optimizer to carry out minimum solution on the Loss function Loss, thereby optimizing all parameters in the fusion network Transfusion and obtaining an optimal fusion model;
step four, utilizing the optimal fusion model to perform Y-channel image or gray level image { I) of all modes1，I2，...，IMProcessing and outputting a primary fusion image F'; splicing the preliminary fusion image F 'with Cb and Cr channels and converting the preliminary fusion image F' into an RGB color space to obtain a final fusion image F;
step five, evaluating the performance of the invention:
in specific implementation, the present invention is compared with the conventional methods CSMCA and the deep learning methods DDcGAN and EMFusion. In addition, for explaining the Transformer-based fusion module and the interactive modal branching network in the inventionTwo comparative experiments were set up. The first experiment removed the Transformer fusion module and the second replaced the interactive modal branching network with a weight-shared modal branching network. Boundary-based similarity measurement Q using mutual information, mean gradientAB/FVisual perception index QCVMutual information, average gradient, Q as evaluation indexAB/FThe larger, QCVThe smaller the quality of the fused image. The average fusion quality for the 30 pairs of MR-T1 and PET test images and the 30 pairs of MR-T2 and SPECT test images is as follows:
TABLE 1. fusion Performance of different methods
Figure BDA0003529948640000111
The experimental result shows that the invention has mutual information, average gradient and QAB/F、QCVThe four indexes are all optimal. The Transformer fusion module of the invention helps to enhance mutual information of 5.10% -10.02%, average gradient of 2.59% -5.28%, Q of 3.04% -4.36%AB/F1.43% -12.66% of QCV(ii) a The inventive interactive modal branching network helps to enhance mutual information of 18.39% -19.91%, average gradient of 1.06% -6.69%, Q of 7.68% -11.02%AB/F27.69% -62.22% of QCV。

Claims (2)

1. A multi-modal medical image fusion method based on global information fusion is characterized by comprising the following steps:
step one, obtaining original medical images of M different modes and carrying out conversion of YCbCr color space to obtain Y channel images { I) of all modes1,…,Im…,IM}; wherein, ImA Y-channel image representing the mth modality, M ∈ {1,2, …, M }; y-channel image for all modalities { I1,…,Im…,IMCutting the image to obtain the image block set S of all the modes1,…,Sm…,SMIn which SmDiagram representing the m-th modalityA set of blocks;
step two, constructing a fusion network Transfusion, which comprises the following steps: m modal branch networks, N fusion modules and a reconstruction module; and sets the image blocks of all the modalities { S1,…,Sm…,SMInputting into the fusion network TransFusion:
step 2.1, constructing the M modal branch networks and the N fusion modules:
step 2.1.1, constructing M modal branch networks:
the mth modal branch network in the M modal branch networks is composed of N convolution modules, and the N convolution modules are respectively recorded as ConvBlockm1,…,COnvBlockmn,…,ConvBlockmNWherein, ConvBlockmnAn nth convolution module representing an mth modal branching network, N ∈ {1,2, …, N };
when n is 1, the nth convolution module convBlock of the mth modal branching networkmnFrom XmnTwo-dimensional convolution layers;
when N is 2,3, …, N, the nth convolution module ConvBlock of the mth modal branching networkmnFrom a maximum pooling layer and XmnTwo-dimensional convolution layers;
the convolution kernel size of the x two-dimensional convolution layer of the n convolution module of the m modal branching network is ksmnx×ksmnxThe number of convolution kernels is knmnx，x∈{1,2,…,Xmn}；
Step 2.1.2, constructing N fusion modules:
any nth fusion module of the N fusion modules is a Transformer network and consists of L self-attention mechanism modules; the first of the L self-attention mechanism modules comprises: 1 multi-head attention layer, 2 layers of normalization and 1 full connection layer;
step 2.2, gathering image blocks of all modes { S1,…,Sm…,SMInputting into M modal branch networks, and performing information fusion through N fusion modules:
when n is 1, the map of the mth modeSet of blocks SmThe nth convolution module ConvBlock input to the mth modal branching networkmnX of (2)mnOutputting characteristic diagram after two-dimensional convolution layer
Figure FDA0003529948630000011
Wherein Hn、Wn、DnRespectively representing the output characteristic diagram of the mth modal branch network in the nth convolution module
Figure FDA0003529948630000012
High, wide, number of channels; thereby obtaining the output characteristic diagram of the M modal branch networks in the nth convolution module
Figure FDA0003529948630000013
Output characteristic diagram of the M modal branch networks in the nth convolution module
Figure FDA0003529948630000014
After the n-th fusion module processes, outputting a characteristic diagram
Figure FDA0003529948630000015
Wherein the content of the first and second substances,
Figure FDA0003529948630000016
an mth feature map representing an output of the nth fusion module;
the mth feature map output by the nth fusion module
Figure FDA0003529948630000021
Feature map output by the nth convolution module of the mth modal branching network
Figure FDA0003529948630000022
Adding to obtain the characteristic diagram of the mth modal branch network after the nth convolution module interaction
Figure FDA0003529948630000023
Thereby obtaining the characteristic diagram of the M modal branch networks after the n convolution module interaction
Figure FDA0003529948630000024
When N is 2,3, …, N, the characteristic diagram of the mth modal branching network after the interaction of the nth-1 convolution module
Figure FDA0003529948630000025
Inputting the data into the maximum pooling layer of the nth convolution module of the mth modal branch network for down-sampling to obtain a feature map of the mth modal branch network after down-sampling in the nth convolution module
Figure FDA0003529948630000026
The feature map after down sampling
Figure FDA0003529948630000027
Inputting the data into the 1 st two-dimensional convolution layer of the nth convolution module of the mth modal branch network, and sequentially passing through XmnAfter the two-dimensional convolution layer is processed, a characteristic diagram is output
Figure FDA0003529948630000028
Thereby obtaining a characteristic diagram output by the nth convolution module of the M modal branch networks
Figure FDA0003529948630000029
The characteristic diagram is combined
Figure FDA00035299486300000210
After the processing of the n-th fusion module, a characteristic diagram is output
Figure FDA00035299486300000211
The mth feature map output by the nth fusion module
Figure FDA00035299486300000212
Characteristic diagram output by nth convolution module of mth modal branch network
Figure FDA00035299486300000213
Adding to obtain an added characteristic diagram
Figure FDA00035299486300000214
Thereby obtaining M added feature maps
Figure FDA00035299486300000215
Further obtaining a characteristic diagram output by the Nth fusion module
Figure FDA00035299486300000216
Step 2.3, the reconstruction module is composed of an N-level convolution network; and outputting the feature maps of the N fusion modules
Figure FDA00035299486300000217
Inputting the image data into the reconstruction module to obtain a preliminary fusion image F':
step 2.3.1, all characteristic graphs output by the n fusion module
Figure FDA00035299486300000218
Adding to obtain a fused feature map phin(ii) a Thereby obtaining a fusion characteristic diagram { phi ] of N fusion modules1,…,Φn,…,ΦN}；
Step 2.3.2, a reconstruction module formed by N-level convolution networks is constructed, and the fusion characteristic diagram phi of the N-th fusion module is usednThe nth stage convolution network input to the reconstruction module:
when n is 1, the nth stage convolutional network includes: b isnA convolution module
Figure FDA00035299486300000219
Figure FDA00035299486300000220
When N is 2,3, …, N, the nth stage convolutional network comprises: b isnA convolution module
Figure FDA00035299486300000221
Figure FDA00035299486300000222
And Bn+1 upsampling layers
Figure FDA00035299486300000223
Figure FDA00035299486300000224
The b convolution module RConvBlock of the nth stage convolution networknbConsists of Y two-dimensional convolution layers, B ∈ {1,2, …, Bn}；
When n is 1 and b is 1, fusing the feature map phi of the n-th fusion modulenA b-th convolution module RConvBlock input to the n-th stage convolution networknbAnd outputting a characteristic diagram phi Rnb；
When N is 2,3, …, N and b is 1, fusing the feature map phi of the N-th fusing modulenUpSample of the b-th upsampling layer input to the n-th convolutional networknbAfter up-sampling processing is carried out, an up-sampled characteristic diagram phi U is outputnb(ii) a Thereby obtaining the up-sampled characteristic diagram { phi U of the convolution network from the 2 nd level to the N-1 st level2b,…,ΦUnb,…,ΦUNb}；
When N is 2,3, …, N and B is 2,3, …, BnThen, the fusion characteristic diagram phi of the n-th fusion module is combinednOutput characteristic diagram { phi R of the first b-1 convolution modules of the nth-level convolution networkn1,…,ΦRn(b-1) B characteristic graphs after up-sampling of the first b of the (n + 1) th level convolution network (phi U)(n+1)1,…,ΦU(n+1)bAfter splicing, obtainingSplicing the characteristic graphs; inputting the spliced feature map into a b-th convolution module RConvBlock of an n-th-level convolution networknbAnd outputting the output characteristic diagram phi R of the b convolution module of the nth-level convolution networknb(ii) a Thereby obtaining the B-th of the 1 st convolution network1Output characteristic diagram of convolution module
Figure FDA0003529948630000031
The B-th of the 1 st level convolution network1Output characteristic diagram of convolution module
Figure FDA0003529948630000032
After the processing of a convolution layer, obtaining a primary fusion image F';
step three, constructing a loss function and training a network to obtain an optimal fusion model:
step 3.1, respectively calculating image block sets { S of all modes1,…,Sm…,SMEntropy of each image block set in the image block set is obtained, and a corresponding entropy value { e }is obtained1,…,em…,eMIn which emEntropy values of a set of image blocks representing an mth modality;
step 3.2, for the entropy value { e1,…,em…,eMRespectively carrying out normalization processing to obtain image block sets (S) of all modes1,…,Sm…,SMWeight of [ omega ]1,…,ωm,…,ωMIn which ω ismWeights representing a set of image blocks of the mth modality;
and 3.3, constructing a total Loss function Loss by using the formula (1):
Figure FDA0003529948630000033
in the formula (1), Lssim(SmF') represents the set S of image blocks of the m-th modalitymStructural similarity with the preliminary fusion image FA loss function;
step 3.4, an optimizer is used for carrying out minimum solving on the total Loss function Loss, so that all parameters in the fusion network Transfusion are optimized, and an optimal fusion model is obtained;
step four, utilizing the optimal fusion model to perform Y-channel image { I) of all modes1,I2,…,IMProcessing and outputting a primary fusion image F'; and converting the preliminary fusion image F' into an RGB color space, thereby obtaining a final fusion image F.
2. The multi-modal medical image fusion method based on global information fusion as claimed in claim 1, wherein the n-th fusion module in the step 2.2 is processed according to the following procedures:
step 2.2.1, the characteristic diagram of the N-th convolution module output by the n-th fusion module to the M modal branch networks
Figure FDA0003529948630000034
Splicing and leveling to obtain the size of (M H)n*Wn)×DnThe flattened feature vectors of (a); adding the flattened characteristic vector and a trainable vector with the same size to obtain a characteristic vector containing position information of the nth fusion module
Figure FDA0003529948630000035
Step 2.2.2, when l is 1, the ith self-attention mechanism module of the nth fusion module uses the feature vector
Figure FDA0003529948630000041
After linear mapping, three matrixes Q are obtainednl，Knl，Vnl(ii) a Recalculate Qnl，Knl，VnlMultiple head attention results in between Znl(ii) a The multi-head attention result ZnlFull connection of the ith attention mechanism module input to the nth fusion moduleIn the layer connection, the output sequence vector of the ith self-attention mechanism module of the nth fusion module is obtained
Figure FDA0003529948630000042
When l is 2,3, …, N, the output sequence vector of the l-1 self-attention mechanism module of the N fusion module is added
Figure FDA0003529948630000043
Inputting the input into the first self-attention mechanism module of the nth fusion module, and obtaining the output sequence vector of the first self-attention mechanism module of the nth fusion module
Figure FDA0003529948630000044
Thereby obtaining the output sequence vector of the L-th self-attention mechanism module of the n-th fusion module
Figure FDA0003529948630000045
Step 2.2.3, outputting the sequence vector
Figure FDA0003529948630000046
Dividing into M modes, and deforming the size of each mode into Hn×Wn×DnTo obtain an output characteristic diagram
Figure FDA0003529948630000047
CN202210202366.1A 2022-03-03 2022-03-03 Multi-mode medical image fusion method based on global information fusion Active CN114565816B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202210202366.1A CN114565816B (en) 2022-03-03 2022-03-03 Multi-mode medical image fusion method based on global information fusion

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202210202366.1A CN114565816B (en) 2022-03-03 2022-03-03 Multi-mode medical image fusion method based on global information fusion

Publications (2)

Publication Number Publication Date
CN114565816A true CN114565816A (en) 2022-05-31
CN114565816B CN114565816B (en) 2024-04-02

Family

ID=81717119

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202210202366.1A Active CN114565816B (en) 2022-03-03 2022-03-03 Multi-mode medical image fusion method based on global information fusion

Country Status (1)

Country Link
CN (1) CN114565816B (en)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115115523A (en) * 2022-08-26 2022-09-27 中加健康工程研究院(合肥)有限公司 CNN and Transformer fused medical image depth information extraction method
CN115134676A (en) * 2022-09-01 2022-09-30 有米科技股份有限公司 Video reconstruction method and device for audio-assisted video completion
CN115511767A (en) * 2022-11-07 2022-12-23 中国科学技术大学 Self-supervised learning multi-modal image fusion method and application thereof

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR20210091276A (en) * 2018-11-16 2021-07-21 아리엘 에이아이, 인크. 3D object reconstruction
CN113469094A (en) * 2021-07-13 2021-10-01 上海中科辰新卫星技术有限公司 Multi-mode remote sensing data depth fusion-based earth surface coverage classification method
US11222217B1 (en) * 2020-08-14 2022-01-11 Tsinghua University Detection method using fusion network based on attention mechanism, and terminal device
CN114049408A (en) * 2021-11-15 2022-02-15 哈尔滨工业大学（深圳） Depth network model for accelerating multi-modality MR imaging

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR20210091276A (en) * 2018-11-16 2021-07-21 아리엘 에이아이, 인크. 3D object reconstruction
US11222217B1 (en) * 2020-08-14 2022-01-11 Tsinghua University Detection method using fusion network based on attention mechanism, and terminal device
CN113469094A (en) * 2021-07-13 2021-10-01 上海中科辰新卫星技术有限公司 Multi-mode remote sensing data depth fusion-based earth surface coverage classification method
CN114049408A (en) * 2021-11-15 2022-02-15 哈尔滨工业大学（深圳） Depth network model for accelerating multi-modality MR imaging

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
沈文祥;秦品乐;曾建潮;: "基于多级特征和混合注意力机制的室内人群检测网络", 计算机应用, no. 12 *
洪炎佳;孟铁豹;黎浩江;刘立志;李立;徐硕?;郭圣文;: "多模态多维信息融合的鼻咽癌MR图像肿瘤深度分割方法", 浙江大学学报(工学版), no. 03 *

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115115523A (en) * 2022-08-26 2022-09-27 中加健康工程研究院(合肥)有限公司 CNN and Transformer fused medical image depth information extraction method
CN115115523B (en) * 2022-08-26 2022-11-25 中加健康工程研究院(合肥)有限公司 CNN and Transformer fused medical image depth information extraction method
CN115134676A (en) * 2022-09-01 2022-09-30 有米科技股份有限公司 Video reconstruction method and device for audio-assisted video completion
CN115134676B (en) * 2022-09-01 2022-12-23 有米科技股份有限公司 Video reconstruction method and device for audio-assisted video completion
CN115511767A (en) * 2022-11-07 2022-12-23 中国科学技术大学 Self-supervised learning multi-modal image fusion method and application thereof

Also Published As

Publication number Publication date
CN114565816B (en) 2024-04-02

Similar Documents

Publication Publication Date Title
Zhang et al. Translating and segmenting multimodal medical volumes with cycle-and shape-consistency generative adversarial network
CN114565816B (en) Multi-mode medical image fusion method based on global information fusion
Li et al. VolumeNet: A lightweight parallel network for super-resolution of MR and CT volumetric data
CN107492071A (en) Medical image processing method and equipment
Du et al. Accelerated super-resolution MR image reconstruction via a 3D densely connected deep convolutional neural network
CN113506222B (en) Multi-mode image super-resolution method based on convolutional neural network
WO2022121100A1 (en) Darts network-based multi-modal medical image fusion method
Zhu et al. Stacked U-shape networks with channel-wise attention for image super-resolution
CN110322402A (en) Medical image super resolution ratio reconstruction method based on dense mixing attention network
CN111487573B (en) Enhanced residual error cascade network model for magnetic resonance undersampling imaging
CN111899165A (en) Multi-task image reconstruction convolution network model based on functional module
CN111598964B (en) Quantitative magnetic susceptibility image reconstruction method based on space adaptive network
Singh et al. Medical image generation using generative adversarial networks
CN114677263A (en) Cross-mode conversion method and device for CT image and MRI image
CN115511767A (en) Self-supervised learning multi-modal image fusion method and application thereof
CN112488971A (en) Medical image fusion method for generating countermeasure network based on spatial attention mechanism and depth convolution
CN115457359A (en) PET-MRI image fusion method based on adaptive countermeasure generation network
CN114821259A (en) Zero-learning medical image fusion method based on twin convolutional neural network
Yu et al. 3D Medical Image Segmentation based on multi-scale MPU-Net
Xie et al. MRSCFusion: Joint Residual Swin Transformer and Multiscale CNN for Unsupervised Multimodal Medical Image Fusion
CN116309754A (en) Brain medical image registration method and system based on local-global information collaboration
Li et al. HRINet: alternative supervision network for high-resolution CT image interpolation
CN117036162B (en) Residual feature attention fusion method for super-resolution of lightweight chest CT image
CN117649422B (en) Training method of multi-modal image segmentation model and multi-modal image segmentation method
Wu et al. Convolutional neural network with coarse-to-fine resolution fusion and residual learning structures for cross-modality image synthesis

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant