CN110827232B - Cross-modality MRI (magnetic resonance imaging) synthesis method based on morphological characteristics GAN (gamma GAN) - Google Patents
Cross-modality MRI (magnetic resonance imaging) synthesis method based on morphological characteristics GAN (gamma GAN) Download PDFInfo
- Publication number
- CN110827232B CN110827232B CN201911113248.8A CN201911113248A CN110827232B CN 110827232 B CN110827232 B CN 110827232B CN 201911113248 A CN201911113248 A CN 201911113248A CN 110827232 B CN110827232 B CN 110827232B
- Authority
- CN
- China
- Prior art keywords
- representative
- modality
- mode
- target
- information
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Expired - Fee Related
Links
- 238000001308 synthesis method Methods 0.000 title claims abstract description 15
- 230000000877 morphologic effect Effects 0.000 title claims abstract description 13
- 238000002595 magnetic resonance imaging Methods 0.000 title abstract description 21
- 238000000605 extraction Methods 0.000 claims abstract description 28
- 238000013256 Gubra-Amylin NASH model Methods 0.000 claims abstract description 21
- 238000010606 normalization Methods 0.000 claims description 36
- 230000015572 biosynthetic process Effects 0.000 claims description 22
- 238000003786 synthesis reaction Methods 0.000 claims description 22
- 238000004364 calculation method Methods 0.000 claims description 17
- 238000009826 distribution Methods 0.000 claims description 14
- 208000037170 Delayed Emergence from Anesthesia Diseases 0.000 claims description 11
- 230000003044 adaptive effect Effects 0.000 claims description 10
- 230000004927 fusion Effects 0.000 claims description 9
- 238000005457 optimization Methods 0.000 claims description 6
- 238000011176 pooling Methods 0.000 claims description 6
- 238000012549 training Methods 0.000 claims description 6
- 230000004913 activation Effects 0.000 claims description 3
- 238000013528 artificial neural network Methods 0.000 abstract description 2
- 238000000034 method Methods 0.000 description 26
- 230000008901 benefit Effects 0.000 description 6
- 230000008569 process Effects 0.000 description 6
- 230000002194 synthesizing effect Effects 0.000 description 6
- 238000001208 nuclear magnetic resonance pulse sequence Methods 0.000 description 5
- 238000010586 diagram Methods 0.000 description 4
- 210000001519 tissue Anatomy 0.000 description 4
- 238000002474 experimental method Methods 0.000 description 3
- 210000003625 skull Anatomy 0.000 description 3
- 210000003484 anatomy Anatomy 0.000 description 2
- 238000002591 computed tomography Methods 0.000 description 2
- 239000000284 extract Substances 0.000 description 2
- 239000012530 fluid Substances 0.000 description 2
- 238000007499 fusion processing Methods 0.000 description 2
- 238000003384 imaging method Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000012360 testing method Methods 0.000 description 2
- 208000003174 Brain Neoplasms Diseases 0.000 description 1
- 206010072731 White matter lesion Diseases 0.000 description 1
- 230000002238 attenuated effect Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 239000002872 contrast media Substances 0.000 description 1
- 210000005257 cortical tissue Anatomy 0.000 description 1
- 230000007423 decrease Effects 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000002059 diagnostic imaging Methods 0.000 description 1
- 230000009977 dual effect Effects 0.000 description 1
- 238000002592 echocardiography Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 210000004884 grey matter Anatomy 0.000 description 1
- 238000002075 inversion recovery Methods 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 238000002600 positron emission tomography Methods 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 230000011218 segmentation Effects 0.000 description 1
- 210000004872 soft tissue Anatomy 0.000 description 1
- 238000013519 translation Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
- 210000004885 white matter Anatomy 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/50—Image enhancement or restoration using two or more images, e.g. averaging or subtraction
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/25—Fusion techniques
- G06F18/253—Fusion techniques of extracted features
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10072—Tomographic images
- G06T2207/10088—Magnetic resonance imaging [MRI]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20212—Image combination
- G06T2207/20221—Image fusion; Image merging
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Life Sciences & Earth Sciences (AREA)
- Evolutionary Computation (AREA)
- General Engineering & Computer Science (AREA)
- Artificial Intelligence (AREA)
- Computing Systems (AREA)
- Mathematical Physics (AREA)
- Computational Linguistics (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Biophysics (AREA)
- Biomedical Technology (AREA)
- Health & Medical Sciences (AREA)
- Software Systems (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Evolutionary Biology (AREA)
- Magnetic Resonance Imaging Apparatus (AREA)
Abstract
The invention discloses a trans-modal MRI (magnetic resonance imaging) synthesis method based on morphological characteristics GAN (neural network), which comprises the steps of establishing an MRFE-GAN model, wherein the MRFE-GAN model comprises a residual error network module and a modal representative characteristic extraction module; the source mode acquires a pseudo target mode through a residual error network module; and extracting representative features of the pseudo target modality through a modality representative feature extraction module, combining the representative features with basic information of the source modality, and fusing to generate a synthetic target modality. The invention can obtain more real and effective target modes; the information difference between cross-domain modes can be effectively overcome, and data of different levels can be effectively extracted; the difference between the synthetic mode and the real target mode is effectively reduced, so that the synthetic image is more real and reliable.
Description
Technical Field
The invention belongs to the technical field of image processing, and particularly relates to a cross-modality MRI synthesis method based on morphological characteristics GAN.
Background
Medical imaging techniques such as Computed Tomography (CT), Magnetic Resonance Imaging (MRI), and Positron Emission Tomography (PET) are important components of modern healthcare. MRI, which can capture contrast differences in soft tissue, has become the primary imaging modality for studying neuroanatomy. By applying different pulse sequences and parameters, a wide variety of tissue contrasts can be generated when imaging the same anatomy, thereby obtaining images of different contrasts, i.e. MRI modalities. For example, by selecting pulse sequences such as magnetization-prepared gradient echoes (MPRAGE) and recalled variance gradients (SPGR), T1-weighted (T1) images can be generated that clearly depict gray and white matter tissue. In contrast, the T2-weighted (T2) image depicts fluid from cortical tissue, generated by applying a pulse sequence, such as a Dual Spin Echo (DSE). Additionally, fluid attenuated inversion recovery (FLAIR) is a T2 weighted pulse sequence that is employed to enhance image contrast of white matter lesions. Different contrast images of the same patient may provide different diagnostic information. Magnetic Resonance Imaging (MRI) techniques can produce various tissue contrasts by using different pulse sequences and parameters. The same anatomy with different tissue contrast increases the diversity of MRI information. However, obtaining multiple different contrast images (or modalities) for the same examiner is time consuming and expensive. In practice, therefore, the number of contrast agent modalities to be acquired for the same patient is always limited due to relevant factors such as limited scan time and high cost.
Although there are some synthesis methods for performing modality synthesis through a depth network, because the difference in feature distribution between a source modality and a target modality is large, even though the existing methods can know the mapping between different modalities to some extent, there may still be some differences between input and generated images, which cannot effectively reduce the differences, resulting in a large difference between a synthesized image and a real image.
Disclosure of Invention
In order to solve the problems, the invention provides a cross-modal MRI synthesis method based on morphological characteristics GAN, which can obtain a more real and effective target modality; the information difference between cross-domain modes can be effectively overcome, and the features of different layers can be effectively extracted; the difference between the synthetic mode and the real target mode is effectively reduced, so that the synthetic image is more real and reliable.
In order to achieve the purpose, the invention adopts the technical scheme that: a cross-modality MRI synthesis method based on morphological characteristics GAN comprises the following steps:
establishing an MRFE-GAN model, which comprises a residual error network module and a modal representative feature extraction module;
the source mode acquires a pseudo target mode through a residual error network module;
and extracting the representative characteristics of the pseudo target mode through a mode representative characteristic extraction module, combining the representative characteristics with the basic information of the source mode, and fusing to generate a synthetic target mode.
Further, the residual error network module takes a source mode as an input, and a pseudo target mode is formed by establishing an intermediate mode in the residual error network module to simulate a target mode. In the image synthesis task, the feature information between the input and the output is completely different, which means that a deeper network is required to improve the synthesis performance, but in the deepening process of the network, the network may be difficult to optimize, thereby reducing the performance. The residual network can effectively overcome this problem.
Further, the residual error network module comprises 3 downsampling blocks, 12 residual error blocks and 3 deconvolution layers which are operated in sequence; the downsampling blocks increase the number of feature maps from 1 to 256 and each downsampling block comprises convolution, instance normalization and ReLU layers run in sequence; the residual block comprises padding, convolution, instance normalization and ReLU layers; the deconvolution block includes deconvolution, instance normalization, ReLU layers, and activation functions. The independence of each modality instance after each convolutional layer can be effectively maintained through instance normalization.
Further, the modality representative feature extraction module in the MRFE-GAN model comprises a basic encoder, a representative encoder and a decoder; inputting a source mode into the basic encoder, and extracting basic information from the source mode by the basic encoder; inputting a pseudo-target modality into the representative encoder, and extracting representative information from the pseudo-target modality by the representative encoder; the basic encoder and the representative encoder are connected to a decoder in parallel, and the decoder fuses the basic information and the representative information to generate a synthetic target modality.
Further, in the modality representative feature extraction module, a source modality x and a pseudo target modality y are input, and the modality distributions are respectively p (x) and p (y); each mode is decomposed into two different distributions in the own space, which are respectively: p (x1| x) and P (x2| x), P (y1| y) and P (y2| y); wherein P (x1| x) and P (y1| y) are basic distributions of the structures of the respective modes, P (x2| x) and P (y2| y) are representative distributions of the structures of the respective modes, and the difference between the two modes is embodied by P (x2| x) and P (y2| y) and is a representative feature of the mode itself;
obtaining y-modal representative features based on an x-modal structure by fusing P (x1| x) and P (y2| y) at the modal representative feature extraction module; or, obtaining x modal representative features based on a y modal structure by fusing P (x2| x) and P (y1| y) at the modal representative feature extraction module.
Further, the base encoder comprises 3 downsampling blocks and 4 residual blocks which are sequentially run, wherein the downsampling blocks comprise convolution, instance normalization and ReLU layers which are sequentially run, and the residual blocks comprise filling, convolution, instance normalization and ReLU layers which are sequentially run;
the representative encoder comprises 5 sets of convolutional and ReLU layer combinations, a global average pooling layer, and 3 sets of linear layers; discarding all instance normalization layers before each ReLU layer of the representative encoder, converting each two-dimensional feature channel into a real number by utilizing global average pooling, and modeling correlation among the channels in three groups of linear layers to obtain a mean value and a standard deviation of representative information; on the basis of improving the synthesis identification precision, the problem that the original characteristic mean value and the standard deviation of important representative information are deleted due to example normalization is solved;
and the decoder combines the standard deviation and the average value obtained by the representative encoder into a proportion parameter and a shift parameter of an adaptive instance normalization layer of the decoder, so that the basic information and the representative information are fused to generate a synthetic target modality.
Further, the step of the decoder fusing the basic information and the representative information to generate the synthetic target modality includes:
first, normalization α is performed by the adaptive instance normalization layer:
wherein α is representative information of an input of the adaptive instance normalization layer, and Δ (α) and xi (α) are a standard deviation and a mean value, respectively, of the representative information α;
then, multiplying the normalized alpha' by a proportional parameter, and adding a shift parameter; and completing the fusion of two kinds of information to achieve the generation of a synthetic target mode by a reconstruction mode:
α"＝Δ′*α'+Ξ'；
wherein the scale parameter is a standard deviation of the representative information, and the shift parameter is an average of the representative information.
Further, the system also comprises a first identification module and a second identification module, wherein the first identification module is used for inputting a real target mode and a pseudo target mode into the first identification module for loss calculation; inputting a synthetic target mode and a real target mode into a second identification module for loss calculation; and combining the loss calculation of the first identification module and the loss calculation of the second identification module to form a total loss function of the MRFE-GAN model, performing optimization training on the total loss function to minimize the loss function, and optimizing a pseudo-target mode by adjusting parameters so as to improve the truth of a synthetic result.
Further, a pseudo target mode g (x) is generated in the residual error network module according to the source mode x, and the pseudo target mode g (x) is used as an input of the mode representative feature extraction module to extract representative feature information, specifically comprising the following steps:
placing the pseudo target modality g (x) in the first identification module D1 for the first loss calculation:
LRESNET(G，D1)＝Ey[logD1(y)]+Ex[log(1-D1(G(x))]；
where y is the target modality, and E is the expected value of the input and output;
the loss reconstruction loss with L1 helps the network to capture the overall appearance and relatively coarse features of the target modality in the synthetic modality:
LRESNET-L1(G)＝Ex，y[||y-G(x)||1]；
the modality represents the feature extraction module, which takes x and G (x) as input and outputs the synthesis target modality M (x, G (x)), and the synthesis target modality M (x, G (x)) is put into the identification module D2 for loss calculation: l is a radical of an alcoholMRFE(M，D2)＝Ey[logD2(y)]+Ex，G(x)[log(1-D2(M(x，G(x))))]；
The L1 penalty is used to calculate the difference between the synthetic target modality and the real target modality:
LMRFE-L1(M)＝Ex，G(x)，y[||y-M(x，G(x))||1]；
the total loss function of the MRFE-GAN model is as follows:
Ltotal＝λ1LRESNET(G，D1)+λ2LRESNET-L1(G)+λ3LMRFE(M，D2)+λ4LMRFE-L1(M)；
a countermeasure of maximizing D1 and D2 by minimizing a loss function in which λ is set during training1＝λ 31 and λ2＝λ4100; the network performance is enhanced, and a more real result is output.
The beneficial effects of the technical scheme are as follows:
the invention provides a novel framework for cross-modal MRI synthesis, a countermeasure network (GAN) is generated based on conditions, and a Modal Representative Feature Extraction (MRFE) strategy is introduced into the network to form the MRFE-GAN model provided by the invention. And extracting representative characteristics of the target mode through an MRFE-GAN model, combining the representative characteristics with basic information of the source mode, and accurately synthesizing the target mode from the source mode. The proposed method is superior to the latest image synthesis methods in both qualitative and quantitative aspects, and achieves better performance.
The present invention divides the modality into two different levels of information, called basic information and representative information, respectively, and extracts them using two different network structures. Two encoders with completely different structures are used to encode the modality into two different pieces of information, which are used to extract the basic information and the representative information, respectively. The target modality is not suitable for the test set, so that before the MRFE module, an RESNET module is provided to generate an intermediate modality as a pseudo target modality; in consideration of the concealment of the target morphology during the test, a residual network is used to synthesize intermediate results as a source of representative information. In addition, in the decoding stage, instead of directly fusing the extracted basic information and the representative information, an AdaIN layer is added to the decoder and the representative information is used as the shifting and scaling parameters thereof to complete the information fusion process, and fuse the information of two different levels. A more realistic and effective target modality is obtained. The information difference between cross-domain modes can be effectively overcome, and data of different levels can be effectively extracted; the difference between the synthetic mode and the real target mode is effectively reduced, so that the synthetic image is more real and reliable.
The MRFE-GAN model in the present invention consists of a residual network and an MRFE network that includes two encoders (i.e., a base encoder and a representative encoder) and a decoder. Compared with the traditional deep neural network, the method has the advantages that: (1) different from the traditional GAN which directly takes a source mode as input to generate a synthetic target mode, the MRFE architecture in the network of the invention simultaneously considers the information of two basic characteristic source modes and the representative characteristic information of the target mode to generate the synthetic target mode; the two encoders respectively extract information of different layers, so that on one hand, a better fusion effect can be obtained during information fusion, on the other hand, image fusion is more flexible, and different fusion images can be obtained through fusion of bottom layer information and representative information between different modules; (2) in order to extract representative information from the target mode, a residual error network (RESNET) is adopted to generate an intermediate mode as a pseudo target mode, so that a representative encoder is facilitated to encode; (3) the two kinds of information are directly fused, the standard deviation and the mean value of the representative information are used as the translation and scaling parameters of the self-adaptive instance normalization layer in the decoder, the characteristic space information fusion process mode is completed, and the fusion speed can be accelerated and the calculation cost is not brought by the conversion of the distribution form in the characteristic space; (4) after each convolutional layer, no Batch Normalization (BN) is used, but instead Instance Normalization (IN) is used to maintain the difference between different MRI mode instances.
Drawings
FIG. 1 is a schematic flow chart of a cross-modality MRI synthesis method based on morphological feature GAN of the present invention;
FIG. 2 is a schematic topological structure diagram of a cross-modality MRI synthesis method based on morphological feature GAN in an embodiment of the present invention;
FIG. 3 is a schematic diagram of a residual error network module according to an embodiment of the present invention;
FIG. 4 is a schematic diagram of a basic encoder according to an embodiment of the present invention;
FIG. 5 is a block diagram of a representative encoder in an embodiment of the present invention;
FIG. 6 is a comparison of the results of the synthesis mode of synthesizing T2 from T1 according to the present method and the prior art method;
FIG. 7 is a comparison of the results of the synthesis modality of FLAIR from T2 by the present method and by the prior art method in accordance with an embodiment of the present invention;
fig. 8 is a comparison of the synthesis mode results of the present method and the prior art method for synthesizing T2 from PD in accordance with the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is further described below with reference to the accompanying drawings.
In this embodiment, referring to fig. 1 and fig. 2, the present invention proposes a cross-modality MRI synthesis method based on morphological feature GAN,
the cross-modality MRI synthesis method based on morphological characteristics GAN comprises the following steps:
establishing an MRFE-GAN model, which comprises a residual error network module and a modal representative feature extraction module;
a source mode acquires a pseudo target mode through a residual error network module;
and extracting representative features of the pseudo target modality through a modality representative feature extraction module, combining the representative features with basic information of the source modality, and fusing to generate a synthetic target modality.
As an optimization scheme of the above embodiment, the residual error network module takes a source mode as an input, and a pseudo target mode is formed by establishing an intermediate mode in the residual error network module to simulate a target mode. In the image synthesis task, the feature information between the input and the output is completely different, which means that a deeper network is needed to improve the synthesis performance, and in the deepening process, the network is difficult to optimize, so that the performance is reduced, and therefore, the problem can be effectively overcome by using the generated intermediate mode in the residual network module.
As shown in fig. 3, the residual network module includes 3 downsampling blocks, 12 residual blocks (ResBlock), and 3 deconvolution layers, which are sequentially run; the downsampling blocks increase the number of feature maps from 1 to 256 and each downsampling block comprises successively run convolution (Conv), instance normalization (instanceNorm) and ReLU layers; the residual block comprises successively running padding (padding), convolution (Conv), instance normalization (instanceNorm), and ReLU layers; the deconvolution layer was run sequentially deconvolution (ConvT), instance normalization (instanceNorm), ReLU layer, and activation function (Tanh). The independence of each modality instance after each convolutional layer can be effectively maintained through instance normalization.
As an optimization solution of the above embodiment, the modality representative feature extraction module in the MRFE-GAN model includes a base encoder, a representative encoder, and a decoder; inputting a source mode into the basic encoder, and extracting basic information from the source mode by the basic encoder; inputting a pseudo target modality into the representative encoder, and extracting representative information from the pseudo target modality by the representative encoder; the basic encoder and the representative encoder are connected to a decoder in parallel, and the decoder fuses the basic information and the representative information to generate a synthetic target modality.
In the modal representative feature extraction module, inputting a source modal x and a pseudo target modal y', wherein modal distributions are P (x) and P (y), respectively; each mode is decomposed into two different distributions in the own space, which are respectively: p (x1| x) and P (x2| x), P (y1| y) and P (y2| y); wherein P (x1| x) and P (y1| y) are basic distributions of the structures of the respective modes, P (x2| x) and P (y2| y) are representative distributions of the structures of the respective modes, and the difference between the two modes is embodied by P (x2| x) and P (y2| y), and the modes have representative characteristics of their own;
obtaining y' modal representative features based on an x modal structure by fusing P (x1| x) and P (y2| y) at the modal representative feature extraction module; or, obtaining x modal representative features based on a y modal structure by fusing P (x2| x) and P (y1| y) at the modal representative feature extraction module.
As shown in fig. 4, the base encoder includes 3 downsample blocks and 4 residual blocks, which are sequentially run, the downsample blocks including sequentially run convolution (Conv), instance normalization (instanceNorm), and ReLU layers, and the residual blocks including sequentially run padding (padding), convolution (Conv), instance normalization (instanceNorm), and ReLU layers;
as shown in fig. 5, the representative encoder includes 5 sets of convolutional and ReLU layer combinations, a global average pooling layer (adaptive avgpool), and 3 sets of Linear layers (Linear); discarding all instance normalization layers before each ReLU layer of the representative encoder, converting each two-dimensional feature channel into a real number by utilizing global average pooling, and modeling correlation among the channels in three groups of linear layers to obtain a mean value and a standard deviation of representative information; on the basis of improving the synthesis identification precision, the problem of deleting the original characteristic mean value and standard deviation of important representative information caused by case normalization is solved;
and the decoder combines the standard deviation and the average value obtained by the representative encoder into a proportion parameter and a shift parameter of an adaptive instance normalization layer of the decoder, so that the basic information and the representative information are fused to generate a synthetic target modality.
The step of the decoder fusing the basic information and the representative information to generate the synthetic target modality comprises the following steps:
first, normalization α is performed by the adaptive instance normalization layer:
wherein α is representative information of the input to the adaptive instance normalization layer, Δ (α) and xi (α) are respectively a standard deviation and a mean value representative of the information α;
then, multiplying the normalized alpha by a proportion parameter, and adding a shift parameter; completing the fusion of two kinds of information, and achieving the reconstruction mode to generate a synthetic target mode:
α″＝Δ′*α′+Ξ′；
wherein the scale parameter is a standard deviation of the representative information, and the shift parameter is an average of the representative information.
As an optimization scheme of the above embodiment, the system further includes a first identifying module and a second identifying module, and the first identifying module inputs a real target mode and a pseudo target mode to perform loss calculation; inputting a synthetic target mode and a real target mode into a second identification module for loss calculation; and combining the loss calculation of the first identification module and the loss calculation of the second identification module to form a total loss function of the MRFE-GAN model, performing optimization training on the total loss function to minimize the loss function, and optimizing a pseudo-target mode by adjusting parameters so as to improve the truth of a synthetic result.
Generating a pseudo target mode G (x) in the residual error network module according to the source mode x, and extracting representative feature information by taking the pseudo target mode G (x) as one input of a mode representative feature extraction module, wherein the specific process comprises the following steps:
placing the pseudo target modality g (x) in the first identification module D1 for the first loss calculation:
LRESNET(G，D1)＝Ey[logD1(y)]+Ex[log(1-D1(G(x))]；
where y is the target modality, and E is the expected value of the input and output;
the loss reconstruction loss with L1 helps the network to capture the overall appearance and relatively coarse features of the target modality in the synthetic modality:
LRESNET-L1(G)＝Ex，y[||y-G(x)||1]；
the mode representative feature extraction module takes x and G (x) as input and outputs a synthesis target mode M (x, G (x)), and the synthesis target mode M (x, G (x)) is put into an identification module D2 for loss calculation:
LMRFE(M，D2)＝Ey[logD2(y)]+Ex，G(x)[log(1-D2(M(x，G(x))))]；
the difference between the synthetic target modality and the real target modality is calculated using the L1 penalty:
LMRFE-L1(M)＝Ex，G(x)，y[||y-M(x，G(x))||1]；
the total loss function of the MRFE-GAN model is as follows:
Ltotal＝λ1LRESNET(G，D1)+λ2LRESNET-L1(G)+λ3LMRFE(M，D2)+λ4LMRFE-L1(M)；
countermeasure mode for maximizing D1 and D2 by minimizing loss function in training process, wherein λ is set in loss function1＝λ 31 and λ2＝λ4100; and the network performance is enhanced, and a more real result is output.
The advantages of the present method were verified by performing extensive experiments, which were mainly performed on a multimodal brain tumor segmentation (BRATS) dataset 1 and a non-peeled image (I × I) dataset 2. To evaluate the proposed model and compare it with the two most recent comparison methods. In particular, since the main contribution of the proposed model is to introduce the representation information from the target modality for synthesis, the advantages of the representation encoder in the MRFE module were first studied. The performance of the model in synthesizing T2 from the T1 modality and FLAIR from the T2 modality was then studied using the BRATS dataset. Finally, the synthetic target modality is displayed in the image where no skull stripping has been performed on the I dataset.
To verify the advantage of using a representative encoder in the MRFE-GAN model: experiments were performed by comparing (MRFE-GAN w/oRE) between the MRFE-GAN model using a representative encoder and the MRFE-GAN model without a representative encoder. Quantitative comparison results of the BRATS dataset (T1 to T2) and the I × I dataset (PD to T2) are given in table 1 and table 2, with standard deviations in parentheses. As shown in tables 1 and 2, the results of combining the representative information indicate that the peak signal-to-noise ratio PSNR and the structural similarity SSIM increase to some extent and the standard root mean square error NRMSE decreases to some extent on both data sets, compared with the structure without a representative encoder. This means that representative information provided by the target modality is advantageous for improving the performance of modality synthesis. The extraction of the representation information by the representation encoder is necessary to improve the synthesis performance in the single-modality synthesis task.
TABLE 1
TABLE 2
The performance of the model synthesizing T2 from T1 was verified by using the method proposed by the present invention on the BRATS data set. Fig. 6 gives an example of the synthesis of the target modality using different methods, from which it can be seen that the proposed model (MRFE-GAN) yields a higher visual quality in terms of structure and details (as indicated by the arrows) than the existing mlr and p-GAN models. The model result provided by the invention can better retain the detailed information in the synthesized T2 image. For quantitative comparison, the average synthetic target modality obtained by the different methods is listed in table 3; the table shows that all three indices obtained by the process of the invention are significantly improved, especially with respect to the PSNR values, compared to the other two processes.
TABLE 3
The performance of the model for synthesizing T2 from PD was verified by using the proposed method of the present invention on IXI data set. In the conventional detection, since a dark skull region surrounded by a bright skin or fat region causes intensity unevenness in an MRI image, it is difficult to synthesize an image that is not stumbled on the skull. In this experiment, the IXI dataset for the non-peeled image was synthesized from PD to T2. As can be seen in fig. 8, the model of the present invention tends to better preserve the detailed information in the synthesized T2 image with minimal differences compared to the two more recent methods, MILR and P-GAN. Table 5 shows the results of a quantitative comparison of the different methods, and it can be seen that all three criteria of the method of the invention are significantly improved compared to MILR and P-GAN.
TABLE 5
The foregoing shows and describes the general principles and features of the present invention, together with the advantages thereof. It will be understood by those skilled in the art that the present invention is not limited to the embodiments described above, which are given by way of illustration of the principles of the present invention, but that various changes and modifications may be made without departing from the spirit and scope of the invention, and such changes and modifications are within the scope of the invention as claimed. The scope of the invention is defined by the appended claims and equivalents thereof.
Claims (3)
1. The cross-modality MRI synthesis method based on morphological characteristics GAN is characterized by comprising the following steps:
establishing an MRFE-GAN model, which comprises a residual error network module and a modal representative feature extraction module;
the source mode acquires a pseudo target mode through a residual error network module;
extracting representative characteristics of a pseudo target mode through a mode representative characteristic extraction module, combining the representative characteristics with basic information of a source mode, and fusing to generate a target mode;
the residual error network module takes a source mode as input, and a pseudo target mode is formed by establishing an intermediate mode in the residual error network module to simulate a target mode;
the residual error network module comprises 3 downsampling blocks, 12 residual error blocks and 3 deconvolution layers which operate in sequence; the downsampling blocks increase the number of feature maps from 1 to 256 and each downsampling block comprises convolution, instance normalization and ReLU layers run in sequence; the residual block comprises a filling layer, a convolution layer, an instance normalization layer and a ReLU layer which are sequentially operated; the deconvolution layer sequentially runs deconvolution, instance normalization, a ReLU layer and an activation function;
the mode representative feature extraction module in the MRFE-GAN model comprises a basic encoder, a representative encoder and a decoder; inputting a source mode into the basic encoder, and extracting basic information from the source mode by the basic encoder; inputting a pseudo-target modality into the representative encoder, and extracting representative information from the pseudo-target modality by the representative encoder; the basic encoder and the representative encoder are connected to a decoder in parallel, and basic information and representative information are fused through the decoder to generate a synthetic target modality;
in the modal representative feature extraction module, inputting a source modal x and a pseudo target modal y', wherein modal distributions are respectively P (x) and P (y); each mode is decomposed into two different distributions in the own space, which are respectively: p (x1| x) and P (x2| x), P (y1| y) and P (y2| y); wherein P (x1| x) and P (y1| y) are the basic distributions of the respective modal structures, and P (x2| x) and P (y2| y) are the representation distributions of the respective modal structures; the difference between the two modalities is embodied by P (x2| x) and P (y2| y), which can be regarded as representative features of the modalities themselves;
obtaining y' modal representative features based on an x modal structure by fusing P (x1| x) and P (y2| y) at the modal representative feature extraction module; or obtaining x modal representative features based on a y modal structure by fusing P (x2| x) and P (y1| y) at the modal representative feature extraction module;
wherein the base encoder comprises 3 downsampling blocks and 4 residual blocks which are run in sequence, the downsampling blocks comprise convolution, instance normalization and ReLU layers which are run in sequence, and the residual blocks comprise padding, convolution, instance normalization and ReLU layers which are run in sequence;
the representative encoder comprises 5 groups of convolutional layers and ReLU layer combinations, a global average pooling layer, and 3 groups of linear layers; discarding all instance normalization layers before each ReLU layer of the representative encoder, converting each two-dimensional characteristic channel into a real number by utilizing global average pooling, and modeling the correlation among the channels in three groups of linear layers to obtain the average value and the standard deviation of the representative information;
the decoder combines the standard deviation and the average value obtained by the representative encoder into a proportion parameter and a shift parameter of an adaptive instance normalization layer of the decoder, so that basic information and representative information are fused to generate a synthetic target mode;
the step of the decoder fusing the basic information and the representative information to generate the synthetic target modality comprises the following steps:
first, a is normalized by the adaptive instance normalization layer:
wherein α is representative information of the input to the adaptive instance normalization layer, Δ (α) and xi (α) are respectively a standard deviation and a mean value representative of the information α;
then, multiplying the normalized alpha' by a proportional parameter, and adding a shift parameter; completing the fusion of two kinds of information, and achieving the reconstruction mode to generate a synthetic target mode:
α″＝Δ′*α′+Ξ′：
wherein, the scale parameter is a standard deviation of the representative information extracted by the morphological coder, and the shift parameter is an average value of the representative information.
2. The modality-signature GAN-based cross-modality MRI synthesis method of claim 1, further comprising two discriminator modules; inputting a real target mode and a pseudo target mode into a first discriminator module for loss calculation; inputting a synthetic target mode and a real target mode into a second discriminator module for loss calculation; and calculating to form a total loss function of the MRFE-GAN model by combining the losses of the first discriminator module and the second discriminator module, performing optimization training on the total loss function to minimize the loss function, and optimizing the synthesized target mode by adjusting parameters.
3. The cross-modality MRI synthesis method based on morphological feature GAN as claimed in claim 2, wherein the residual network module generates a pseudo-target modality G (x) according to a source modality x, and uses the pseudo-target modality G (x) as an input of the modality representative feature extraction module to extract representative feature information, which comprises the following steps:
placing the pseudo target modality g (x) in the first identification module D1 for the first loss calculation:
LRESNET(G，D1)＝Ey[logD1(y)]+Ex[log(1-D1(G(x))]；
where y is the target modality, and E is the expected value of the input and output;
the loss reconstruction loss with L1 helps the network to capture the overall appearance and relatively coarse features of the target modality in the synthetic modality:
LRESNET-L1(G)＝Ex，y[||y-G(x)||1]；
the mode representative feature extraction module takes x and G (x) as input and outputs a synthesis target mode M (x, G (x)), and the synthesis target mode M (x, G (x)) is put into an identification module D2 for loss calculation:
LMRFE(M，D2)＝Ey[logD2(y)]+Ex，G(x)[log(1-D2(M(x，G(x))))]；
the L1 penalty is used to calculate the difference between the synthetic target modality and the real target modality:
LMRFE-L1(M)＝Ex，G(x)，y[||y-M(x，G(x))||1]；
the total loss function of the MRFE-GAN model is as follows:
Ltotal＝λ1LRESNET(G，D1)+λ2LRESNET-L1(G)+λ3LMRFE(M，D2)+λ4LMRFE-L1(M)；
a countermeasure of maximizing D1 and D2 by minimizing a loss function in which λ is set during training1＝λ31 and λ2＝λ4＝100。
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911113248.8A CN110827232B (en) | 2019-11-14 | 2019-11-14 | Cross-modality MRI (magnetic resonance imaging) synthesis method based on morphological characteristics GAN (gamma GAN) |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911113248.8A CN110827232B (en) | 2019-11-14 | 2019-11-14 | Cross-modality MRI (magnetic resonance imaging) synthesis method based on morphological characteristics GAN (gamma GAN) |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110827232A CN110827232A (en) | 2020-02-21 |
| CN110827232B true CN110827232B (en) | 2022-07-15 |
Family
ID=69555297
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911113248.8A Expired - Fee Related CN110827232B (en) | 2019-11-14 | 2019-11-14 | Cross-modality MRI (magnetic resonance imaging) synthesis method based on morphological characteristics GAN (gamma GAN) |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110827232B (en) |
Families Citing this family (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP3872754A1 (en) * | 2020-02-28 | 2021-09-01 | Siemens Healthcare GmbH | Method and system for automated processing of images when using a contrast agent in mri |
| CN111524147B (en) * | 2020-04-14 | 2022-07-12 | 杭州健培科技有限公司 | Brain tumor segmentation method based on generative confrontation network |
| CN111797891B (en) * | 2020-05-21 | 2024-08-23 | 南京大学 | Method and device for generating unpaired heterogeneous face image based on generation countermeasure network |
| CN111862261B (en) * | 2020-08-03 | 2022-03-29 | 北京航空航天大学 | FLAIR modal magnetic resonance image generation method and system |
| CN113012086B (en) * | 2021-03-22 | 2024-04-16 | 上海应用技术大学 | Cross-modal image synthesis method |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2002076489A1 (en) * | 2001-03-09 | 2002-10-03 | Dyax Corp. | Serum albumin binding moieties |
| CN109213876A (en) * | 2018-08-02 | 2019-01-15 | 宁夏大学 | Based on the cross-module state search method for generating confrontation network |
| CN110047056A (en) * | 2018-01-16 | 2019-07-23 | 西门子保健有限责任公司 | With the cross-domain image analysis and synthesis of depth image to image network and confrontation network |
| CN110210422A (en) * | 2019-06-05 | 2019-09-06 | 哈尔滨工业大学 | It is a kind of based on optical imagery auxiliary naval vessel ISAR as recognition methods |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20030083652A1 (en) * | 2001-10-31 | 2003-05-01 | Oratec Interventions, Inc | Method for treating tissue in arthroscopic environment |
-
2019
- 2019-11-14 CN CN201911113248.8A patent/CN110827232B/en not_active Expired - Fee Related
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2002076489A1 (en) * | 2001-03-09 | 2002-10-03 | Dyax Corp. | Serum albumin binding moieties |
| CN110047056A (en) * | 2018-01-16 | 2019-07-23 | 西门子保健有限责任公司 | With the cross-domain image analysis and synthesis of depth image to image network and confrontation network |
| CN109213876A (en) * | 2018-08-02 | 2019-01-15 | 宁夏大学 | Based on the cross-module state search method for generating confrontation network |
| CN110210422A (en) * | 2019-06-05 | 2019-09-06 | 哈尔滨工业大学 | It is a kind of based on optical imagery auxiliary naval vessel ISAR as recognition methods |
Non-Patent Citations (5)
| Title |
|---|
| Prevalence of spinal pathology in patients presenting for lumbar MRI as referred from general practice;Evelien de Schepper等;《Family Practice》;20160228;第33卷(第1期);51-56 * |
| Statistical iterative reconstruction using adaptive fractional order regularization;Yan Wang 等;《Biomedical Optics Express》;20160301;第7卷(第3期);1015-1029 * |
| 不同分子亚型乳腺癌的MRI和病理特征初探;于洋 等;《中华放射学杂志》;20140310;第48卷(第3期);184-188 * |
| 基于U-net模型的全自动鼻咽肿瘤MR图像分割;潘沛克 等;《计算机应用》;20181116;第39卷(第4期);1183-1188 * |
| 基于相似度的双搜索多目标识别算法;冷何英等;《红外与激光工程》;20021225(第06期);3-6 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110827232A (en) | 2020-02-21 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110827232B (en) | Cross-modality MRI (magnetic resonance imaging) synthesis method based on morphological characteristics GAN (gamma GAN) | |
| Liu et al. | Multimodal MR image synthesis using gradient prior and adversarial learning | |
| Zhan et al. | Multi-modal MRI image synthesis via GAN with multi-scale gate mergence | |
| CN110288609B (en) | Multi-modal whole-heart image segmentation method guided by attention mechanism | |
| CN113554669B (en) | Unet network brain tumor MRI image segmentation method with improved attention module | |
| CN114266939B (en) | Brain extraction method based on ResTLU-Net model | |
| CN112365556B (en) | Image extension method based on perception loss and style loss | |
| CN116188452A (en) | Medical image interlayer interpolation and three-dimensional reconstruction method | |
| CN112634265B (en) | Method and system for constructing and segmenting fully-automatic pancreas segmentation model based on DNN (deep neural network) | |
| CN113674330A (en) | Pseudo CT image generation system based on generation countermeasure network | |
| CN110782427A (en) | Magnetic resonance brain tumor automatic segmentation method based on separable cavity convolution | |
| CN114170244A (en) | Brain glioma segmentation method based on cascade neural network structure | |
| CN117437420A (en) | Cross-modal medical image segmentation method and system | |
| CN117333750A (en) | Spatial registration and local global multi-scale multi-modal medical image fusion method | |
| CN115861464A (en) | Pseudo CT (computed tomography) synthesis method based on multimode MRI (magnetic resonance imaging) synchronous generation | |
| CN117974832B (en) | Multi-modal liver medical image expansion algorithm based on generation countermeasure network | |
| CN115496732A (en) | Semi-supervised heart semantic segmentation algorithm | |
| CN117769722A (en) | Generating a lossless image compression model for tomographic imaging based on an antagonism network | |
| CN109741439B (en) | Three-dimensional reconstruction method of two-dimensional MRI fetal image | |
| CN112785540B (en) | Diffusion weighted image generation system and method | |
| CN117237627A (en) | Multi-mode MRI brain tumor image segmentation method | |
| CN114820636A (en) | Three-dimensional medical image segmentation model and training method and application thereof | |
| CN115691770A (en) | Cross-modal medical image completion method, device and equipment based on condition score | |
| CN115908451A (en) | Heart CT image segmentation method combining multi-view geometry and transfer learning | |
| CN114332271A (en) | Dynamic parameter image synthesis method and system based on static PET image |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CF01 | Termination of patent right due to non-payment of annual fee | ||
| CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20220715 |