WO2022120731A1

WO2022120731A1 - Mri-pet image modality conversion method and system based on cyclic generative adversarial network

Info

Publication number: WO2022120731A1
Application number: PCT/CN2020/135319
Authority: WO
Inventors: 胡战利; 郑海荣; 张娜; 刘新; 杨永峰; 梁栋; 唐政
Original assignee: 深圳先进技术研究院
Priority date: 2020-12-10
Filing date: 2020-12-10
Publication date: 2022-06-16

Abstract

The present application relates to an MRI-PET image modality conversion method and system based on a cyclic generative adversarial network. The method comprises the following steps: obtaining an MRI image data set and a PET image data set, and constructing an input data set from the MRI image data set and the PET image data set; constructing a cyclic generative adversarial network model, and performing adversarial training on the cyclic generative adversarial network model by means of the input data set; training and generating the cyclic generative adversarial network model, and gradually reaching a convergence state; and performing modality conversion processing on an MRI image to a PET image by using the trained cyclic generative adversarial network model. According to the present application, by applying the method and system, the blank in the field of generating a PET image by an MRI image is filled, and the problem of low resolution of PET generated images is solved.

Description

MRI-PET image modality conversion method and system based on recurrent generative adversarial network

technical field

The present application relates to the field of imaging technologies, and in particular, to a method and system for MRI-PET image modality conversion based on a recurrent generative adversarial network.

Background technique

Positron Emission Computed Tomography (PET) is a kind of radioactive elements with short half-life such as F18 and C11 that use substances necessary for human metabolism, such as glucose, protein, nucleic acid, etc., to be injected into the human body. By observing its aggregation in the process of human tissue metabolism, it can reflect the metabolism of the tissue, so as to achieve the purpose of diagnosis. The most commonly used is fluorodeoxyglucose (Fludeoxyglucose, 18F-FDG). PET imaging requires the injection of radioisotopes into the human body for imaging, so there are certain operational risks and a certain dose of radiation to the patient.

Magnetic resonance imaging (MR) is a type of tomography, which uses the magnetic resonance phenomenon of tissue under strong magnetic field to obtain electromagnetic signals of tissue, and reconstruct human tissue accordingly. MR has excellent performance for imaging soft tissue structures, and can directly obtain native 3D cross-sectional imaging without the reconstruction step of the image. Unlike nuclear medicine imaging methods, MR does not involve any ionizing radiation in the imaging process, so it does not cause any form of radiation to the patient.

At present, the diagnosis of many brain diseases such as Alzheimer's disease (AD) and Parkinson's disease (PD) relies on the comprehensive information provided by multimodal neuroimaging (such as MRI and PET). However, when conducting multimodal information research, it is inevitable to encounter the lack of image data, such as the public database Alzheimer's Disease Neuroimaging Initiative (ADNI) database often has missing PET data for the same patient. . Due to the low popularity of PET equipment, the lack of PET image data is an inevitable problem, but if only cases with both modalities are studied, the data available for deep learning model training will be greatly reduced, which will seriously affect Model training results and diagnostic performance.

Therefore, the research and development of different modality tomographic image conversion methods from MR to PET has important scientific significance and broad application prospects for the current medical diagnosis field.

SUMMARY OF THE INVENTION

In order to fill the gap in the field of generating PET images from MRI images, the present application provides an MRI-PET image modality conversion method based on a recurrent generative adversarial network.

A kind of MRI-PET image modality conversion method based on cyclic generative confrontation network provided by the application adopts the following technical scheme:

A MRI-PET image modality conversion method based on recurrent generative adversarial network, comprising the following steps:

Acquire an MRI image dataset and a PET image dataset, and construct an input dataset from the MRI image dataset and the PET image dataset;

Build a recurrent generative adversarial network model, and conduct adversarial training on the recurrent generative adversarial network model with the input data set;

Train the generation cycle to generate the adversarial network model, and gradually reach the convergence state;

A trained recurrent generative adversarial network model is used to perform modality conversion processing from MRI images to PET images.

Optionally, the constructed recurrent generative adversarial network model includes a pair of first generators, a pair of second generators, a first discriminator and a second discriminator, wherein the recurrent generative adversarial network model is performed on the input data set. Adversarial training includes the following steps:

generating, by the first generator, the MRI image as a PET generated image;

generating, by the second generator, the PET image into an MRI generated image;

Distinguish whether the PET generated image is a PET image by the first discriminator, and output the first discrimination result to the first generator;

Distinguish whether the MRI generated image is an MRI image by the second discriminator, and output the second discrimination result to the second generator;

The first generator and the second generator perform the next iteration according to the first discrimination result and the second discrimination result, until the first discriminator and the second discriminator cannot distinguish the authenticity of the PET-generated image and the MRI-generated image.

Optionally, the first generator and the second generator are constructed based on an improved U-Net model, wherein the improved U-Net model adopts a self-attention unit to replace the skip connection step in the original U-Net model. Crop and expand step.

Optionally, the self-attention unit is designed as a cross self-attention sub-module.

Optionally, the first discriminator and the second discriminator output the first discrimination result and the second discrimination result, including the following steps:

The PET image and the PET-generated image are input into the first discriminator as input pictures, and the MRI image and the MRI-generated image are input into the second discriminator as input pictures;

apply a wavelet affine transform layer inline in the first discriminator and the second discriminator;

In the wavelet affine transformation layer, the input image is extracted through two convolution layers to obtain a feature map with spatial domain, and the input image is subjected to Haar wavelet transform at the same time to obtain a wavelet domain feature set. And the wavelet domain feature set is input into the affine transformation layer to obtain a feature map with spatial domain and wavelet domain features;

The above-mentioned wavelet affine transformation layer is repeated at least twice, and finally through the Softmax function, the first discriminator outputs the first discrimination result, and the second discriminator outputs the second discrimination result.

Optionally, the convolution layer uses a ReLU function as an activation function.

Optionally, the recurrent generative adversarial network model includes an adversarial loss function and a cycle consistency loss function.

Optionally, the adversarial loss function is determined by the first discrimination result and the second discrimination result of the first discriminator and the second discriminator, and the specific form is as follows:

L( _GMRI-PET _,DPET , _IMRI , _IPET )=CE(DPET( _GMRI-PET ₍ _IMRI )),label);

L( _GPET-MRI _,DMRI , _IPET , _IMRI )=CE(DMRI( _GPET _-MRI ₍ IPET)),label);

Among them, G represents the generator that generates another modal image from one modal image, D represents the discriminator that distinguishes whether the modal image is the generated image, I represents the image of this modal, CE represents the Softmax The function is the cross-entropy function of the activation function, and the label is the real label used for evaluation.

Optionally, the cycle consistency loss function has the following specific form:

where μ and σ represent the mean and standard deviation of the image, respectively, and C ₁ =(k ₁ L) ² and C ₂ =(k ₂ L) ² are two smaller constant terms, avoiding a denominator of 0, where L represents the maximum pixel value of the image;

Among them, the image length is m,n, I(i,j)K(i,j) is the pixel value corresponding to the two input images;

L _cyccon = μ _SSIM L _SSIM (G _MRI-PET , G _PET-MRI )+μ _PSNR L _PSNR (G _MRI-PET , G _PET-MRI );

where μ is the control loss function value that conforms to the cycle consistency constant parameter term;

From this, the mathematical expression of the global loss function is obtained:

In order to fill the gap in the field of generating PET images from MRI images, the present application provides an MRI-PET image modality conversion system based on a recurrent generative adversarial network.

A kind of MRI-PET image modality conversion system based on cyclic generative adversarial network provided by this application adopts the following technical scheme:

An MRI-PET image modality conversion system based on recurrent generative adversarial network, including:

an acquisition module, acquiring an MRI image dataset and a PET image dataset, and constructing an input dataset from the MRI image dataset and the PET image dataset;

Building modules, constructing a recurrent generative adversarial network model, and conducting adversarial training on the recurrent generative adversarial network model with the input data set;

Training module, training and generating a cyclic generative adversarial network model, and gradually reach a state of convergence;

The verification module uses the trained recurrent generative adversarial network model to perform modality conversion processing from MRI images to PET images.

To sum up, the present application includes at least one of the following beneficial technical effects:

1. The present application performs modal conversion processing from MRI images to PET images based on recurrent generative adversarial networks. By combining the self-attention unit and the wavelet affine transformation layer, they are applied in the generator and the discriminator respectively, which greatly improves the characteristics and characteristics of the generated images. The utilization of wavelet domain features, and the occupancy ratio of GPU memory during training is reduced;

2. This application designs a joint loss function based on the SSIM function and the PSNR function. On the basis of the traditional cycle consistency comparison, the requirements for the generated image structure and signal-to-noise ratio are added, which significantly improves the quality of the PET generated image.

Description of drawings

FIG. 1 is a schematic flowchart of an MRI-PET image modality conversion method based on a recurrent generative adversarial network in an embodiment of the present application;

2 is a schematic structural diagram of a cyclic generative adversarial network model in an embodiment of the present application;

3 is a schematic structural diagram of a first generator or a second generator in an embodiment of the present application;

4 is a schematic structural diagram of a cross self-attention sub-module in an embodiment of the present application;

5 is a schematic structural diagram of a first discriminator or a second discriminator in an embodiment of the present application;

6 is a schematic structural diagram of a wavelet affine transformation layer in an embodiment of the present application;

FIG. 7 is a system block diagram of an MRI-PET image modality conversion system based on a recurrent generative adversarial network in an embodiment of the present application.

Detailed ways

The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, but not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

In related technologies, the diagnosis of many brain diseases such as Alzheimer's disease (AD) and Parkinson's disease (PD) currently needs to rely on the comprehensive information provided by multimodal neuroimaging (such as MRI and PET). However, when conducting multimodal information research, it is inevitable to encounter the lack of image data, such as the public database Alzheimer's Disease Neuroimaging Initiative (ADNI) database often has missing PET data for the same patient. . Due to the low popularity of PET equipment, the lack of PET image data is an inevitable problem, but if only cases with both modalities are studied, the data available for deep learning model training will be greatly reduced, which will seriously affect Model training results and diagnostic performance. Therefore, the research and development of different modality tomographic image conversion methods from MR to PET has important scientific significance and broad application prospects for the current medical diagnosis field.

Y.S.Pan et al. and the 2018 paper "Synthesizing Missing PET from MRI with Cycle-consistent Generative Adversarial Networks for Alzheimer's Disease Diagnosis" utilized Cycle-Gan networks to attempt to generate PET images from MRI images, using imaging data from the ADNI public database The feasibility of generating PET images from MRI images is demonstrated.

X.Dong et al. published the article "Synthetic CT generation from non-attenuation corrected PET images for whole-body PET imaging" in the journal Physics in Medicine & Biology in 2019. The article applies 3D recurrent generative adversarial networks to the field of low-dose CT imaging , where the input pixel block is 64*64*64, and the generator uses a U-Net network based on self-attention unit, in which the self-attention unit helps the network to identify the image block with the largest amount of information, so as to achieve better image reduction noise.

The article "Synthesized 7T MRI from 3T MRI via deep learning in spatial and wavelet domains" published in the journal Medical Image Analysis in 2019 by L.Q.Qu et al. used wavelet transform to achieve effective multi-scale reconstruction, while using low-frequency tissue angiography and High frequency anatomical details. The network structure combines the wavelet affine transform, and uses the Haar wavelet transform method to modulate the feature map of the spatial domain information in the wavelet domain.

For the above-mentioned related technologies, it can be seen that there are the following defects:

1. Most of the existing image modal conversions focus on the two modalities of MRI images and CT images, and very few of them perform PET image information processing.

2. Due to the PET imaging technology, the resolution of PET images is generally not high and contains little spatial information, which increases the difficulty of training deep learning models.

3. The existing image modality conversion work from MRI images to PET images uses the public database ADNI. Since this database comes from data collected by hospitals around the world, the image data is uneven and it is difficult to find MR images\PET with a large amount of data. The image matching data set uses the traditional Cycle-GAN model, which is not improved on the basis of the model, so the training effect is not ideal.

Based on the above content, in order to fill the gap in the field of PET image generation from MRI images and solve the problem of low resolution of PET generated images, this application designs an MRI-PET image modality conversion method and system based on a recurrent generative adversarial network. This application will realize the pixel-to-pixel conversion of MRI images to PET images based on the recurrent generative adversarial network, and obtain PET images with clinical diagnostic value through imaging methods combined with deep learning models.

Referring to FIG. 1 , a method for MRI-PET image modality conversion based on a recurrent generative adversarial network proposed in this application includes the following steps:

Step S100, acquiring an MRI image data set and a PET image data set, and constructing an input data set from the MRI image data set and the PET image data set.

Step S200, constructing a cyclic generative adversarial network model, and performing adversarial training on the cyclic generative adversarial network model by using the input data set.

According to the technical solution defined in step S200, specifically, the cyclic generative adversarial network model constructed in the present application will conduct adversarial training on the cyclic generative adversarial network model by using the input data set, and the adversarial training can make the cyclic generative adversarial network model obtain reinforcement learning , compared with the common generative adversarial network processing, the recurrent generative adversarial network model of the present application does not require a large number of registered MRI images and PET images, and the synthesis speed is faster.

Referring to FIG. 2 , the recurrent generative adversarial network model in this embodiment of the present application includes a pair of first generators, a pair of second generators, a first discriminator, and a second discriminator, wherein the pair of Adversarial training of a recurrent generative adversarial network model includes the following steps:

Step S210, the MRI image is generated as a PET generated image by the first generator;

Step S220, the PET image is generated as an MRI generated image by the second generator;

Step S230, distinguish whether the PET generated image is a PET image by the first discriminator, and output the first discrimination result to the first generator;

Step S240, distinguish whether the MRI generated image is an MRI image by the second discriminator, and output the second discrimination result to the second discriminator;

Step S250, the first generator and the second generator perform the next iteration according to the first discrimination result and the second discrimination result, until the first discriminator and the second discriminator cannot distinguish the authenticity of the PET-generated image and the MRI-generated image. sex.

In the adversarial training process of the recurrent generative adversarial network model, the generator (referring to the above-mentioned first generator and second generator) completes the mutual mapping between the MRI image and the PET image, and the discriminator (referring to the above-mentioned first discrimination and a second discriminator) are used to distinguish between genuine and false images generated by MRI and PET.

The so-called confrontation refers to the confrontation between the generator and the discriminator. The generator learns the data distribution of the real data, the discriminator distinguishes the true and false from the real data and the data generated by the generator, and the generator expects the generated data as much as possible. To fool the discriminator, and the discriminator expects to be able to discern the data generated by the generator, thus creating an adversary. The so-called cycle refers to the above-mentioned pair of generators (referring to the above-mentioned first generator and second generator) and a pair of discriminators (referring to the above-mentioned first discriminator and second discriminator). loop structure.

Therefore, the generator and the discriminator continue to play games in the generation and confrontation, learn together, and gradually reach the Nash equilibrium. In the end, the data generated by the generator is enough to mix the fake with the real, so that the discriminator cannot distinguish between the real and the fake.

Referring to Figure 3, the first generator and the second generator are constructed based on the improved U-Net model. The original U-Net model consists of an encoder and a decoder. There are skip connections between them. The main function of skip connections is to transmit some low-level image information directly through the network between the input and output. The improved U-Net model in the embodiment of the present application uses a self-attention unit to replace the clipping and expansion step in the skip connection step in the original U-Net model. The self-attention unit can effectively utilize the most important information features in the feature map, effectively improving different Dimensional information can better complete the fusion of global information and local information. Therefore, using the self-attention unit can also obtain a feature map of the same size as the expansion path, and can improve the utilization of global features and local features. Reduce the amount of calculation At the same time, it can greatly improve the performance of the model.

Referring to FIG. 4 , in the embodiment of the present application, the self-attention unit is designed as a cross self-attention sub-module, and the cross-cross self-attention sub-module can obtain non-local features from the horizontal and vertical directions, which is different from the traditional attention. The difference between the modules is that the cross self-attention sub-module reduces the original H*W feature map to H+W-1, which can aggregate pixels in the horizontal and vertical directions, enhance the representation ability at the pixel level, and greatly reduce the model GPU memory and algorithm complexity occupied during training.

The following describes the cross self-attention sub-module in conjunction with FIG. 4 .

Given a feature map I(i,j) of size C×H×W, where the elements are defined as i _i,j , X(i,j) is obtained after dimension reduction by two convolution filters of size 1x1 first and Y(i,j), where C' is less than C. The AFFINE operation is as follows. Select the element x(i,j) at any position in X(i,j), and select the row element and column of the column where the element is located in Y(i,j). combination of elements

Then pass a 1×1 convolutional unit filter and utilize a softmax unit to obtain the feature weight map, where at position u, the feature value of channel t is defined as f _t,u . Z(i,j) is obtained by using another 1×1 convolution unit filter, and any element z(i,j) defines its corresponding column vector and row vector as

The AGGREGATE operation is defined as follows:

The cross self-attention sub-module constructed above only calculates feature vectors in the vertical and horizontal directions, and the cross-sub-self-attention sub-module is designed as the above self-attention unit twice using the cross-sub-self-attention sub-module, which can obtain global correlation and can greatly Reduce the amount of calculation and the memory space occupied.

The structural design and identification method of the first discriminator and the second discriminator are described below.

5, the first discriminator and the second discriminator output the first discrimination result and the second discrimination result, including the following steps:

According to the technical solution described in the above steps, specifically, as shown in FIG. 6 , the wavelet affine transformation layer comes from the conditional normalization method, and uses the affine map learned by the model to transform the feature map. Conditional normalization methods have been shown to be very effective in improving the performance of the model on the task of image style transfer.

Haar wavelet transform uses Haar function as wavelet function for wavelet packet transform.

In Haar wavelet transform, four wavelet domain feature sets including low frequency and high frequency can be obtained after the input image is processed by Haar wavelet transform. The Averaging step is to average the adjacent pixels, the Diqqerencing step is to calculate the difference between the pixel and the result of the Averaging step, Thresholding is the threshold processing to filter the results that are not within the threshold range, and the four wavelet domain features are set according to two types. Different thresholds are obtained by performing Haar wavelet transform twice in the order of row and column. It is worth noting that the final four wavelet domain features will be input into the affine transformation layer after being processed by two convolution filters.

In the affine transformation layer, the affine transformation layer obtains the feature map with the spatial domain and the wavelet domain feature set respectively, where the specific definition of the affine transformation layer is as follows, Output=λ*F+δ, where F is The feature map with spatial domain, λ and δ represent the wavelet domain features processed by two convolution filters, respectively perform element-wise multiplication and addition of affine transformation on F, and finally get the spatial domain and wavelet Feature map of domain features.

Further, in the embodiment of the present application, the above-mentioned convolutional layer uses the ReLU function as the activation function.

Referring to Fig. 2 , based on the above-mentioned cyclic generative adversarial network model, MRI images and PET images are input into the above-mentioned cyclic generative adversarial network model.

The MRI images are respectively sent to the first generator and the second discriminator after block extraction, the first generator generates PET generated images, and the PET generated images generated by the first generator are sent to the second generator and the first generator respectively. Among the discriminators, the first discriminator discriminates the authenticity of the PET-generated image and outputs the first discrimination result to the first generator, and the second generator generates a cyclic MRI-generated image, and the cyclic MRI-generated image is compared with the original MRI image. Get the losses of the first generator and the second generator.

On the contrary, the PET image is sent to the second generator and the first discriminator after block extraction, the second generator generates the MRI generated image, and the MRI generated image generated by the second generator is sent to the first generator and the first generator respectively. In the second discriminator, the second discriminator discriminates the authenticity of the MRI generated image and outputs the second discrimination result to the second generator. The first generator generates a cyclic PET generated image, and the cyclic PET generated image is compared with the original MRI image. A comparison is made to get the losses of the first generator and the second generator.

The above embodies the adversarial method and cyclic structure in the recurrent generative adversarial network model.

It is worth noting that the cyclic generative adversarial network model contains an adversarial loss function and a cycle consistency loss function.

The adversarial loss function is determined by the first discrimination result and the second discrimination result of the first discriminator and the second discriminator, and the specific form is as follows:

L( _GMRI-PET _,DPET , _IMRI , _IPET )=CE(DPET( _GMRI-PET ₍ _IMRI )),label);

L( _GPET-MRI _,DMRI , _IPET , _IMRI )=CE(DMRI( _GPET _-MRI ₍ IPET)),label);

Among them, G represents the generator that generates another modal image from one modal image, D represents the discriminator that distinguishes whether the modal image is the generated image, I represents the image of this modal, CE represents the Softmax The function is the cross-entropy function of the activation function, and the label is the real label used for evaluation. The above two forms represent the adversarial loss function models of the first discriminator and the second discriminator, respectively.

Among them, the cycle consistency loss function, the specific form is as follows:

From this, the mathematical expression of the global loss function is obtained:

Step S300, training and generating a cyclic generative adversarial network model, and gradually reaching a convergence state.

According to the technical solution defined in step S300, specifically, the cyclic generative adversarial network model generated by training uses the Nadam optimizer to optimize the model.

Step S400, using the trained recurrent generative adversarial network model to perform modal conversion processing on the MRI image to the PET image.

According to the technical solution defined in step S400, specifically, in the testing phase, the present application mainly uses the already trained first generator to solidify and form an inference model. Therefore, in the testing stage, the MRI image is extracted from the block and sent to the inference model, and the inference model is used to generate the PET generated image from the MRI image, and the PET generated image is required.

Therefore, the present application performs modal conversion processing from MRI images to PET images based on the recurrent generative adversarial network. By combining the self-attention unit and the wavelet affine transformation layer, they are applied in the generator and the discriminator respectively, which greatly improves the characteristics and characteristics of the generated images. The utilization of wavelet domain features and the occupancy ratio of GPU memory during training are reduced. In addition, the present application designs a joint loss function based on the SSIM function and the PSNR function. On the basis of the traditional cycle consistency comparison, the requirements for the generated image structure and signal-to-noise ratio are added, which significantly improves the quality of the PET generated image.

Referring to FIG. 7 , another object of the present application is to provide an MRI-PET image modality conversion system based on a recurrent generative adversarial network, including an acquisition module, a construction module, a training module and a verification module.

The acquisition module is used to acquire the MRI image dataset and the PET image dataset, and construct the input dataset from the MRI image dataset and the PET image dataset;

The building module is used to construct a recurrent generative adversarial network model, and the recurrent generative adversarial network model is trained against the input data set;

The training module is used to train the generative cycle generative adversarial network model, and gradually reach the convergence state;

The validation module is used for modality conversion processing from MRI images to PET images using the trained recurrent generative adversarial network model.

Therefore, the above system will perform modal conversion processing from MRI images to PET images based on recurrent generative adversarial networks. By combining the self-attention unit and the wavelet affine transformation layer, they are applied in the generator and discriminator respectively, which greatly improves the characteristics of the generated image. and the utilization of wavelet domain features, and reduce the occupancy ratio of GPU memory during training. In addition, the present application designs a joint loss function based on the SSIM function and the PSNR function. On the basis of the traditional cycle consistency comparison, the requirements for the generated image structure and signal-to-noise ratio are added, which significantly improves the quality of the PET generated image.

The above are all preferred embodiments of the present application, and are not intended to limit the protection scope of the present application. Therefore: all equivalent changes made according to the structure, shape and principle of the present application should be covered within the scope of the present application. Inside.

Claims

A MRI-PET image modality conversion method based on a recurrent generative adversarial network, characterized in that it comprises the following steps:

Acquire an MRI image dataset and a PET image dataset, and construct an input dataset from the MRI image dataset and the PET image dataset;

Build a recurrent generative adversarial network model, and conduct adversarial training on the recurrent generative adversarial network model with the input data set;

Train the generation cycle to generate the adversarial network model, and gradually reach the convergence state;

A trained recurrent generative adversarial network model is used to perform modality conversion processing from MRI images to PET images.
The MRI-PET image modality conversion method based on a recurrent generative adversarial network according to claim 1, wherein the constructed recurrent generative adversarial network model comprises a pair of first generators, a pair of second generators, The first discriminator and the second discriminator, wherein the adversarial training of the recurrent generative adversarial network model by means of the input dataset includes the following steps:

generating, by the first generator, the MRI image as a PET generated image;

generating, by the second generator, the PET image into an MRI generated image;

Distinguish whether the PET generated image is a PET image by the first discriminator, and output the first discrimination result to the first generator;

Distinguish whether the MRI generated image is an MRI image by the second discriminator, and output the second discrimination result to the second generator;

The first generator and the second generator perform the next iteration according to the first discrimination result and the second discrimination result, until the first discriminator and the second discriminator cannot distinguish the authenticity of the PET-generated image and the MRI-generated image.
The MRI-PET image modality conversion method based on a recurrent generative adversarial network according to claim 2, wherein the first generator and the second generator are constructed based on an improved U-Net model, Among them, the improved U-Net model adopts the self-attention unit to replace the clipping and expansion step in the skip connection step in the original U-Net model.
The MRI-PET image modality conversion method based on a recurrent generative adversarial network according to claim 3, wherein the self-attention unit is designed as a cross self-attention sub-module.
The MRI-PET image modality conversion method based on a recurrent generative adversarial network according to claim 2, wherein the first discriminator and the second discriminator output the first discrimination result and the second discrimination result, including the following step:

The PET image and the PET-generated image are input into the first discriminator as input pictures, and the MRI image and the MRI-generated image are input into the second discriminator as input pictures;

apply a wavelet affine transform layer inline in the first discriminator and the second discriminator;

In the wavelet affine transformation layer, the input image is extracted through two convolution layers to obtain a feature map with spatial domain, and the input image is simultaneously subjected to Haar wavelet transform to obtain a wavelet domain feature set, and the feature map with spatial domain is obtained. And the wavelet domain feature set is input into the affine transformation layer to obtain a feature map with spatial domain and wavelet domain features;

The above-mentioned wavelet affine transformation layer is repeated at least twice, and finally through the Softmax function, the first discriminator outputs the first discrimination result, and the second discriminator outputs the second discrimination result.
The MRI-PET image modality conversion method based on a recurrent generative adversarial network according to claim 5, wherein the convolutional layer uses a ReLU function as an activation function.
The MRI-PET image modality conversion method based on a recurrent generative adversarial network according to claim 1, wherein the recurrent generative adversarial network model comprises an adversarial loss function and a cycle consistency loss function.
The MRI-PET image modality conversion method based on a recurrent generative adversarial network according to claim 6, wherein the adversarial loss function is determined by the first discrimination result and the first discrimination result of the first discriminator and the second discriminator. Second, the identification result is determined, and the specific form is as follows:

L( GMRI-PET ,DPET , IMRI , IPET )=CE(DPET( GMRI-PET ( IMRI )),label);

L( GPET-MRI ,DMRI , IPET , IMRI )=CE(DMRI( GPET -MRI ( IPET)),label);

Among them, G represents the generator that generates another modal image from one modal image, D represents the discriminator that distinguishes whether the modal image is the generated image, I represents the image of this modal, CE represents the Softmax The function is the cross-entropy function of the activation function, and the label is the real label used for evaluation.
The MRI-PET image modality conversion method based on cyclic generative adversarial network according to claim 6, is characterized in that, described cyclic consistency loss function, specific form is as follows:

where μ and σ represent the mean and standard deviation of the image, respectively, and C 1 =(k 1 L) 2 and C 2 =(k 2 L) 2 are two smaller constant terms, avoiding a denominator of 0, where L represents the maximum pixel value of the image;

Among them, the image length is m,n, I(i,j)K(i,j) is the pixel value corresponding to the two input images;

L cyccon = μ SSIM L SSIM (G MRI-PET , G PET-MRI )+μ PSNR L PSNR (G MRI-PET , G PET-MRI );

where μ is the control loss function value that conforms to the cycle consistency constant parameter term;

This leads to the mathematical expression of the global loss function:
An MRI-PET image modality conversion system based on a recurrent generative adversarial network, comprising:

an acquisition module, which acquires the MRI image dataset and the PET image dataset, and constructs the input dataset from the MRI image dataset and the PET image dataset;

Building modules, constructing a recurrent generative adversarial network model, and conducting adversarial training on the recurrent generative adversarial network model with the input data set;

A training module, which trains and generates a cyclic generative adversarial network model, and gradually reaches a state of convergence;

The verification module uses the trained recurrent generative adversarial network model to perform modality conversion processing from MRI images to PET images.