WO2021022752A1 - Multimodal three-dimensional medical image fusion method and system, and electronic device - Google Patents

Abstract

Disclosed are a multimodal three-dimensional medical image fusion method and system, and an electronic device. The method comprises: constructing a multi-task generative adversarial network, wherein the multi-task generative adversarial network comprises a generator, a discriminator and a classifier; training the multi-task generative adversarial network according to an MRI image, a PET image and diagnosis label information of a subject; and inputting an MRI image of a person to be tested into the trained multi-task generative adversarial network, the generator synthesizing a corresponding PET image according to the MRI image and inputting the MRI image of said person and the synthesized PET image into the classifier, and outputting a disease classification prediction label of said person after the MRI image of said person and the synthesized PET image are fused. By means of the method, the problem in a traditional generative adversarial network of a conflict between loss function convergence points, which possibly occurs when the performance of a generator and the performance of a classifier are taken into consideration, is solved, such that the generator and the classifier can be optimal at the same time.

Description

Multimodal three-dimensional medical image fusion method, system and electronic equipment Technical field

This application belongs to the technical field of medical image processing, and particularly relates to a multi-modal three-dimensional medical image fusion method, system and electronic equipment.

Background technique

Different from MRI (Magnetic Resonance Imaging), CT (Computed Tomography) and other images, PET (Positron Emission Computed Tomography, Positron Emission Computed Tomography) is a kind of The functional imaging technology of in vivo observation has gradually been widely used in clinical diagnosis and early intervention. Specifically, the PET system can detect gamma rays emitted indirectly from the radioactive tracer. First, the tracer is injected into the human body through biologically active molecules, and then computer analysis technology is used to construct a three-dimensional PET image of the tracer concentration in the human body. The PET collection process poses an inevitable potential threat to human health. Investigations show that a PET scan of the brain can increase the risk of cancer by 0.04%. Although this number is small, repeated scans during treatment can double the risk of cancer. Because structural imaging MRI and functional imaging PET are complementary, fusing the image data of the two modalities can obtain more key features for classification and diagnosis. The multimodal fusion method is currently effective in improving the performance of auxiliary diagnosis models. One of the methods. However, due to the expensive PET data collection and insufficient sample size, it is difficult to obtain sufficient data to train the model.

The generative confrontation network was first proposed by LanGoodfellow and others in 2014, which led to the upsurge of GAN improvement and application research. In 2016, Salimans and others conducted a theoretical analysis and explanation of the problems that occurred during GAN training and application, and gave an empirical solution (Improved-GAN). Odena et al. improved the GAN network and applied it to semi-supervised learning, using unlabeled data and adversarial training strategies to improve the performance of the classifier. When GAN is used for image generation, it usually faces the problem that the semantics of the generated image is difficult to control and the image diversity cannot be guaranteed. To this end, Mehdi et al. designed a conditional-based generative confrontation network (conditional-GAN, cGAN), which can use data labels or multimodal attributes as conditional variables to guide image generation. Berkeley AI Lab scholar Phillip et al., based on cGAN, used different image styles as constraints, and realized the style transfer from image to image by optimizing the generator network through adversarial training strategies. However, the image style transfer network based on cGAN is restricted by the task requirements. The input image and output image must be paired and have the same content. Zhu et al. combined the two cGANs to design a cycle-based generation confrontation network CycleGAN, and increased the penalty of the loss function, which improved the effect of image style transfer and was no longer restricted by content consistency.

Wang et al. borrowed the basic framework of cGAN and adopted the U-Net structure as the generator network to realize the generation of high-quality PET images from PET collected by low-meter contrast agents. And proposed a progressive generation network architecture to gradually improve the quality of the synthesized image. This research is of great significance for reducing the cost of PET imaging and the potential radiation threat to the human body. Nie et al. fused cGAN network and FCN to realize the synthesis of brain MRI to CT image. Aiming at the problems of high spatial complexity and large synthesis errors in abdominal medical imaging, Hiasa et al. designed a cross-modal image synthesis method based on CycleGAN, which realized the migration from MRI to CT, and reduced the difference between the synthesized image and the real image. error. Recently, Pan et al. designed an MRI-PET synthesis model based on 3D-CycleGAN, which achieved accurate synthesis from MRI to PET. And using synthetic PET and MRI for feature fusion for AD (Alzheimer, Alzheimer's disease) and MCI (mild cognition impairment, mild cognitive impairment) diagnosis.

However, the existing generative confrontation model usually consists of a classification network and a discriminant network. When dealing with classification problems, the discriminant network takes into account the tasks of sample authenticity and pattern classification. This will cause conflicts between sample generation and the convergence point of pattern classification. In other words, traditional generative models cannot be used to deal with multitasking problems. At the same time, the existing cross-modal image synthesis method based on the conditional generation confrontation network uses the image of a given modal as the conditional constraint information without considering the label information of the sample. In addition, the existing cross-modal image synthesis and classification diagnosis research is to design independent models for the two tasks and train them separately, without considering the correlation between the two in the optimization process.

Summary of the invention

The present application provides a multi-modal three-dimensional medical image fusion method, system and electronic equipment, which aim to solve one of the above-mentioned technical problems in the prior art at least to a certain extent.

In order to solve the above-mentioned problems, this application provides the following technical solutions:

A multi-modal three-dimensional medical image fusion method includes the following steps:

Step a: construct a multi-task generative confrontation network, the multi-task generative confrontation network including a generator, a discriminator and a classifier;

Step b: Training the multi-task generation confrontation network according to the subject's MRI image, PET image and diagnostic tag information, so that the multi-task generation confrontation network automatically learns the associated features between the MRI image and the PET image;

Step c: Input the MRI image of the person to be inspected into the trained multi-task generating confrontation network, the generator synthesizes the corresponding PET image according to the MRI image, and inputs the MRI image of the person to be inspected and the synthesized PET image into the classifier, The classifier merges the MRI image of the person to be detected and the synthesized PET image and outputs the disease classification prediction label of the person to be detected.

The technical solution adopted in the embodiment of the present application further includes: in the step b, the training of the multi-task generation confrontation network specifically includes:

Step b1: Construct the confrontation loss function of the multi-task generation confrontation network; the confrontation loss in the training process is represented by an improved maximum-minimum cost function:

In the above formula, (C, G, D) respectively represent the classifier, discriminator and generator, (x, y, z) respectively represent the MRI image, PET image and diagnostic label information; α ∈ (0, 1) is a Constant, used to control the proportion of classifier and generator loss in the training process, E _{(x,y,z)～p(x,y,z)} [log D(x,y,z)] represents the discriminator The samples from the real data distribution are judged as real samples;

Indicates that the discriminator has identified a pair of pseudo samples in the output data space of the classifier;

Indicates that the discriminator recognizes the pseudo sample label pair from the generator, where G(x,z) represents the data distribution generated by the generator;

Step b2: Introduce the cross-entropy loss K _c =E _{(x,y,z)～p(x,y,z)} [-log p _c (x,y,z)] under supervised learning to the classifier, and classify The convergence point of the sample distribution p _c (x,y,z) of the generator is limited to p(x,y), so that the global optimum of the model satisfies the sample distribution generated by the generator G and the classifier C is the same as the real data distribution;

Step b3: Introduce the generator supervision loss, and use the gradient mutual information between the target image and the generated image as a similarity measure:

K _g ＝NI(A,B)=G(A,B)·I(A,B)

In the above formula, I (A, B) and G (A, B) respectively represent the gradient information and gradient difference between the generated image and the target image.

The technical solution adopted in the embodiment of the application further includes: in the step b, the training the multi-task generation confrontation network according to the subject's MRI image, PET image, and diagnostic tag information also includes: the generation The MRI image is used as a conditional constraint, and the random noise input of the same dimension as the target image is mapped to the PET image. The discriminator determines whether the input sample distribution (x, y, z) comes from the real data distribution or the pseudo data distribution, so The classifier takes the joint distribution of MRI and PET images as input and predicts its label type; in the data-driven mode, with the gradual optimization of the generator, the discriminator updates the network parameters to identify the pseudo data distribution generated by the generator ; With the optimization of the discriminator, the classifier is optimized to make the predicted disease classification and prediction label tend to be real data without being judged as fake data by the discriminator, and then reversely affect the training of the generator; through iterative adversarial training, The generator learns the potential correlation features between the MRI image and the PET image, thereby synthesizing the corresponding PET image from the input MRI image, and makes the classifier extract the key feature information from the input MRI image and the PET image. Predict the corresponding disease classification prediction label.

The technical solution adopted in the embodiment of the application further includes: the generator adopts a U-Net network structure, which includes an encoder and a decoder with a symmetrical network structure; in the step c, the generator synthesizes corresponding MRI images The PET image specifically includes: outputting the feature map of the MRI image through the feature extraction operation of the encoder multi-layer convolution; the decoder performs the multi-layer deconvolution operation on the feature map output by the encoder, and generates the feature map The feature map of the same size as the corresponding position of the encoder undergoes multiple splicing operations, and finally the target reconstructed image is output, which is the synthesized PET image.

The technical solution adopted in the embodiment of the present application further includes: in the step c, the classifier fused the MRI image of the subject to be detected and the synthesized PET image and then output the disease classification prediction label of the subject to be detected specifically includes: The feature extraction network extracts the feature value of the MRI image, performs convolution operation on the synthesized PET image, and extracts the feature value of the PET image; splicing the feature value of the MRI image and the PET image to form the spliced feature value. The fully connected layer performs fusion and high-dimensional abstraction on the spliced feature values; the fused feature information is operated by the Softmax function to obtain the corresponding disease classification prediction label.

Another technical solution adopted by the embodiment of the present application is: a multi-modal three-dimensional medical image fusion system, including:

Model building module: used to construct a multi-task generative confrontation network, which includes a generator, a discriminator, and a classifier;

Model training module: used to train the multi-task generation confrontation network according to the subject's MRI images, PET images and diagnostic tag information, so that the multi-task generation confrontation network can automatically learn the association between MRI images and PET images feature;

Model application module: used to input the MRI image of the person to be inspected into a trained multi-task generation confrontation network. The generator synthesizes the corresponding PET image according to the MRI image, and inputs the MRI image of the person to be inspected and the synthesized PET image A classifier, which merges the MRI image of the person to be detected and the synthesized PET image and outputs the disease classification prediction label of the person to be detected.

The technical solution adopted in the embodiment of the application further includes: the model training module includes:

Loss function construction unit: used to construct the confrontation loss function of the multi-task generation confrontation network; the confrontation loss in the training process is represented by an improved maximum-minimum cost function:

In the above formula, (C, G, D) respectively represent the classifier, discriminator and generator, (x, y, z) respectively represent the MRI image, PET image and diagnostic label information; α ∈ (0, 1) is a Constant, used to control the proportion of classifier and generator loss in the training process, E _{(x,y,z)～p(x,y,z)} [log D(x,y,z)] represents the discriminator The samples from the real data distribution are judged as real samples;

Indicates that the discriminator has identified a pair of pseudo samples in the output data space of the classifier;

It means that the discriminator recognizes the pseudo-sample label pair from the generator, where x represents the MRI modal image of the sample, z represents the sample label, and G(x,z) represents the PET modal image synthesized by the conditional generation network.

Classifier optimization unit: used to introduce the cross entropy loss under supervised learning to the classifier K _c =E _{(x,y,z)～p(x,y,z)} [-log p _c (x,y,z) ], the convergent point of the sample distribution p _c (x,y,z) of the classifier is limited to p(x,y), so that the global optimum of the model satisfies the sample distribution generated by generator G and classifier C and the real The data distribution is the same;

Generator optimization unit: used to introduce generator supervision loss, using the gradient mutual information between the target image and the generated image as a similarity measure:

K _g ＝NI(A,B)=G(A,B)·I(A,B)

In the above formula, I (A, B) and G (A, B) respectively represent the gradient information and gradient difference between the generated image and the target image.

The technical solution adopted in the embodiment of the application further includes: the model training module trains the multi-task generation confrontation network specifically: the generator uses the MRI image as a conditional constraint, and maps the random noise input of the same dimension as the target image into For PET images, the discriminator determines whether the input sample distribution (x, y, z) comes from a real data distribution or a pseudo data distribution, and the classifier takes the joint distribution of MRI images and PET images as input, and predicts its label type ; In the data-driven mode, with the gradual optimization of the generator, the discriminator updates the network parameters to identify the pseudo data distribution generated by the generator; with the optimization of the discriminator, the classifier is optimized to encourage the classification of the disease to predict the label trend Based on real data without being judged as pseudo data by the discriminator, it then acts in the reverse direction on the training of the generator; through iterative confrontation training, the generator learns the potential correlation characteristics between MRI images and PET images, thereby The input MRI images are synthesized to obtain corresponding PET images, and the classifier is made to extract key feature information from the input MRI images and PET images and predict the corresponding disease classification prediction labels.

The technical solution adopted by the embodiment of the application further includes: the generator adopts a U-Net network structure, which includes an encoder and a decoder with a symmetric network structure; the generator synthesizing the corresponding PET image according to the MRI image specifically includes: The feature extraction operation of the encoder multi-layer convolution to output the feature map of the MRI image; the decoder performs the multi-layer deconvolution operation on the feature map output by the encoder, and the generated feature map is the same size as the corresponding position of the encoder Perform multiple stitching operations on the feature map of, and finally output the target reconstructed image, which is the synthesized PET image.

The technical solution adopted in the embodiment of the present application further includes: the classifier integrates the MRI image of the subject to be detected and the synthesized PET image and then outputs the disease classification prediction label of the subject to be detected. Specifically, it includes: extracting the MRI image by using a feature extraction network. Eigenvalues, and perform convolution operations on the synthesized PET images to extract the eigenvalues of the PET images; splicing the MRI images and the eigenvalues of the PET images to form the spliced characteristic values, and the fully connected layer pairs the spliced Feature values are fused and high-dimensional abstracted; the fused feature information is operated by the Softmax function to obtain the corresponding disease classification prediction label.

Another technical solution adopted by the embodiments of the present application is: an electronic device, including:

At least one processor; and

A memory communicatively connected with the at least one processor; wherein,

The memory stores instructions executable by the one processor, and the instructions are executed by the at least one processor, so that the at least one processor can execute the following of the above-mentioned multi-modal three-dimensional medical image fusion method operating:

Step a: construct a multi-task generative confrontation network, the multi-task generative confrontation network including a generator, a discriminator and a classifier;

Step b: Training the multi-task generation confrontation network according to the subject's MRI image, PET image and diagnostic tag information, so that the multi-task generation confrontation network automatically learns the associated features between the MRI image and the PET image;

Step c: Input the MRI image of the person to be inspected into the trained multi-task generating confrontation network, the generator synthesizes the corresponding PET image according to the MRI image, and inputs the MRI image of the person to be inspected and the synthesized PET image into the classifier, The classifier merges the MRI image of the person to be detected and the synthesized PET image and outputs the disease classification prediction label of the person to be detected.

Compared with the prior art, the beneficial effects produced by the embodiments of the present application are: the multi-modal three-dimensional medical image fusion method, system and electronic equipment of the embodiments of the present application propose a multi-task generation confrontation model, which is based on the subject's lesion location Synthesize the pattern image in the PET image from the MRI image, and fuse the real MRI image and the synthesized PET image to obtain more key features for classification and diagnosis, and classify the disease types according to the key features. Compared with the prior art, this application has at least the following beneficial effects:

1. By setting up a separate discriminator, its sole function is to identify the authenticity of the data distribution, which solves the conflict problem of the convergence point of the loss function that may occur when the traditional generative confrontation network takes into account the performance of the generator and the classifier, and can make the generator At the same time as the classifier achieve the optimal.

2. It can realize the one-step collaborative training of cross-modal image synthesis model and multi-modal fusion classification model, which can achieve better training effects. The trained generative model learns the associated features of MRI and PET imaging, and can synthesize the corresponding PET from the MRI of the examinee. The classification model fuses the characteristic information of MRI and synthetic PET to classify and diagnose diseases, avoiding the high cost of PET collection At the same time as the risk of radiation exposure, functional imaging features are effectively integrated, which can achieve higher classification accuracy.

3. This application considers the joint distribution of the three attributes of MRI, PET, and diagnostic tags. The model can extract richer correlation feature information between multimodal imaging and classification diagnosis, and improve image generation errors and classification diagnosis performance. Through the cumulative training of a large number of cases, the accuracy and robustness of the prediction model are gradually improved.

4. The multi-task generative confrontation network proposed by the present invention can also be used in other collaborative optimization application scenarios.

Description of the drawings

FIG. 1 is a flowchart of a multi-modal three-dimensional medical image fusion method according to an embodiment of the present application;

Figure 2 is the overall framework diagram of the multi-task generation confrontation network;

Figure 3 is a schematic diagram of the network structure of the generator;

Figure 4 is a schematic diagram of the network structure of the classifier;

Figure 5 is an application flow chart of a multi-task generating confrontation network;

6 is a schematic structural diagram of a multi-modal three-dimensional medical image fusion system according to an embodiment of the present application;

FIG. 7 is a schematic diagram of the hardware device structure of the multi-modal three-dimensional medical image fusion method provided by an embodiment of the present application.

detailed description

In order to make the purpose, technical solutions, and advantages of this application clearer, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the application, and not used to limit the application.

In view of the complementarity of multi-modal images in clinical diagnosis and the high cost and radiation exposure risks in the PET acquisition process, the multi-modal three-dimensional medical image fusion method in the embodiments of the present application proposes a multi-task generation confrontation model (Multi- Task GAN, MT-GAN), predict the pattern image in PET imaging based on the MRI image of the subject's lesion, and realize the confrontation between the cross-modal image synthesis network and the multi-modal fusion classification network in the data-driven mode Co-training, the optimized system successfully learned the potential correlation features between MRI imaging, PET imaging and disease diagnosis, and solved the conflict problem between the generation network and the discrimination network convergence point in the traditional generative confrontation model. This application integrates multi-source medical imaging features without the need for PET collection by the examinee, which can more accurately assist doctors in clinical diagnosis. In order to clearly illustrate the specific implementation of this application, the following examples are based on MRI and PET imaging of Alzheimer’s disease as an example, but the scope of application of this application is not limited to diseases of Alzheimer’s disease and MRI-PET imaging. It can also be widely used in CT-PET, MRI-CT and other modal images of other diseases.

Please refer to FIG. 1, which is a flowchart of a multi-modal three-dimensional medical image fusion method according to an embodiment of the present application. The multi-modal three-dimensional medical image fusion method of the embodiment of the present application includes the following steps:

Step 100: Collect MRI images and PET images of the subject, and preprocess the collected MRI images and PET images to obtain a data set for training the model;

In step 100, the acquisition of MRI images and PET images is specifically: selecting Alzheimer's disease (AD) to be tested, mild cognitive impairment (MCI) to be tested, and normal elderly (Normal) as the recipients. The examinee collects the MRI image and PET image of his brain as the original data set, and in the clinical observation and diagnosis of each subject, a professional physician will give the diagnosis information, and use the diagnosis information as the Diagnostic label information.

The preprocessing of the original data set is specifically: using FSL, SPM and other technologies to perform redundant tissue removal and image correction processing on the collected brain MRI and PET, and use FSL brain image processing tools to perform linear registration operations on MRI and PET images. Make the anatomical points of MRI and PET images in the diagnostic sense reach the same spatial position.

Step 200: Construct a multi-task generating confrontation network;

In step 200, the multi-task generation confrontation network framework is shown in FIG. 2. The multi-task generation confrontation network needs to consider the three attributes of MRI image, PET image and diagnostic label information of each subject to be detected. It includes classifier C, generator G, and discriminator D. Generator G is used to pass real MRI images Synthesize the corresponding PET images; the discriminator D is used to determine whether the data pattern distribution comes from the real data or the pseudo sample distribution; the classifier C is used to fuse the MRI image and the synthesized PET image to output the disease classification prediction label of the subject . Mark the MRI image, PET image, and diagnostic label information as (x, y, z), then the real data distribution p _true (x, y, z) and the generator sample distribution p _{G are} included in the multi-task generation confrontation network (x,y _g ,z) and the sample distribution of the classifier p _c (x,y,z _l ) three data distributions. The generator takes the MRI image as a conditional constraint, and maps the random noise input of the same dimension as the target image to the PET image to realize the conditional feature mapping of p _G (x, y _g , z) = p _true (x, y, z); The classifier takes the joint distribution of MRI and PET images as input, predicts its label type, and realizes the conditional feature mapping of p _c (x, y, z _l ) = p _true (x, y, z); the discriminator is based on The input sample distribution (x, y, z) determines whether it comes from the real data distribution, which is essentially a binary classification problem.

In the embodiment of this application, the network structure of the generator is shown in FIG. 3. The generator adopts the U-Net network structure, and the U-Net model is designed based on a jump-connected full convolutional network. The main idea is to design an encoder and decoder with a symmetric network structure to have the same number and size of feature maps , And combine the corresponding feature maps of the encoder and the decoder through skip connection, which can retain the feature information in the down-sampling process to the maximum extent, thereby improving the efficiency of feature expression. MRI images and PET images come from the same sample and share a large amount of primary feature information between them. Therefore, the U-net model is very suitable for complex feature mapping between the two modal images.

Based on the above-mentioned U-Net network structure, the generator synthesizes the corresponding PET image with real MRI image samples as follows:

(1) Extract the feature information of the MRI image through the encoder; take the sample 128×128×128 size MRI image as input, after 64 2×2×2 size convolution kernels feature extraction operations, slide in three dimensions The step size is 2, and 64 feature maps with a size of 64×64×64 are output. Then 128 2×2×2 size convolution kernels are used to perform convolution operation on it, and 128 32×32×32 size feature maps are generated. By analogy, after the encoder's 6-layer convolution feature extraction operation, 1024 feature maps are output.

(2) The decoder reconstructs the feature map output by the encoder; first, the 1024 feature maps output by the encoder are deconvolved to generate 512 2×2×2 feature maps, which correspond to the encoder Feature maps of the same size at the same location are stitched together. After 6 layers of deconvolution operation and splicing operation in turn, the final output is the target reconstructed image of 128×128×128 size, which is the synthesized PET image.

In order to reduce the computational complexity of the model and improve the efficiency of collaborative training of the network, the classifier in the embodiment of the present application selects a relatively simple multi-modal fusion classification network, and its structure is shown in FIG. 4. The processing flow of the multi-modal image data by the classifier is:

(1) The feature extraction network is used to extract the feature information of the MRI image; first, the two convolutional layers of the 2×2×2 size convolution kernel are used to extract the primary features of the image to generate 32 feature maps, and then a layer of window size is 2 The ×2×2 pooling layer reduces the dimensionality of the feature map. Subsequently, a 3×3×3 size convolution kernel is used to extract advanced features. The third and fourth convolution layers use 64 convolution kernels respectively, and the extracted features are pooled and reduced in dimension, and then 128 convolution kernels are used for Higher-dimensional feature extraction.

(2) Using the same structure of feature extraction network to perform convolution operation on PET images to generate 128 feature values.

(3) The feature values extracted from the MRI image and the PET image are spliced together to form 256 feature values, and the feature information of the two modalities is fused and high-dimensional abstracted by a fully connected layer containing 54 nodes.

(4) The integrated fusion feature information is operated by the Softmax function to obtain the corresponding label prediction type (that is, the probability of the predicted image data corresponding to the disease level).

Step 300: Train the multi-task generation confrontation network according to the subject's MRI image, PET image and diagnostic tag information;

In step 300, the training process of the multi-task generating confrontation network includes the following steps:

Step 301: Construct the model's anti-loss function;

In the actual application process of the model, it is necessary to collect the MRI data of each person to be tested, so the prediction process of the generator and the classifier are respectively the following conditional distributions:

p _g (x,y _g )=p(y|x)p(x) (1)

p _c (x,y,z)=p[z|(x,y)]p(x,y) (2)

The adversarial loss of the training process can be represented by an improved minimax cost function:

In formula (3), α∈(0,1) is a constant, which is used to control the proportion of classifier and generator loss in the training process, that is, the relative importance in the confrontation training task. E _{(x,y,z)～p(x,y,z)} [log D(x,y,z)] means that the discriminator judges the samples from the real data distribution as real samples;

Indicates that the discriminator has identified a pair of pseudo samples in the output data space of the classifier;

It means that the discriminator recognizes the pseudo-sample label pair from the generator, where x represents the MRI modal image of the sample, z represents the sample label, and G(x,z) represents the PET modal image synthesized by the conditional generation network. As a result, the multi-task generating confrontation network's confrontation loss function is constructed.

Step 302: Introduce the classifier supervision loss; from the optimization principle of the general confrontation generation network, it can be known that the model is only if p(x,y,z)=(1-α)p _g (x,y _g ,z)+αp Nash equilibrium is reached at _c (x, y, z). The equilibrium of the adversarial game shows that when one of the generator G and the classifier C reaches the optimum, the other also approaches the optimum. But in fact, the global optimum of the model should satisfy that the sample distributions generated by generator G and classifier C are the same as the real data distribution, that is, p(x,y,z)=p _g (x,y,z)=p _c (x,y,z). But the solution of the above loss function is a subset of p(x,y,z)=(1-α)p _g (x,y,z)+αp _c (x,y,z), which cannot guarantee p(x, y, z) = p _g (x, y, z) = p _c (x, y, z). Therefore, this application introduces the cross-entropy loss K _c =E _{(x,y,z)～p(x,y,z)} [-logp _c (x,y,z) under supervised learning to the classifier in training ], thus limiting the convergence point of p _c (x, y, z) to near p (x, y), and then ensuring that the solution of the loss function is the global optimal solution.

Step 303: Introduce generator supervision loss;

In generator training, in addition to the need to generate samples to make the discriminator difficult to identify from the loss function design, it is also necessary to ensure that the generated samples are as similar as possible to the target image. This application uses the gradient mutual information between the target image and the generated image as a similarity measure:

K _g ＝NI(A,B)=G(A,B)·I(A,B) (4)

In the above formula, I (A, B) and G (A, B) respectively represent the gradient information and gradient difference between the generated image and the target image.

In summary, the objective function of the multi-task generative confrontation network proposed in this application is:

Step 304: Divide the data set of 900 subjects into a training set and a test set, train the multi-task generation confrontation network through the training set, and test the performance of the multi-task generation confrontation network through the test set;

In step 304, there are 700 sample data in the training set, and 200 sample data in the test set. The specific training process of the model is as follows: In the data-driven mode, as the generator G is gradually optimized, the discriminator D needs to update the network parameters to identify the pseudo data distribution generated by the generator G; as the discriminator D optimizes the incentive classification The device C is optimized so that the predicted disease classification prediction label tends to be real data without being judged as fake data by the discriminator D, and then acts on the training of the generator G in reverse. Through the iterative training of the multi-task generation confrontation network in this way, the generator G and the classifier C are optimized in the confrontation training, and in the process of the three confrontation games, the classifier and the generator are better than the independent training. Performance.

Step 400: Input the MRI image of the person to be detected into the trained multi-task generating confrontation network, and the multi-task generating confrontation network outputs the disease classification prediction label of the person to be detected;

In step 400, after the confrontation training, the generator G learns the potential correlation features between the MRI image and the PET image, and can more accurately synthesize the corresponding PET image from the input MRI image. The parameters of the classifier are also optimized, and key feature information can be extracted from the input MRI images and PET images, and the corresponding disease classification prediction labels can be predicted based on the features.

Please refer to Figure 5 for details. The application process of the multi-task generating confrontation network specifically includes the following steps:

Step 401: Collect MRI images of the person to be tested;

Step 402: Input the MRI image into the trained generator for synthesis, and the generator synthesizes the corresponding PET image according to the MRI image;

Step 403: Input the MRI image and the synthesized PET image into the trained classifier, and the classifier outputs the disease classification prediction label of the person to be detected.

Please refer to FIG. 6, which is a schematic structural diagram of a multi-modal three-dimensional medical image fusion system according to an embodiment of the present application. The multi-modal three-dimensional medical image fusion system of the embodiment of the present application includes a data acquisition module, a model construction module, a model training module, and a model application module.

Data acquisition module: used to acquire MRI images and PET images of subjects, and preprocess the acquired MRI images and PET images to obtain a data set for training the model; among them, the acquisition of MRI images and PET images is specifically: Select 300 people each for Alzheimer's disease (AD) to be tested, mild cognitive impairment (MCI) to be tested, and normal elderly (Normal) as subjects, and collect MRI and PET images of their brains As the original data set, and in the clinical observation and diagnosis of each subject, professional physicians will give diagnostic information, and use the diagnostic information as the diagnostic label information of each subject.

The preprocessing of MRI and PET images is specifically: using FSL, SPM and other technologies to perform redundant tissue removal and image correction processing on the collected brain MRI and PET, and use FSL brain image processing tools to linearly register MRI and PET images Operation to make the anatomical points of MRI and PET images in the diagnostic sense reach the same spatial position.

Model building module: used to build a multi-task generative confrontation network; among them, the multi-task generative confrontation network includes a classifier C, a generator G, and a discriminator D. The generator G is used to synthesize corresponding PET images from real MRI images; The device D is used to determine whether the data pattern distribution comes from real data or a pseudo sample distribution; the classifier C is used to fuse the MRI image and the synthesized PET image to output the disease classification prediction label of the subject to be detected. Mark the MRI image, PET image, and diagnostic label information as (x, y, z), then the real data distribution p _true (x, y, z) and the generator sample distribution p _{G are} included in the multi-task generation confrontation network (x,y _g ,z) and the sample distribution of the classifier p _c (x,y,z _l ) three data distributions. The generator takes the MRI image as a conditional constraint, and maps the random noise input of the same dimension as the target image to the PET image to realize the conditional feature mapping of p _G (x, y _g , z) = p _true (x, y, z); The classifier takes the joint distribution of MRI images and PET images as input, predicts its label type, and realizes the conditional feature mapping of p _c (x,y,z _l )=p _true (x,y,z); the discriminator is based on The input sample distribution (x, y, z) determines whether it comes from the real data distribution, which is essentially a binary classification problem.

In the embodiment of this application, the generator adopts the U-Net network structure. The U-Net model is designed based on a jump-connected full convolutional network. The main idea is to design an encoder and a decoder with a symmetric network structure to have the same The number and size of feature maps, and the corresponding feature maps of the encoder and decoder are combined through skip connection, which can retain the feature information in the downsampling process to the maximum extent, thereby improving the efficiency of feature expression. MRI images and PET images come from the same sample and share a large amount of primary feature information between them. Therefore, the U-net model is very suitable for complex feature mapping between the two modal images.

Based on the above-mentioned U-Net network structure, the generator synthesizes the corresponding PET image with real MRI image samples as follows:

(1) Extract the feature information of the MRI image through the encoder; take the sample 128×128×128 size MRI image as input, after 64 2×2×2 size convolution kernels feature extraction operations, slide in three dimensions The step size is 2, and 64 feature maps with a size of 64×64×64 are output. Then 128 2×2×2 size convolution kernels are used to perform convolution operation on it, and 128 32×32×32 size feature maps are generated. By analogy, after the encoder's 6-layer convolution feature extraction operation, 1024 feature maps are output.

(2) The decoder reconstructs the feature map output by the encoder; first, the 1024 feature maps output by the encoder are deconvolved to generate 512 2×2×2 feature maps, which correspond to the encoder Feature maps of the same size at the same location are stitched together. After 6 layers of deconvolution operation and splicing operation in turn, the final output is the target reconstructed image of 128×128×128 size, which is the synthesized PET image.

In order to reduce the computational complexity of the model and improve the efficiency of collaborative training of the network, the classifier in this embodiment of the application selects a relatively simple multi-modal fusion classification network, and the classifier processes the multi-modal image data as follows:

(1) Extract the feature information of the MRI image; first use the two convolutional layers of the 2×2×2 size convolution kernel to extract the primary features of the image to generate 32 feature maps, and then use a layer of window size of 2×2×2 The pooling layer reduces the dimensionality of the feature map. Subsequently, a 3×3×3 size convolution kernel is used to extract advanced features. The third and fourth convolution layers use 64 convolution kernels respectively, and the extracted features are pooled and reduced in dimension, and then 128 convolution kernels are used for Higher-dimensional feature extraction.

(2) Using the same structure of feature extraction network to perform convolution operation on PET images to generate 128 feature values.

(3) The feature values extracted from the MRI image and the PET image are spliced together to form 256 feature values, and the feature information of the two modalities is fused and high-dimensional abstracted by a fully connected layer containing 54 nodes.

(4) The integrated fusion feature information is operated by the Softmax function to obtain the corresponding label prediction type (that is, the probability of the predicted image data corresponding to the disease level).

Model training module: used to train the multi-task generation confrontation network based on the subject’s MRI images, PET images and diagnostic tag information; the model training module includes:

Loss function construction unit: used to construct the model's adversarial loss function; in the actual application process of the model, the MRI data of each person to be tested needs to be collected, so the prediction process of the generator and the classifier are respectively the following conditional distributions:

p _g (x,y _g )=p(y|x)p(x) (1)

p _c (x,y,z)=p[z|(x,y)]p(x,y) (2)

The adversarial loss of the training process can be represented by an improved minimax cost function:

In formula (3), α∈(0,1) is a constant, which is used to control the proportion of classifier and generator loss in the training process, that is, the relative importance in the confrontation training task. E _{(x,y,z)～p(x,y,z)} [log D(x,y,z)] means that the discriminator judges the samples from the real data distribution as real samples;

Indicates that the discriminator has identified a pair of pseudo samples in the output data space of the classifier;

As a result, the multi-task generating confrontation network's confrontation loss function is constructed.

Classifier optimization unit: used to introduce classifier supervision loss; it can be known from the optimization principle of general confrontation generation network that the model is and only if p(x,y,z)=(1-α)p _g (x,y _g , z)+αp _c (x, y, z) reaches the Nash equilibrium. The equilibrium of the adversarial game shows that when one of the generator G and the classifier C reaches the optimum, the other also approaches the optimum. But in fact, the global optimum of the model should satisfy that the sample distributions generated by generator G and classifier C are the same as the real data distribution, that is, p(x,y,z)=p _g (x,y,z)=p _c (x,y,z). But the solution of the above loss function is a subset of p(x,y,z)=(1-α)p _g (x,y,z)+αp _c (x,y,z), which cannot guarantee p(x, y, z) = p _g (x, y, z) = p _c (x, y, z). Therefore, this application introduces the cross-entropy loss K _c =E _{(x,y,z)～p(x,y,z)} [-log p _c (x,y,z ₎ under supervised learning to the classifier in training )], thus limiting the convergence point of p _c (x,y,z) near p(x,y), and then ensuring that the solution of the loss function is the global optimal solution.

Generator optimization unit: used to introduce generator supervision loss; in generator training, in addition to the need to generate samples to make the discriminator difficult to identify from the loss function design, it is also necessary to ensure that the generated samples are as similar as possible to the target image. This application uses the gradient mutual information between the target image and the generated image as a similarity measure:

K _g ＝NI(A,B)=G(A,B)·I(A,B) (4)

In the above formula, I(A,B) and G(A,B) respectively represent the gradient information and gradient difference between the generated image and the target image.

In summary, the objective function of the multi-task generative confrontation network proposed in this application is:

Model training unit: used to divide the data set of 900 subjects into a training set and a test set, train the multi-task generating confrontation network through the training set, and test the performance of the multi-task generating confrontation network through the test set; , The sample data in the training set is 700, and the sample data in the test set is 200. The specific training process of the model is as follows: In the data-driven mode, as the generator G is gradually optimized, the discriminator D needs to update the network parameters to identify the pseudo data distribution generated by the generator G; as the discriminator D optimizes the incentive classification The device C is optimized so that the predicted disease classification prediction label tends to be real data without being judged as fake data by the discriminator D, and then acts on the training of the generator G in reverse. Through the iterative training of the multi-task generation confrontation network in this way, the generator G and the classifier C are optimized in the confrontation training, and in the process of the three confrontation games, the classifier and the generator are better than the independent training. Performance.

Model application module: used to input the MRI image of the person to be detected into the trained multi-task generation confrontation network, and the multi-task generation confrontation network outputs the disease classification prediction label of the person to be detected; after the confrontation training, the generator G learns the MRI image The potential associated features with PET images can be more accurately synthesized from the input MRI images to obtain the corresponding PET images. The parameters of the classifier are also optimized, and key feature information can be extracted from the input MRI images and PET images, and the corresponding disease classification prediction labels can be predicted based on the features.

Specifically, the application process of the multi-task generation confrontation network is specifically: collecting the MRI image of the person to be detected, inputting the MRI image into the trained generator for synthesis, and the generator synthesizing the corresponding PET image according to the MRI image; combining the MRI image with The synthesized PET image is input to the trained classifier, and the classifier outputs the disease classification prediction label of the person to be detected.

FIG. 7 is a schematic diagram of the hardware device structure of the multi-modal three-dimensional medical image fusion method provided by an embodiment of the present application. As shown in Figure 7, the device includes one or more processors and memory. Taking a processor as an example, the device may also include: an input system and an output system.

The processor, the memory, the input system, and the output system may be connected by a bus or in other ways. In FIG. 7, the connection by a bus is taken as an example.

As a non-transitory computer-readable storage medium, the memory can be used to store non-transitory software programs, non-transitory computer executable programs, and modules. The processor executes various functional applications and data processing of the electronic device by running non-transitory software programs, instructions, and modules stored in the memory, that is, realizing the processing methods of the foregoing method embodiments.

The memory may include a program storage area and a data storage area, where the program storage area can store an operating system and an application program required by at least one function; the data storage area can store data and the like. In addition, the memory may include a high-speed random access memory, and may also include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid state storage devices. In some embodiments, the storage may optionally include storage remotely arranged with respect to the processor, and these remote storages may be connected to the processing system through a network. Examples of the aforementioned networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

The input system can receive input digital or character information, and generate signal input. The output system may include display devices such as a display screen.

The one or more modules are stored in the memory, and when executed by the one or more processors, the following operations of any of the foregoing method embodiments are performed:

Step a: construct a multi-task generative confrontation network, the multi-task generative confrontation network including a generator, a discriminator and a classifier;

Step b: Training the multi-task generating confrontation network according to the subject's MRI image, PET image and diagnostic tag information, so that the multi-task generating confrontation network automatically learns the associated features between the MRI image and the PET image;

Step c: Input the MRI image of the person to be inspected into the trained multi-task generating confrontation network, the generator synthesizes the corresponding PET image according to the MRI image, and inputs the MRI image of the person to be inspected and the synthesized PET image into the classifier, The classifier merges the MRI image of the person to be detected and the synthesized PET image and outputs the disease classification prediction label of the person to be detected.

The above-mentioned products can execute the methods provided in the embodiments of the present application, and have functional modules and beneficial effects corresponding to the execution methods. For technical details not described in detail in this embodiment, please refer to the method provided in the embodiment of this application.

The embodiments of the present application provide a non-transitory (non-volatile) computer storage medium, the computer storage medium stores computer executable instructions, and the computer executable instructions can perform the following operations:

Step a: construct a multi-task generative confrontation network, the multi-task generative confrontation network including a generator, a discriminator and a classifier;

Step b: Training the multi-task generation confrontation network according to the subject's MRI image, PET image and diagnostic tag information, so that the multi-task generation confrontation network automatically learns the associated features between the MRI image and the PET image;

Step c: Input the MRI image of the person to be inspected into the trained multi-task generating confrontation network, the generator synthesizes the corresponding PET image according to the MRI image, and inputs the MRI image of the person to be inspected and the synthesized PET image into the classifier, The classifier merges the MRI image of the person to be detected and the synthesized PET image and outputs the disease classification prediction label of the person to be detected.

The embodiment of the present application provides a computer program product, the computer program product includes a computer program stored on a non-transitory computer-readable storage medium, the computer program includes program instructions, when the program instructions are executed by a computer To make the computer do the following:

Step a: construct a multi-task generative confrontation network, the multi-task generative confrontation network including a generator, a discriminator and a classifier;

Step b: Training the multi-task generation confrontation network according to the subject's MRI image, PET image and diagnostic tag information, so that the multi-task generation confrontation network automatically learns the associated features between the MRI image and the PET image;

Step c: Input the MRI image of the person to be inspected into the trained multi-task generating confrontation network, the generator synthesizes the corresponding PET image according to the MRI image, and inputs the MRI image of the person to be inspected and the synthesized PET image into the classifier, The classifier merges the MRI image of the person to be detected and the synthesized PET image and outputs the disease classification prediction label of the person to be detected.

The multi-modal three-dimensional medical image fusion method, system and electronic equipment of the embodiments of the present application propose a multi-task generation confrontation model, which synthesizes the pattern image in the PET image according to the MRI image of the subject’s lesion, and merges it After real MRI images and synthetic PET images, more key features for classification and diagnosis are obtained, and disease types are classified according to the key features. Compared with the prior art, this application has at least the following beneficial effects:

1. By setting up a separate discriminator, its sole function is to identify the authenticity of the data distribution, which solves the conflict problem of the convergence point of the loss function that may occur when the traditional generative confrontation network takes into account the performance of the generator and the classifier, and can make the generator At the same time as the classifier achieve the optimal.

2. It can realize the one-step collaborative training of cross-modal image synthesis model and multi-modal fusion classification model, which can achieve better training effects. The trained generative model learns the associated features of MRI and PET imaging, and can synthesize the corresponding PET from the MRI of the examinee. The classification model fuses the characteristic information of MRI and synthetic PET to classify and diagnose diseases, avoiding the high cost of PET collection At the same time as the risk of radiation exposure, functional imaging features are effectively integrated, which can achieve higher classification accuracy.

3. This application considers the joint distribution of the three attributes of MRI, PET, and diagnostic tags. The model can extract richer associated feature information between multimodal imaging and classification diagnosis, and improve image generation errors and classification diagnosis performance. Through the cumulative training of a large number of cases, the accuracy and robustness of the prediction model are gradually improved.

4. The multi-task generative confrontation network proposed by the present invention can also be used in other collaborative optimization application scenarios.

The above are only the preferred embodiments of the present invention. It should be pointed out that for those of ordinary skill in the art, without departing from the principle of the present invention, several improvements and modifications can be made, and these improvements and modifications are also It should be regarded as the protection scope of the present invention.

Claims (11)

1. A multi-modal three-dimensional medical image fusion method is characterized in that it comprises the following steps:

   Step a: construct a multi-task generative confrontation network, the multi-task generative confrontation network including a generator, a discriminator and a classifier;

   Step b: Training the multi-task generation confrontation network according to the subject's MRI image, PET image and diagnostic tag information, so that the multi-task generation confrontation network automatically learns the associated features between the MRI image and the PET image;

   Step c: Input the MRI image of the person to be inspected into the trained multi-task generating confrontation network, the generator synthesizes the corresponding PET image according to the MRI image, and inputs the MRI image of the person to be inspected and the synthesized PET image into the classifier, The classifier merges the MRI image of the person to be detected and the synthesized PET image and outputs the disease classification prediction label of the person to be detected.
2. The multi-modal three-dimensional medical image fusion method according to claim 1, wherein in the step b, the training of the multi-task generation confrontation network specifically includes:

   Step b1: Construct the confrontation loss function of the multi-task generation confrontation network; the confrontation loss in the training process is represented by an improved maximum-minimum cost function:

   

   In the above formula, (C, G, D) respectively represent the classifier, discriminator and generator, (x, y, z) respectively represent the MRI image, PET image and diagnostic label information; α ∈ (0, 1) is a Constant, used to control the proportion of classifier and generator loss in the training process, E _{(x,y,z)～p(x,y,z)} [logD(x,y,z)] means that the discriminator will The samples from the real data distribution are judged as real samples;

   

   Indicates that the discriminator has identified a pair of pseudo samples in the output data space of the classifier;

   

   Indicates that the discriminator recognizes the pseudo sample label pair from the generator, where x represents the MRI modal image of the specimen, z represents the sample label, and G(x,z) represents the PET modal image synthesized by the conditional generation network;

   Step b2: Introduce the cross-entropy loss K _c =E _{(x,y,z)～p(x,y,z)} [-log p _c (x,y,z)] under supervised learning to the classifier, and classify The convergence point of the sample distribution p _c (x,y,z) of the generator is limited to p(x,y), so that the global optimum of the model satisfies the sample distribution generated by the generator G and the classifier C is the same as the real data distribution;

   Step b3: Introduce the generator supervision loss, and use the gradient mutual information between the target image and the generated image as a similarity measure:

   K _g ＝NI(A,B)=G(A,B)·I(A,B)

   

   

   In the above formula, I (A, B) and G (A, B) respectively represent the gradient information and gradient difference between the generated image and the target image.
3. The multi-modal three-dimensional medical image fusion method according to claim 2, wherein in the step b, the MRI image, PET image, and diagnostic tag information of the subject are used to generate a confrontation against the multi-task The network training also includes: the generator uses the MRI image as a conditional constraint to map the random noise input of the same dimension as the target image to the PET image, and the discriminator determines that the input sample distribution (x, y, z) comes from Real data distribution or pseudo data distribution. The classifier takes the joint distribution of MRI and PET images as input and predicts its label type; in the data-driven mode, with the gradual optimization of the generator, the discriminator updates the network parameters to Recognize the pseudo data distribution generated by the generator; with the optimization of the discriminator, the classifier is optimized to make the predicted disease classification and prediction label tend to be real data without being judged as pseudo data by the discriminator, and then the generator is reversed Training; through iterative adversarial training, the generator learns the potential correlation features between the MRI image and the PET image, thereby synthesizing the corresponding PET image from the input MRI image, and making the classifier from the input MRI Images and PET images extract key feature information and predict corresponding disease classification prediction labels.
4. The multi-modal three-dimensional medical image fusion method according to any one of claims 1 to 3, wherein the generator adopts a U-Net network structure, which includes an encoder and a decoder with a symmetric network structure; In the step c, the generator synthesizing the corresponding PET image according to the MRI image specifically includes: outputting the feature map of the MRI image through the feature extraction operation of the encoder multi-layer convolution; the decoder outputting the feature map to the encoder Perform a multi-layer deconvolution operation, and perform multiple stitching operations on the generated feature map and the feature map of the same size at the corresponding position of the encoder, and finally output the target reconstructed image, which is the synthesized PET image.
5. The multi-modal three-dimensional medical image fusion method according to claim 4, wherein in the step c, the classifier merges the MRI image of the subject to be detected and the synthesized PET image and then outputs the subject to be detected The disease classification prediction label specifically includes: extracting the feature value of the MRI image using a feature extraction network, and performing a convolution operation on the synthesized PET image to extract the feature value of the PET image; and stitching the feature value of the MRI image and the PET image , Compose the spliced feature values, and the fully connected layer will fusion and high-dimensional abstraction of the spliced feature values; the fused feature information is operated by the Softmax function to obtain the corresponding disease classification prediction label.
6. A multi-modal three-dimensional medical image fusion system is characterized by comprising:

   Model building module: used to construct a multi-task generative confrontation network, which includes a generator, a discriminator, and a classifier;

   Model training module: used to train the multi-task generation confrontation network according to the subject’s MRI images, PET images and diagnostic tag information, so that the multi-task generation confrontation network can automatically learn the association between MRI images and PET images feature;

   Model application module: used to input the MRI image of the person to be inspected into a trained multi-task generation confrontation network. The generator synthesizes the corresponding PET image according to the MRI image, and inputs the MRI image of the person to be inspected and the synthesized PET image A classifier, which merges the MRI image of the person to be detected and the synthesized PET image and outputs the disease classification prediction label of the person to be detected.
7. The multi-modal three-dimensional medical image fusion system according to claim 6, wherein the model training module comprises:

   Loss function construction unit: used to construct the confrontation loss function of the multi-task generation confrontation network; the confrontation loss in the training process is represented by an improved maximum-minimum cost function:

   

   In the above formula, (C, G, D) respectively represent the classifier, discriminator and generator, (x, y, z) respectively represent the MRI image, PET image and diagnostic label information; α ∈ (0, 1) is a Constant, used to control the proportion of classifier and generator loss in the training process, E _{(x,y,z)～p(x,y,z)} [log D(x,y,z)] represents the discriminator The samples from the real data distribution are judged as real samples;

   

   Indicates that the discriminator has identified a pair of pseudo samples in the output data space of the classifier;

   

   Indicates that the discriminator recognizes the pseudo sample label pair from the generator, where x represents the MRI modal image of the specimen, z represents the sample label, and G(x,z) represents the PET modal image synthesized by the conditional generation network;

   Classifier optimization unit: used to introduce the cross entropy loss under supervised learning to the classifier K _c =E _{(x,y,z)～p(x,y,z)} [-log p _c (x,y,z) ], the convergent point of the sample distribution p _c (x,y,z) of the classifier is limited to p(x,y), so that the global optimum of the model satisfies the sample distribution generated by generator G and classifier C and the real The data distribution is the same;

   Generator optimization unit: used to introduce generator supervision loss, using the gradient mutual information between the target image and the generated image as a similarity measure:

   K _g ＝NI(A,B)=G(A,B)·I(A,B)

   

   

   In the above formula, I (A, B) and G (A, B) respectively represent the gradient information and gradient difference between the generated image and the target image.
8. The multi-modal three-dimensional medical image fusion system according to claim 7, wherein the training of the multi-task generation confrontation network by the model training module is specifically: the generator takes the MRI image as a conditional constraint, and compares it with the target The random noise input of the same dimension of the image is mapped to the PET image. The discriminator determines whether the input sample distribution (x, y, z) comes from the real data distribution or the pseudo data distribution. The classifier uses a combination of MRI and PET images. The distribution is used as input and its label type is predicted; in the data-driven mode, with the gradual optimization of the generator, the discriminator updates the network parameters to identify the pseudo data distribution generated by the generator; with the optimization of the discriminator, the classifier is optimized to stimulate Make its predicted disease classification and prediction labels tend to be real data without being judged as fake data by the discriminator, and then reversely affect the training of the generator; through iterative confrontation training, the generator can learn MRI images and PET images Therefore, the corresponding PET image is synthesized from the input MRI image, and the classifier extracts key feature information from the input MRI image and PET image and predicts the corresponding disease classification prediction label.
9. The multi-modal three-dimensional medical image fusion system according to any one of claims 6 to 8, wherein the generator adopts a U-Net network structure, which includes an encoder and a decoder with a symmetric network structure; The generator synthesizing the corresponding PET image according to the MRI image specifically includes: outputting the feature map of the MRI image through the feature extraction operation of the encoder multi-layer convolution; the decoder performs the multi-layer deconvolution operation on the feature map output by the encoder , And perform multiple stitching operations on the generated feature map and the feature map of the same size at the corresponding position of the encoder, and finally output the target reconstructed image, which is the synthesized PET image.
10. The multi-modal three-dimensional medical image fusion system according to claim 9, wherein the classifier merges the MRI image of the subject to be detected and the synthesized PET image and then outputs the disease classification prediction label of the subject to be detected. : Use the feature extraction network to extract the feature values of the MRI image, and perform convolution operations on the synthesized PET image to extract the feature value of the PET image; stitch the feature values of the MRI image and the PET image to form the stitched feature value , The fully connected layer fusion and high-dimensional abstraction of the spliced feature values; the fused feature information is operated by the Softmax function to obtain the corresponding disease classification prediction label.
11. An electronic device including:

    At least one processor; and

    A memory communicatively connected with the at least one processor; wherein,

    The memory stores instructions that can be executed by the one processor, and the instructions are executed by the at least one processor, so that the at least one processor can execute the multi-mode operation described in any one of 1 to 5 above. The following operations of the three-dimensional medical image fusion method:

    Step a: construct a multi-task generative confrontation network, the multi-task generative confrontation network including a generator, a discriminator and a classifier;

    Step b: Training the multi-task generation confrontation network according to the subject's MRI image, PET image and diagnostic tag information, so that the multi-task generation confrontation network automatically learns the associated features between the MRI image and the PET image;

    Step c: Input the MRI image of the person to be inspected into the trained multi-task generating confrontation network, the generator synthesizes the corresponding PET image according to the MRI image, and inputs the MRI image of the person to be inspected and the synthesized PET image into the classifier, The classifier merges the MRI image of the person to be detected and the synthesized PET image and outputs the disease classification prediction label of the person to be detected.

Images