CN110544275B

CN110544275B - Methods, systems, and media for generating registered multi-modality MRI with lesion segmentation tags

Info

Publication number: CN110544275B
Application number: CN201910764408.9A
Authority: CN
Inventors: 瞿毅力; 王莹; 苏琬棋; 邓楚富; 卢宇彤; 陈志广
Original assignee: Sun Yat Sen University
Current assignee: Sun Yat Sen University
Priority date: 2019-08-19
Filing date: 2019-08-19
Publication date: 2022-04-26
Anticipated expiration: 2039-08-19
Also published as: CN110544275A

Abstract

The invention discloses a method, a system and a medium for generating registered multi-mode MRI with a focus segmentation label, wherein the method comprises the steps of obtaining a normally distributed random matrix and inputting the matrix into a decoder for generating trained structural features in a countermeasure network to decode and generate a structural feature graph; the structural characteristic graph and the randomly selected focus segmentation label graph are fused through random input to obtain a fusion result, and the fusion result is input to a trained random encoder in a generated countermeasure network to obtain a code; and generating decoders for resisting each trained mode in the network by using the coded input, and respectively generating the multi-mode MRI after registration. The generator in the generation countermeasure network is modularized into the encoder and the decoder, and through the combined training of a plurality of groups of encoders, decoders and discriminators, a random input meeting the design specification can be received to generate a group of registered multi-mode MRI images with focus segmentation labels, so that the method can be widely applied to the field of medical imaging.

Description

Methods, systems, and media for generating registered multi-modality MRI with lesion segmentation tags

Technical Field

The invention relates to an image generation technology in the medical field, in particular to a method, a system and a medium for generating a multi-modality MRI with a lesion segmentation tag, which are used for acquiring a multi-modality MRI image with a lesion segmentation tag according to given random input meeting the specification.

Background

With the development of deep learning, more and more fields begin to adopt deep neural networks to perform image processing tasks. However, training of deep neural networks requires a large amount of data, but in many fields, construction of data sets is very difficult. Therefore, the image generation technology has important applications in image intelligent processing scenes in many fields, such as medical images and biological cell images. In medical image intelligent processing, there are many modalities for medical images, such as Magnetic Resonance Imaging (MRI) X-ray, CT, and so on. When different modalities are obtained from the same part of the same patient by different imaging techniques, the modalities are considered to be registered if the imaging positions and viewing angles coincide. Compared with single-mode data, the registered multi-mode image data can provide more information, can support more complex application scenes, meets the requirement of training data of a deep neural network, and is beneficial to providing more efficient and reliable intelligent diagnosis service. However, medical image collection is very difficult, especially for rare diseases, making medical image data sets both scarce and small, which makes many training tasks impossible. Naturally, registered multi-modality image data is more scarce. Therefore, by applying image generation techniques, the generation of registered multimodal images has a wide range of uses and profound significance.

The generation of a countermeasure network (GAN) is a flexible deep neural network that can be subjected to unsupervised training and also supervised training, and has been widely used in the field of computer vision. The generation of a countermeasure network generally includes a generator that can generate realistic images by accepting random input and a discriminator that distinguishes between real images and generated images by learning them and thereby directs the generator to generate more realistic images. However, how to acquire a multi-modality MRI image with a lesion segmentation tag according to a given random input meeting the specification when generating the multi-modality MRI with the lesion segmentation tag, is still a key technical problem to be solved urgently.

Disclosure of Invention

The technical problems to be solved by the invention are as follows: the invention discloses a method, a system and a medium for generating registered multi-modal MRI with focus segmentation labels, aiming at the problems in the prior art, the invention modularizes a generator in a generation countermeasure network into an encoder and a decoder, can receive a random input meeting design specifications through the combined training of a plurality of groups of encoders, decoders and discriminators so as to generate a set of registered multi-modal MRI images with focus segmentation labels, and can be widely applied to the field of medical images.

In order to solve the technical problems, the invention adopts the technical scheme that:

a method of generating registered multi-modality MRI with lesion segmentation tags, the implementation steps comprising:

1) from a normal distribution N (0, 1)²) Obtaining random matrix Code_F,RM；

2) Code random matrix_F,RMTrained structural feature Decoder in input generation countermeasure network_FDecoding to generate a structural feature map F_RM；

3) Structural feature map F_RMObtaining a fusion result by random input fusion with a randomly selected focus segmentation label graph L;

4) inputting the fusion result into a trained random Encoder Encoder in the countermeasure network_RMObtaining a Code_RM；

5) Code to Code_RMI-modal Decoder for generating trained individual modal i in countermeasure network_iSeparately generating registered i-mode MRIi_g。

Optionally, step 2) is preceded by training generationDecoder for structural features in countermeasure networks_FThe detailed steps comprise:

A1) randomly selecting a mode, obtaining a graph n from the mode, extracting structural features to obtain a structural feature graph F, and extracting a Mask to obtain a corresponding Mask;

A2) encoder for generating structural features in countermeasure networks_FCoding the structural feature graph F to obtain a coding mean matrix Code_F,meanAnd variance matrix Code_F,logvarFrom a normal distribution N (0, 1)²) To obtain random noise Code_eFrom the mean matrix Code_F,meanAnd variance matrix Code_F,logvarRandom noise Code_eThe three codes are synthesized to obtain an approximate normal distribution matrix Code added with noise_F；

A3) Decoder with mask_MaskNormal distribution matrix Code_FDecoding to obtain reconstructed Mask_r(ii) a Decoder using structural features_FTo Code_FDecoding to obtain a reconstructed structural feature map F_r；

A4) Random generation of a normal distribution N (0, 1)²) Matrix Code of_F,RMDecoder using structural features_FCode matrix pair_F,RMDecoding to obtain a generated random structure characteristic diagram F_RMUsing a mask Decoder_MaskCode matrix pair_F,RMDecoding to obtain the generated random Mask_RM；

A5) Using a structural feature map Discriminator_FRespectively to (F, Mask) and (F)_RM,Mask_RM) Carrying out identification, wherein the former is identified as true, and the latter is identified as false; wherein F is a structural feature diagram, Mask is a Mask, F_RMIs a random structural feature map, Mask_RMIs a random mask; using a structural feature identifier FeatureDiscrimidator_FRespectively to Code_FAnd Code_F,RMPerforming identification to identify the former as false and the latter as true, wherein the Code_FTo approximate a normal distribution matrix, Code, after the addition of noise_F,RMFor random generation of a conforming normal distribution N (0, 1)²) A matrix of (a);

A6) calculating loss according to the output result of each step and the corresponding loss function, calling an optimizer to derive the loss function to obtain the gradient of the model parameter in each component, and then calculating the difference between each parameter and the corresponding gradient to complete the update of the network parameter;

A7) judging whether a preset iteration ending condition is met, wherein the iteration ending condition is that the loss function value is lower than a set threshold value or the iteration frequency reaches a set step number, and if not, skipping to execute the step A1); otherwise, exiting.

Optionally, step a2) is integrated to obtain an approximate normal distribution matrix Code after noise is added_FThe functional expression of (a) is:

Code_F＝Code_F,mean+exp(0.5*Code_F,logvar)*Code_e；

in the above formula, Code_F,meanEncoder for structural features_FMean matrix, Code, obtained by encoding the structural feature map F_F,logvarEncoder for structural features_FCode obtained by coding the structural feature map F_eIs normally distributed with N (0, 1)²) To obtain random noise.

Optionally, the detailed steps of step 3) include:

3.1) randomly selecting a focus segmentation label graph L, converting the focus segmentation label graph L containing a plurality of categories into a one-hot matrix to obtain a multi-dimensional label matrix with the same channel number and category number, wherein only part of pixels in each channel are effective, the rest parts are filled 0, and the non-0 pixel regions are registered with each segmentation region in the focus segmentation label graph;

3.2) dividing the multi-dimensional label matrix and the structural feature map F_RMStacking is carried out in the channel dimension together, and a matrix fusing two input sources is obtained as a fusion result.

Optionally, the step 4) is preceded by an i-mode Decoder for training each mode i_iI-mode Encoder Encoder_iAnd lesion segmentation label solutionDecoder_LAnd training the random Encoder Encoder_RMThe detailed steps comprise:

B1) i-modal Decoder for training each modal i_iI-mode Encoder Encoder_iAnd a lesion segmentation tag Decoder_LTraining random Encoder_RM；

B2) Calculating loss according to the output result of each training step and the corresponding loss function, calling an optimizer to conduct derivation on the loss function to obtain the gradient of the model parameter in each component, and then performing difference calculation on each parameter and the corresponding gradient to complete updating of the network parameter;

B3) judging whether a preset iteration ending condition is met, wherein the iteration ending condition is that the loss function value is lower than a set threshold value or the iteration frequency reaches a set step number, and if not, skipping to execute the step B1); otherwise, exiting.

Optionally, training the i-mode Decoder of each mode i in step B1)_iI-mode Encoder Encoder_iAnd a lesion segmentation tag Decoder_LThe detailed steps comprise:

step 1, inputting an original image i of a random modality i;

step 2, using an i-mode Encoder Encoder_iEncoding the original image i to obtain an encoded Code_i；

Step 3, using i-mode Decoder_iCode pair_iDecoding to obtain a reconstructed picture i_r(ii) a Decoder for focus segmentation label_LCode pair_iDecoding to obtain a focus segmentation label map L_i,f(ii) a Meanwhile, for any other mode j, firstly using a j-mode Decoder_jCode pair_iDecoding to obtain a j mode conversion graph j of the original graph i_tReuse the j mode Encoder Encoder_jJ mode conversion chart j for original image i_tCoding is carried out to obtain a Code_j,tReuse the i-mode Decoder_iCode pair_j,tDecoding to obtain a cyclic reconstruction image i of the original image i with j mode as an intermediate mode_c；

Step 4, respectively passing through a mode Discriminator_xI-mode conversion diagram i for converting original image i and each mode j into mode i_tDiscrimination is performed to discriminate the former as true and the latter as false.

Optionally, training the random Encoder in step B1)_RMThe detailed steps comprise:

step 1, randomly selecting a mode, and acquiring a graph n and a corresponding lesion segmentation label graph L from the mode_nObtaining a structural feature map F by using a structural feature extraction method₁Obtaining a corresponding Mask by using a Mask extraction method; using lesion segmentation label map L_nRemoving and extracting to obtain a structural feature map F₁Obtaining a structural characteristic diagram F without focus information;

step 2, the structural feature graph F and the randomly input focus segmentation label graph L are randomly input and fused to obtain a fusion result F_RM,expand；

Step 3, fusing the result F_RM,expandSending into a random Encoder Encoder_RMEncoding into Code_RM；

Step 4, Code is coded_RMDecoder for input focus segmentation label_LDecoding a reconstructed focus segmentation label map L_r(ii) a Simultaneously for each modality i: code to Code_RMI-mode Decoder for input mode i_iGet i Modal to generate graph i_gGenerating the i mode into a graph i_gExtracting structural features to obtain a structural feature map F_i,gGenerating the i mode into a graph i_gInput i-mode Encoder Encoder_iObtaining a Code_i,gCode to Code_i,gDecoder for input focus segmentation label_LDecode L_y,gA Decoder of j mode for inputting other modes j_jObtaining corresponding j mode to generate focus segmentation label graph j_g,t；

Step 5, aiming at each mode i, i mode Discriminator of each mode i_iGenerating a graph i for the original image n and i of the mode_gPerforming identification on the formerThe identification is true, the latter is false; respectively pairing the Code codes through a feature discriminator_RMAnd Code of each modality i_iPerforming identification to obtain Code_RMCode of each mode i identified as false_iThe authentication is true.

Further, the present invention provides a system for generating registered multi-modality MRI with lesion segmentation tags, comprising:

a random matrix generation program unit for generating a random matrix from a normal distribution N (0, 1)²) Obtaining random matrix Code_F,RM；

A structural feature extraction program unit for extracting a random matrix Code_F,RMTrained structural feature Decoder in input generation countermeasure network_FDecoding to generate a structural feature map F_RM；

Structural feature fusion program unit for fusing a structural feature map F_RMObtaining a fusion result by random input fusion with a randomly selected focus segmentation label graph L;

a random coding program unit for inputting the fusion result into a trained random coder Encoder in the countermeasure network_RMObtaining a Code_RM；

A registration structure feature map generation program unit for encoding the Code_RMI-modal Decoder for generating trained individual modal i in countermeasure network_iSeparately generating registered i-mode MRIi_g。

Furthermore, the present invention also provides a system for generating a registered lesion segmentation tagged multi-modality MRI, comprising a computer device programmed or configured to perform the steps of the method for generating a registered lesion segmentation tagged multi-modality MRI, or a computer program stored on a storage medium of the computer device and programmed or configured to perform the method for generating a registered lesion segmentation tagged multi-modality MRI.

Furthermore, the invention also provides a computer readable storage medium having stored thereon a computer program programmed or configured to perform the generating of the registered multi-modality MRI with lesion segmentation tags.

Compared with the prior art, the invention has the following advantages:

1. the generator in the generation countermeasure network is modularized into the encoder and the decoder, and through the combined training of a plurality of groups of encoders, decoders and discriminators, a random input meeting the design specification can be received to generate a group of registered multi-mode MRI images with focus segmentation labels, so that the method can be widely applied to the field of medical imaging.

2. The data used by the training of the invention does not need to be registered, is unsupervised learning, can realize the generation of multi-mode registration MRI, and the generated data is labeled, thus having no limit on the number of modes.

3. The invention adopts the modularized design, can conveniently carry out the mode expansion, leads the model training to be more flexible, leads the training to be carried out independently or synchronously, and leads the trained modules to be combined and reused when in use.

Drawings

FIG. 1 is a schematic diagram of the basic principle of the method according to the embodiment of the present invention.

FIG. 2 is a schematic diagram of a basic flow of a method according to an embodiment of the present invention.

FIG. 3 is a diagram of a training architecture generated from a structural feature diagram according to an embodiment of the present invention.

Fig. 4 is a diagram of an auxiliary training architecture in a modality registration image generation training according to an embodiment of the present invention.

Fig. 5 is a generation training architecture diagram in a modality registration image generation training of an embodiment of the present invention.

Fig. 6 is a main flow chart of a modality registration image generation training according to an embodiment of the present invention.

Detailed Description

The method, system and medium for generating registered lesion segmentation tagged multi-modality MRI of the present invention will be described in further detail below, using x and y modalities as an example. However, it should be noted that the method, system and medium for generating registered multi-modality MRI with lesion segmentation tags according to the present invention are not limited to two-modality registration image generation, but can be applied to three-modality and more registered multi-modality MRI generation.

As shown in fig. 1 and 2, the implementation steps of the method for generating registered multi-modality MRI with lesion segmentation tags of the present embodiment include:

1) from a normal distribution N (0, 1)²) Obtaining random matrix Code_F,RMCan be represented as N (0, 1)²)→Code_F,RM；

2) Code random matrix_F,RMTrained structural feature Decoder in input generation countermeasure network_F(abbreviated as DC)_F) Decoding to generate a structural feature map F_RM；

3) Structural feature map F_RMObtaining a fusion result by random input fusion with a randomly selected focus segmentation label graph L; referring to fig. 2, the randomly selected lesion segmentation label map L is obtained by randomly performing simple image transformation operations such as translation, flipping, rotation, scaling and the like on a real lesion label map;

4) inputting the fusion result into a trained random Encoder Encoder in the countermeasure network_RM(abbreviation EC)_R) Obtaining a Code_RM；

5) Code to Code_RMI-modal Decoder for generating trained individual modal i in countermeasure network_iSeparately generating registered i-mode MRIi_g. Referring to FIG. 1, taking x and y two modes as examples, Code will be encoded_RMInput generation resists the x modal Decoder of the good modal x of training in the network_x(abbreviated as DC)_x) Respectively generating registered i-mode generation structure feature maps x_g(ii) a Code to Code_RMInput generation confronts the y modal Decoder of good modal y in the network_y(abbreviated as DC)_y) Respectively generating registered i-mode generation structure feature map y_g。

The present embodiment has the following random input specifications for the method of generating registered lesion segmentation tagged multi-modality MRI: the random input comprises a random structural feature map and a random segmentation label map. The random structural feature map is generated by decoding from a standard normal distribution matrix by a decoder. The real structural feature map is extracted from a real image by a structural feature map extraction method based on a Sobel operator. The random segmentation label map is a segmentation label of each part in a randomly selected real mode image. The present embodiment is capable of receiving random input that meets this specification, generating a set of segmentation tagged, registered multi-modality MRI data.

The method also comprises training a Decoder for generating structural features in the countermeasure network before the step 2) of the embodiment_F(abbreviated as DC)_F) Training the Decoder for generating structural features in the countermeasure network_F(abbreviated as DC)_F) The step (a) can be independently trained, and after the training is completed, the generated random structural feature map is used for further generating a group of registered multi-modality MRI data with a lesion segmentation label. As shown in FIG. 3, the Decoder for structure feature in training generative countermeasure network in this embodiment_F(abbreviated as DC)_F) The detailed steps comprise:

A1) randomly selecting a mode, obtaining a graph n from the mode, performing structural feature extraction (in the embodiment, a Sobel operator is adopted, so that the Sobel operator is used for representing and extracting structural features in the graph) to obtain a structural feature graph F, and performing Mask extraction (represented by Mask in the graph) to obtain a corresponding Mask;

A2) encoder for generating structural features in countermeasure networks_FCoding the structural feature graph F to obtain a coding mean matrix Code_F,meanAnd variance matrix Code_F,logvarCan be expressed as Code_F,mean,Code_F,logvar＝Encoder_F(F) From a normal distribution N (0, 1)²) To obtain random noise Code_eCan be represented as N (0, 1)²)→Code_eFrom the mean matrix Code_F,meanAnd variance matrix Code_F,logvarRandom noise Code_eThe three codes are synthesized to obtain an approximate normal distribution matrix Code added with noise_F(ii) a Following trainingIncrease in number of iterations, Code_FWill be closer and closer to the normal distribution.

A3) Decoder with mask_MaskFor approximate normal distribution matrix Code_FDecoding to obtain reconstructed Mask_rCan be expressed as Mask_r＝Decoder_Mask(Code_F) (ii) a Decoder using structural features_FTo Code_FDecoding to obtain a reconstructed structural feature map F_rCan be represented as F_r＝Decoder_F(Code_F)；

A4) Random generation of a normal distribution N (0, 1)²) Matrix Code of_F,RMDecoder using structural features_F(abbreviated as DC)_F) Code matrix pair_F,RMDecoding to obtain a generated random structure characteristic diagram F_RMCan be represented as F_RM＝Decoder_F(Code_F,RM) (ii) a Decoder with mask_Mask(abbreviated as DC)_Mask) Code matrix pair_F,RMDecoding to obtain the generated random Mask_RMCan be expressed as Mask_RM＝Decoder_Mask(Code_F,RM)；

A5) Using a structural feature map Discriminator_F(abbreviation D)_F) Respectively to (F, Mask) and (F)_RM,Mask_RM) Carrying out identification, wherein the former is identified as true, and the latter is identified as false; wherein F is a structural feature diagram, Mask is a Mask, F_RMIs a random structural feature map, Mask_RMIs a random mask; using a structural feature identifier FeatureDiscrimidator_F(abbreviation FD)_F) Respectively to Code_FAnd Code_F,RMPerforming identification to identify the former as false and the latter as true, wherein the Code_FTo approximate a normal distribution matrix, Code, after the addition of noise_F,RMFor random generation of a conforming normal distribution N (0, 1)²) A matrix of (a);

In the training process of the foregoing steps a1) -a 7), it is desirable that the structural feature map decoded by the random normal distribution matrix in this embodiment is more realistic, so that the structural feature map Discriminator_FPerforming countermeasure learning on the structural features and masks extracted from the original image and the structural features and masks decoded by the random normal distribution matrix, and adding a generation countermeasure network GAN of coding features: through structural feature discriminator FeatureDiscrimidator_FCode for encoding result of real structure characteristic diagram_FAnd a matrix Code following a standard normal distribution_F,RMPerforming counterlearning to make Encoder Encoder_FThe structural feature map F can be coded as a result of following a standard normal distribution.

In this embodiment, the corresponding loss functions in step a6) include the following four loss functions (1) to (4):

(1) make the structural feature map code obey the normally distributed antagonism loss:

in the above formula, loss_{FeatureDiscriminator,1}Representing the loss of classification training of the feature discriminator, FeatureDiscrimidator_F(Code_F,RM) Expressed in Code_F,RMAs the result of discrimination output of the input feature discriminator, FeatureDiscrimator_F(Code_F) Expressed in Code_FAs the result of discrimination output of the input feature discriminator, omega_iFor the weight lost for each term (the same below), 0 and 1 indicate true or false, loss_Generator,1Representing the penalty of antagonism provided by the feature discriminator to the structural feature map encoder.

(2) The intermediate coding result of the structural feature graph obeys the normally distributed self-supervision loss:

in the above formula, loss_{Supervision,1}Representing constrained Code_FSupervised loss, mean (Code) obeying normal distribution_F,mean) Represents Code_F,meanMean of (mean of) (Code)_F,logvar) Represents Code_F,logvarIs measured.

(3) So that the structural characteristic graph decoded by the random normal distribution matrix is more vivid in adversity loss:

in the above formula, loss_{Discriminator,1}Representing the loss of classification training of the structural feature map discriminator,

Discriminator_F(F, Mask) represents the true and false identification output of the structural characteristic diagram identifier with F, Mask as the input,

Discriminator_F(F_RM,Mask_RM) Is represented by F_RM,Mask_RMFor true and false discrimination output of the input structural feature map discriminator, loss_Generator,2Representing the impedance penalty that the structural feature map evaluator provided to the structural feature map and the mask generation component.

(4) And (3) performing pairwise self-supervision consistency loss on the fused structural feature graph and mask with the original structural feature graph and mask:

in the above formula, loss_{Supervision,2}Representing the reconstructed auto-supervised loss of the texture feature map and the mask and the loss of consistency of the texture feature map and the mask, F representing the input texture feature map, F_rRepresenting the reconstructed structural features, Mask representing the input Mask, Mask_rMask representing the reconstruction, F_RMRepresentation from random generationThe structural characteristic graph, Mask, obtained by decoding the normal distribution matrix_RMRepresenting a mask decoded from a randomly generated normal distribution matrix.

Thus, the loss of the generator component is:

loss_Generator,F＝loss_Generator,1+loss_{Supervision,1}+loss_Generator,2+loss_{Supervision,2}

in the above formula, loss_Generator,FTo generate the total loss of components, loss_Generator,1Loss of antagonism for the feature discriminator to provide to the texture feature map encoder_{Supervision,1}Representing constrained Code_FLoss of supervision following a normal distribution_Generator,2Loss of contrast, which represents the structural feature pattern provided by the structural feature pattern discriminator to the structural feature pattern and mask generation component_{Supervision,2}Representing the reconstructed self-monitoring loss of the structural feature map and the mask and the consistency loss of the structural feature map and the mask.

The loss of the discriminator component is:

loss_{Discriminator,F}＝loss_{FeatureDiscriminator,1}+loss_{Discriminator,1}

in the above formula, loss_{Discriminator,F}For total loss of discriminator components, loss_{FeatureDiscriminator,1}Loss of class representing feature discriminators_{Discriminator,1}The classification training loss _ discriminator component and the generator component, which represent the structural feature map discriminator, are each updated separately. In this embodiment, the corresponding loss function in step a6) may also adopt other loss functions as needed.

It is well known that medical images generated directly from random noise by generating a counter-training are often difficult to train and to generate true structural information. In this embodiment, the image providing the basic contour structure information in the medical image is referred to as its structure feature map, for example, the retinal blood vessel distribution map can be regarded as the structure feature map of the retinal image. The structural feature map can provide necessary guiding information for the synthesis of medical images, for example, some studies obtain basic structural information from brain segmentation tag maps when synthesizing brain MRI images. However, the conventional structural feature maps such as the retinal vessel distribution map and the brain segmentation label map require additional data and training to extract the structural feature map from the original image. Therefore, the method for directly extracting the structural feature map is designed in the embodiment, and the method has the advantages of being fast in operation, free of training, free of additional data and the like. In the field of traditional digital image processing, people adopt Roberts operators, Prewitt operators, Sobel operators and the like which are all excellent edge detection operators. The Prewitt operator and Sobel operator are both templates of 3x3, and the resulting partial derivatives approximation is more accurate than that obtained with the templates of Roberts operator 2x 2. Compared with the Prewitt operator, the Sobel operator weights the influence of the position of the pixel, so that noise can be better suppressed, the edge blurring degree is reduced, and the effect is better. The Sobel operator is commonly used for processing medical images, and two gray level images are output after processing.

The detailed steps of extracting the structural features in step a1) in this embodiment are as follows: (1) inputting a real image n, wherein beta is a set pixel threshold; (2) respectively carrying out bitwise maximum value reduce _ max () on two outputs of the sobel operator to obtain f2 and bitwise minimum value reduce _ min () to obtain f 1; (3) difference values (mean (f1) -f1, f2-mean (f2) are obtained by respectively solving f1 and f2 and the mean pixel, and mean is a mean function), so that a low pixel edge feature map and a high pixel edge feature map are obtained; (4) and (3) performing binarization processing on the low pixel edge feature map and the high pixel edge feature map by using a pixel threshold value beta, and finally adding the two binary maps and performing binarization with a threshold value of 0 to obtain a clear structural feature map f. In this embodiment, the method for extracting the structural feature in step a1) is expressed as F ═ get _ F (n) using a function.

The detailed steps of mask extraction in step a1) in this embodiment are as follows: (1) firstly, carrying out binarization processing on an image n with a pixel threshold value of 0 to obtain a standard mask; (2) according to the size of the obtained input image, expanding p pixels of the original image size of the mask by adopting the nearest neighbor difference value, and grinding and amplifying the whole image; (3) and cutting the length and width of p pixels on the outermost layer to keep the size of the final output mask consistent with that of the original input image. In this embodiment, the method for performing Mask extraction in step a1) is expressed as Mask _ Mask (n) using a function.

In this embodiment, step a2) is performed to obtain an approximate normal distribution matrix Code after adding noise_FThe functional expression of (a) is:

Code_F＝Code_F,mean+exp(0.5*Code_F,logvar)*Code_e；

In this embodiment, the detailed steps of step 3) include:

3.1) randomly selecting a focus segmentation label graph L, converting the focus segmentation label graph L containing 5 categories into a one-hot matrix to obtain a multi-dimensional label matrix with the same channel number and category number, wherein only part of pixels in each channel are effective, the rest part of the channels are filled with 0, and the non-0 pixel regions are registered with each segmentation region in the focus segmentation label graph;

In this embodiment, before step 4), an i-mode Decoder for training each mode i is further included_iI-mode Encoder Encoder_iAnd a lesion segmentation tag Decoder_LAnd training the random Encoder Encoder_RMThe detailed steps comprise:

Referring to fig. 4, i-mode Decoder for training each mode i in step B1)_iI-mode Encoder Encoder_iAnd a lesion segmentation tag Decoder_LFor additional training, i-mode Decoder for training each mode i in step B1)_iI-mode Encoder Encoder_iAnd a lesion segmentation tag Decoder_LThe detailed steps comprise:

step 1, inputting an original image i of a random modality i;

step 2, using an i-mode Encoder Encoder_iEncoding the original image i to obtain an encoded Code_iIn this embodiment, Code_x＝Encoder_x(x)，Code_y＝Encoder_y(y)；

Step 3, using i-mode Decoder_iCode pair_iDecoding to obtain a reconstructed picture i_rIn this embodiment, x_r＝Decoder_x(Code_x)，y_r＝Decoder_y(Code_y) (ii) a Decoder for focus segmentation label_LCode pair_iDecoding to obtain a focus segmentation label map L_i,fIn this embodiment, L_x,f＝Decoder_L(Code_x)，L_y,f＝ Decoder_L(Code_y) (ii) a Meanwhile, for any other mode j, firstly using a j-mode Decoder_jCode pair_iDecoding to obtain a j mode conversion graph j of the original graph i_tIn this embodiment, y_t＝Decoder_y(Code_x)，x_t＝Decoder_x(Code_y) (ii) a Encoder using j-mode Encoder_jJ mode conversion chart j for original image i_tCoding is carried out to obtain a Code_j,tIn this embodiment, Code_x,t＝Encoder_x(x_t)，Code_y,t＝Encoder_y(y_t) Reuse the i-mode Decoder_iCode pair_j,tDecoding to obtain j-mode cyclic conversion graph i of original graph i_cIn this embodiment, x_c＝Decoder_x(Code_y,t)，y_c＝Decoder_y(Code_x,t)；

As shown in fig. 4, in the present embodiment, taking two modalities as an example, in a specific training process for generating images of the registered X, Y modalities from random input meeting the specification, the x-modality Decoder of the training modality x_xX-mode Encoder, Encoder_xAnd a lesion segmentation tag Decoder_LThe detailed steps comprise:

s1, inputting random X-mode image X;

s2 Encoder using x mode_xOriginal image x is coded to obtain Code_xDecoder for reuse_xTo Code_xDecoding to obtain a reconstructed picture x_r；

S3 Decoder for focus segmentation label_LTo Code_xDecoding to obtain a segmentation label graph L_x,f；

S4 Decoder using y mode_yTo Code_xDecoding to obtain a conversion graph y_t；

S5 Encoder using y mode_yFor conversion chart y_tCode is obtained by encoding_y,tReuse of x-mode Decoder_xTo Code_y,tDecoding to obtain cyclic conversion chart x_c；

S6 mode Discriminator_xFor x and x respectively_tPerforming identification to identify the former asTrue, the latter is identified as false.

The training process of the mode Y is the same as that of the mode Y, and the synchronous auxiliary training is mainly used for training the Encoder_x、Encoder_y、 Decoder_x、Decoder_yAnd Decoder_LModule for training the random Encoder Encoder in step B1)_RMThe generation training of (2) is easier to learn.

Referring to fig. 5 and 6, step B1) train the random Encoder_RMThe detailed steps comprise:

Step 3, fusing the result F_RM,expandSending into a random Encoder Encoder_RM(abbreviation EC)_RM) Encoding into Code_RM；

Step 4, Code is coded_RMDecoder for input focus segmentation label_L(abbreviated as DC)_L) Decoding a reconstructed focus segmentation label map L_r(ii) a Simultaneously for each modality i: code to Code_RMI-mode Decoder for input mode i_i(abbreviated as DC)_i) Get i Modal to generate graph i_gGenerating the i mode into a graph i_gExtracting structural features to obtain a structural feature map F_i,gGenerating the i mode into a graph i_gInput i-mode Encoder Encoder_i(abbreviation EC)_i) Obtaining a Code_i,gCode to Code_i,gDecoder for input focus segmentation label_L(abbreviated as DC)_L) Decode L_y,gA Decoder of j mode for inputting other modes j_j(abbreviated as DC)_j) Obtaining corresponding j mode to generate focus segmentation label graph j_g,t；

Step 5, aiming at each mode i, i mode Discriminator of each mode i_i(abbreviation D)_i) Generating a graph i for the original image n and i of the mode_gCarrying out identification, wherein the former is identified as true, and the latter is identified as false; respectively encoding Code by Feature Discriminator (FD)_RMAnd Code of each modality i_iPerforming identification to obtain Code_RMCode of each mode i identified as false_iThe authentication is true.

As shown in fig. 5 and fig. 6, in this embodiment, a specific generation training process of X, Y modality images and segmentation labels registered by inputting the random structure feature map and the random segmentation labels L is as follows:

s1, randomly selecting one mode at a time, and acquiring a graph n and a corresponding segmentation label graph L from the mode_nObtaining a structural feature map F by using a structural feature extraction method₁In the present example, it is represented as F₁Obtaining a corresponding Mask by using a Mask extraction method, which is denoted as Mask _ Mask (n) in the embodiment;

s2, utilizing the segmentation label graph L_nRemoving and extracting to obtain a structural feature map F₁The lesion information of (a) is obtained as a structural feature map F without lesion information, which is expressed as F (remove _ L) in the present embodiment_n,F₁) Wherein remove_L()A simple function for binarizing the segmentation label into a mask and then performing product calculation with the structural feature map so as to eliminate the focus part pixels;

s3, fusing the structural feature graph F with the randomly input segmentation label graph L to obtain F_RM,expandIn the present example, it is represented as F_RM,expandConcat (one ot (L), F), where concat () is a channel splicing function and one ot () is the aforementioned one-hot vector extension method;

s4, adding F_RM,expandSending into a random Encoder Encoder_RMCoded as Code_RMIn this example, Code is shown_RM＝Encoder_RM(F_RM,expand)；

S5, Code_RMRespectively to X-mode decoders_xDecoder of Y mode_yAnd a Decoder for segmenting the label graph_LGenerating a diagram X by decoding X modes respectively_gAnd a generation diagram Y of the Y mode_gReconstructing the segmentation label map L_rIn this embodiment, x is shown_g＝Decoder_x(Code_RM),y_g＝Decoder_y(Code_RM),L_r＝Decoder_L(Code_RM)；

S6, extracting structural features and generating a graph X from the X mode_gAnd generation diagram Y of the Y mode_gExtract characteristic graph F_x,g、 F_y,gThis embodiment can be expressed as F_x,g＝get_F(x_g),F_y,g＝get_F(y_g) Get _ F () is a structural feature map extraction method based on the sobel operator;

s7 Encoder using X mode_xGeneration diagram X for X modality_gCoding to obtain Code_x,gIn this example, Code is shown_x,g＝Encoder_x(x_g),Code_y,g＝Encoder_y(y_g)；

S8、Code_x,gBy a Decoder_LDecoding to obtain L_x,gIn this example, it is represented as L_x,g＝Decoder_L(Code_x,g),L_y,g＝Decoder_L(Code_y,g)；

S9、Code_x,gBy a Decoder_yDecoding to obtain y_g,tIn the present embodiment, is represented by y_g,t＝Decoder_y(Code_x,g),x_g,t＝Decoder_x(Code_y,g)；

S10, similarly, the same operation is carried out on the Y mode, and an Encoder Encoder is used_yGenerate the graph y_gCoding to obtain Code_y,g；

S11, sending to Decoder_LDecoding to obtain L_y,g；

S12、Code_y,gSent to decodingDecoder_xDecoding to obtain x_g,t。

S13 mode X Discriminator_xFor x and x respectively_gDiscrimination is performed to discriminate the former as true and the latter as false. Modal Y Discriminator_yFor y and y respectively_gDiscrimination is performed to discriminate the former as true and the latter as false. Feature discriminator for Code_RM、Code_xAnd Code_yPerforming identification to obtain Code_RMThe identification is false, the latter two are identified as true. In the above image generation training process, the structural feature map F₁Is extracted from the random image n, the extracted structural features may contain focus structural information, which may interfere with the focus information in the random label L and affect the image generated after fusion, so F₁The lesion information needs to be eliminated before being fused with the random label L to obtain a structural feature map F without lesion information, so that the lesion information of the generated image is only from the label L.

In this embodiment, the loss functions of the multimodal map generation training and the auxiliary training process include the following loss functions (5) to (15):

(5): the antagonism loss that makes the X, Y modal graph generated by the random structural feature graph more realistic:

in the above formula, loss_{Discriminator,2}Representing a loss of classification training of the modality Discriminator on the x-modality, Discriminator_x(x) Representing the output of a modality discriminator with x as input, loss_Generator,3Representing the loss of antagonism, loss, of the x-modality provided to the generating component by the modality discriminator_{Discriminator,3}Representing a loss of classification training of the modality Discriminator on the y-modality, Discriminator_y(y) represents the output of the modality discriminator with y as input, loss_Generator,4Representing the antagonism loss of the y-modality provided to the generating component by the modality discriminator.

(6): the encoding result of the random structure characteristic diagram is more approximate to the antagonism loss of the encoding result of the real mode diagram (so as to reduce the decoding difficulty of the decoder and ensure that the decoder can successfully decode the mode diagram).

In the above formula, loss_{FeatureDiscriminator,2}Loss of class training representing the feature discriminator_Generator,5Representing the penalty provided by the feature discriminator to the texture feature map encoder to direct the encoder to encode the input fused map into an encoding result that approximates the encoding result of the real mode map, FeatureDiscriminitor (Code)_RM) Expressed in Code_RMThe true and false authentication output of the input feature authenticator.

(7) Reconstruction of the input structural feature map self-supervision loss:

in the above formula, loss_{supervision,3}Represents the reconstruction of the structural feature map and the loss of the self-supervision, and removeL (L, F) represents the structural feature of the disease-free information obtained by removeL () of L and FFeature output, removeL (L, F)_x,g) Denotes that L and F_x,gStructural feature map output of lesion-free information by remove (), where remove_L()A simple function for binarizing the segmentation label into a mask and then performing product calculation with the structural feature map so as to eliminate the focus part pixels;

(8) and (3) reconstructing the input focus segmentation label graph after being fused with the input structural feature graph, wherein the reconstruction loss is as follows:

in the above formula, loss_{supervision,4}Representing reconstructed unsupervised loss of a lesion segmentation label map, L being an input label, L_x,gFor generation of tags for x modality, L_rIs a directly reconstructed tag.

(9) X, Y supervised loss of modal graph segmentation training:

in the above formula, loss_{supervision,5}Supervised loss, L, training for X, Y mode graph segmentation_xTrue tags for the x modality, L_x,fTags are generated for the x modality.

(10) Converting the generated X-mode and Y-mode graphs to obtain a conversion graph and generating the self-supervision loss of the graph:

in the above formula, loss_{supervision,6}Conversion graph obtained by converting generated X-mode and Y-mode graphs and self-supervision loss, X, of generated graphs_gFor generating an MRI of modality x, x_g,tIn order to convert the generated y-mode MRI into the x-mode MRI, the registration constraint of the generated multi-mode MRI is realized through the loss, and when the generated multi-mode MRI is expanded to more modes, the generated graph of each mode is similar to different modesThe mean square error loss is calculated from the conversion map of the mode into which the state generation map is converted.

(11) Supervision loss limiting the pixel generation range to the range of the organ body mask:

in the above formula, loss_{supervision,7}And the Mask is a binary Mask map, the organ range of the organ described by the Mask map is 0 value, and the range outside the organ described by the Mask map is 1 value.

(12) The self-supervision loss of the original image and a reconstructed image obtained by reconstructing the X-mode image and the Y-mode image is as follows:

in the above formula, loss_{supervision,8}The self-supervision loss of the original image and the reconstructed image obtained by reconstructing the X-mode image and the Y-mode image is shown.

(13) And (3) a loop conversion graph obtained by converting the X-mode graph and the Y-mode graph and an original graph have supervision loss:

in the above formula, loss_supervisin,cIndicating the supervised loss, X, of the original graph and the cyclic conversion graph obtained by converting the X-mode graph and the Y-mode graph_cAnd converting the MRI in the x mode into the MRI in the y mode, and then converting the MRI in the y mode into the circular reconstruction image of the MRI in the x mode, wherein when the X mode is expanded to more modes, the mean square error loss is calculated by the original image and the circular reconstruction images in different intermediate modes.

(14) Generating the self-supervision semantic consistency loss of the codes of the X mode and the Y mode graph and the codes obtained by the X mode and the Y mode graph through the encoder by the decoder:

in the above formula, loss_{Supervision,9}Code representing the loss of self-supervised semantic consistency between the Code generated by the decoder into the X-mode and Y-mode maps and the Code obtained by the X-mode and Y-mode maps through the encoder_x,gAnd recoding the obtained coding result for the generated graph of the x mode.

(15) Self-supervised semantic consistency loss of X-modality and Y-modality graph coding:

in the above formula, loss_{Supervision,10}Representing the loss of self-supervised semantic consistency of true X-mode and Y-mode graph coding, Code_xCode representing the result of MRI encoding of the true x-mode_y,tAnd when the MRI representing the real x modality is converted into the recoding result after the MRI of the y modality is expanded to more modalities, the mean square error loss is calculated by the original image coding result and the recoding result of different intermediate modalities.

The generating component in the multi-modal graph generating training and auxiliary conversion training process is a common component, and synchronously trains and updates.

In the multi-modal graph generation training and auxiliary training process of the embodiment, the total loss of the discriminator component is as follows:

loss_{Discriminator}＝∑_i＝2loss_{Discriminator,i}+∑_i＝2loss_{FeatureDiscriminator,i}

in the above formula, loss_{Discriminator}Represents the loss of the total discriminator component, Σ_i＝2loss_{Discriminator,i}Represents the total loss, Σ, of the mode discriminator_i＝2loss_{FeatureDiscriminator,i}Representing the total loss of the feature discriminator.

The total loss of the resulting component is:

loss_Generator＝∑_i＝3loss_Generator,i+∑_i＝3loss_{Supervision,i}

in the above formula, loss_GeneratorRepresents the total loss of the generator component, Σ_i＝3loss_Generator,iiRepresents the total antagonism loss, Σ, provided by the discriminator_i＝3loss_{supervision,i}The overall lesion signature is represented by a supervised loss and an individual self-supervised loss.

The above loss functions are not illustrated by mode expansion, and can directly increase the loss terms of the corresponding modes according to the number of the generated and converted modes, and are not limited to the two modes x and y in this embodiment.

For the training of a plurality of modes, only the mode registration image generation training method is needed to supplement the mode conversion training.

When training n (n >1) modes, the auxiliary training needs to perform n-mode training, each mode comprises n-1 steps S4-S5, and the training is respectively corresponding to the cycle conversion training of the current mode and other n-1 modes; in the generating training step S5, a label graph and a generating graph of n modalities are generated, each modality includes n-1 steps S9, and the conversion training is respectively corresponding to the current modality and other n-1 modalities.

The generation countermeasure network of this embodiment has an encoder for receiving random input, a decoder for decoding an input segmentation label map from the encoded result, and a feature discriminator for discriminating the encoded result from true or false, and then, for each mode, there are an encoder, a decoder, and a mode discriminator. In addition, the embodiment includes a feature extraction module for extracting structural features from a real image, an encoder for encoding the structural feature map into a normal distribution, a decoder for decoding the structural feature map from the normal distribution, a true and false structural feature map discriminator for guiding the reconstruction of the structural feature map, and a structural feature discriminator for performing true and false discrimination on the encoding result. After the training of the steps A) -A7) and the steps B1) -B3) is completed, only part of module components in the steps A) -A7) and the steps B1) -B3) need to be recombined, and then a large number of multi-modal registration images can be generated conveniently and quickly. From a normal distribution N (0, 1)²) Obtaining random matrix Code_F,RMBy usingTrained structural feature Decoder_FDecoding to generate a structural feature map F_RM，F_RMFusing with randomly selected segmentation label graph L, and inputting the fused result into the trained random input Encoder Encoder_RMTo obtain a Code_RMFinally, the Decoder of different modes is trained_x、Decoder_yTo Code_RMDecoding to generate a registered X mode image X_gY mode diagram Y_g。

Compared with the prior art, the method for generating the registered multi-modality MRI with the lesion segmentation tag in the embodiment has the following advantages: 1) data used for training is not required to be registered, unsupervised learning is adopted, multi-mode registration MRI images can be generated, generated data are labeled, and the number of modes is not limited. 2) The modularized design can conveniently carry out mode expansion, and make model training more flexible, training can be independently carried out or can be carried out synchronously, and the trained modules are combined and reused when in use. 3) The method for extracting the structural features is improved based on the traditional Sobel operator method, the Sobel operator is used for extracting the features and then further processing is carried out, a maximum value graph and a minimum value graph are obtained, and finally, the structural feature graphs are obtained through fusion, and enough structural information is reserved.

In the above formulas of this embodiment, x, y, and x_r、y_r、x_t、y_t、x_c、y_c、x_g、y_g、x_g,t、y_g,tX, Y original image, reconstruction graph, conversion graph, circulation conversion graph, generation graph and generation conversion graph, Encoder_x、 Encoder_y、Decoder_x、Decoder_yCoder and decoder, respectively, representing a modality X, Y_x、Code_y、Code_x,t、 Code_y,t、Code_x,g、Code_y,gRepresents pairs x, y, x_t、y_t、x_g、y_gBy corresponding Encoder Encoder_x、Encoder_yAnd respectively coding the obtained characteristic results. F. F_x,g、F_y,gRepresenting the structural feature extraction method get _ F on any modal graph n and x_g、y_gExtracting the structure characteristic diagram, generating the structure characteristic diagram in an X mode, generating the structure characteristic diagram in a Y mode, wherein the Mask represents a Mask extracted by a Mask extraction method get _ Mask, and a Decoder_F、Decoder_Mask、Decoder_L、Encoder_RMRespectively, a structural feature decoder, a mask decoder, a segmentation tag decoder, and a random input encoder. Code_F,mean、Code_F,logvarRepresentation pair F passes through Encoder Encoder_FCharacteristic result obtained by encoding, Code_e、Code_F,RMRepresents a distribution from normal N (0, 1)²) Randomly acquired matrix, Code_FRepresenting the normal distribution matrix after the addition of noise, F_r、F_RMRepresent respective pair Code_F、Code_F,RMBy Decoder_FThe reconstructed structure characteristic diagram, the random structure characteristic diagram and the Mask are obtained by decoding_r、Mask_RMRepresent respective pair Code_F、Code_F,RMBy Decoder_MaskA reconstruction mask, a random mask, F, obtained by decoding_RM,expandIs represented by F_RMA feature tag map, Code, obtained by fusing with the segmentation tag map L_RMRepresents a pair F_RM,expandBy Encoder Encoder_RMThe characteristic result obtained by coding, L_x,f、L_y,f、L_r、L_x,g、L_y,gRepresent respective pair Code_x、Code_y、Code_RM、Code_x,g、 Code_y,gBy Decoder_LDecoding obtained X-mode segmentation label graph, Y-mode segmentation label graph and reconstruction segmentation label graph L_rAnd generating a segmentation label map in an X mode and generating a segmentation label map in a Y mode. Discriminator, also mentioned earlier in relation to the training method_FIs a structural feature diagram discriminator, FeatureDiscrimidator_FIs a structural feature identifier, Discriminator_x、 Discriminator_yShown is a discriminator for modality X, Y, FeatureDiscrimidator is a multipleA feature discriminator common to the modalities.

Further, the present embodiments also provide a system for generating registered multi-modality MRI with lesion segmentation tags, comprising:

Furthermore, the present embodiment also provides a system for generating registered lesion segmentation tagged multi-modality MRI, comprising a computer device programmed or configured to execute the steps of the aforementioned method for generating registered lesion segmentation tagged multi-modality MRI, or a storage medium of the computer device having stored thereon a computer program programmed or configured to execute the aforementioned method for generating registered lesion segmentation tagged multi-modality MRI.

Furthermore, the present embodiments also provide a computer readable storage medium having stored thereon a computer program programmed or configured to perform the aforementioned method of generating registered lesion segmentation tagged multi-modality MRI.

The above description is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments, and all technical solutions belonging to the idea of the present invention belong to the protection scope of the present invention. It should be noted that modifications and embellishments within the scope of the invention may occur to those skilled in the art without departing from the principle of the invention, and are considered to be within the scope of the invention.

Claims

1. A method of generating registered multi-modality MRI with lesion segmentation tags, characterized by the implementation steps comprising:

1) from a normal distribution N (0, 1)²) Obtaining random matrix Code_F，RM；

2) Code random matrix_F，RMTrained structural feature Decoder in input generation countermeasure network_FDecoding to generate a structural feature map F_RM；

5) Code to Code_RMI-modal Decoder for generating trained individual modal i in countermeasure network_iSeparately generating registered i-mode MRI i_g。

2. The method of generating registered multi-modality MRI with lesion segmentation tags according to claim 1, wherein step 2) is preceded by training a structure feature Decoder in a generation countermeasure network_FThe detailed steps comprise:

A2) encoder for generating structural features in countermeasure networks_FCoding the structural feature graph F to obtain a coding mean matrix Code_F，meanAnd variance matrix Code_F，logvarFrom a normal distribution N (0)，1²) To obtain random noise Code_eFrom the mean matrix Code_F，meanAnd variance matrix Code_F，logvarRandom noise Code_eThe three codes are synthesized to obtain an approximate normal distribution matrix Code added with noise_F；

A3) Decoder with mask_MaskFor the approximately normal distribution matrix Code after adding noise_FDecoding to obtain reconstructed Mask_r(ii) a Decoder using structural features_FTo Code_FDecoding to obtain a reconstructed structural feature map F_r；

A4) Random generation of a normal distribution N (0, 1)²) Matrix Code of_F，RMDecoder using structural features_FCode matrix pair_F，RMDecoding to obtain a generated random structure characteristic diagram F_RMUsing a mask Decoder_MaskCode matrix pair_F，RMDecoding to obtain the generated random Mask_RM；

A5) Using a structural feature map Discriminator_FRespectively to (F, Mask) and (F)_RM，Mask_RM) Carrying out identification, wherein the former is identified as true, and the latter is identified as false; wherein F is a structural feature diagram, Mask is a Mask, F_RMIs a random structural feature map, Mask_RMIs a random mask; using a structural feature identifier FeatureDiscrimidator_FRespectively to Code_FAnd Code_F，RMPerforming identification to identify the former as false and the latter as true, wherein the Code_FFor normal distribution matrices, Code, after the addition of noise_F，RMFor random generation of a conforming normal distribution N (0, 1)²) A matrix of (a);

A6) calculating loss according to the output result of each step and the corresponding loss function, calling an optimizer to derive the loss function to obtain the gradient of the model parameter in each component, and then calculating the difference between each parameter and the corresponding gradient to complete the update of the network parameter; each component comprises a structural feature Encoder Encoder_FDecoder for mask_MaskDecoder for decoding structural characteristics_FStructural feature diagram Discriminator_FAnd a structural feature identifier FeatureDiscrimidator_F；

3. The method for generating registered multi-modality MRI with lesion segmentation labels as claimed in claim 2, wherein the step A2) is integrated to obtain an approximately normal distribution matrix Code with noise added_FThe functional expression of (a) is:

Code_F＝Code_F，mean+exp(0.5*Code_F，logvar)*Code_e；

in the above formula, Code_F，meanEncoder for structural features_FMean matrix, Code, obtained by encoding the structural feature map F_F，logvarEncoder for structural features_FCode obtained by coding the structural feature map F_eIs normally distributed with N (0, 1)²) To obtain random noise.

4. The method of generating registered lesion segmentation tagged multi-modality MRI as set forth in claim 1, wherein the detailed steps of step 3) include:

5. The banding of generating registration of claim 1The multi-mode MRI method of focus segmentation labels is characterized in that the step 4) is preceded by an i-mode Decoder for training each mode i_iI-mode Encoder Encoder_iAnd a lesion segmentation tag Decoder_LAnd training the random Encoder Encoder_RMThe detailed steps comprise:

B2) Calculating loss according to the output result of each training step and the corresponding loss function, calling an optimizer to derive the loss function to obtain the gradient of the model parameter in each component, and then calculating the difference between each parameter and the corresponding gradient to complete the update of the network parameter; the components include i-mode Decoder of i-mode_iI-mode Encoder Encoder_iFocus segmentation label Decoder_LAnd a random Encoder Encoder_RM；

6. Method for generating registered lesion segmentation tagged multi-modality MRI according to claim 5, characterized in that in step B1) an i-mode Decoder of each modality i is trained_iI-mode Encoder Encoder_iAnd a lesion segmentation tag Decoder_LTraining the i-mode Decoder of a certain mode i_iI-mode Encoder Encoder_iAnd a lesion segmentation tag Decoder_LThe detailed steps comprise:

step 1, inputting an original image i of a random modality i;

Step 3, usingDecoder of i-mode_iCode pair_iDecoding to obtain a reconstructed picture i_r(ii) a Decoder for focus segmentation label_LCode pair_iDecoding to obtain a focus segmentation label map L_i，f(ii) a Meanwhile, for any other mode j, firstly using a j-mode Decoder_jCode pair_iDecoding to obtain a j mode conversion graph j of the original graph i_tReuse the j mode Encoder Encoder_jJ mode conversion chart j for original image i_tCoding is carried out to obtain a Code_j，tReuse the i-mode Decoder_iCode pair_j，tDecoding to obtain a cyclic reconstruction image i of the original image i with j mode as an intermediate mode_c；

7. Method for generating registered lesion segmentation tagged multi-modality MRI according to claim 5, characterized in that a stochastic Encoder Encoder is trained in step B1)_RMThe detailed steps comprise:

step 2, the structural feature graph F and the randomly input focus segmentation label graph L are randomly input and fused to obtain a fusion result F_RM，expand；

Step 3, fusing the result F_RM，expandSending into a random Encoder Encoder_RMEncoding into Code_RM；

Step 4, Code is coded_RMDecoder for input focus segmentation label_LDecoding out of weightEstablishing a focus segmentation label chart L_r(ii) a Simultaneously for each modality i: code to Code_RMI-mode Decoder for input mode i_iGet i Modal to generate graph i_gGenerating the i mode into a graph i_gExtracting structural features to obtain a structural feature map F_i，gGenerating the i mode into a graph i_gInput i-mode Encoder Encoder_iObtaining a Code_i，gCode to Code_i，gDecoder for input focus segmentation label_LDecode L_y，gA Decoder of j mode for inputting other modes j_jObtaining corresponding j mode to generate focus segmentation label graph j_g，t；

Step 5, aiming at each mode i, i mode Discriminator of each mode i_iGenerating a graph i for the original image n and i of the mode_gCarrying out identification, wherein the former is identified as true, and the latter is identified as false; respectively pairing the Code codes through a feature discriminator_RMAnd Code of each modality i_iPerforming identification to obtain Code_RMCode of each mode i identified as false_iThe authentication is true.

8. A system for generating registered multi-modality MRI with lesion segmentation tags, comprising:

a random matrix generation program unit for generating a random matrix from a normal distribution N (0, 1)²) Obtaining random matrix Code_F，RM；

A structural feature extraction program unit for extracting a random matrix Code_F，RMTrained structural feature Decoder in input generation countermeasure network_FDecoding to generate a structural feature map F_RM；

9. A system for generating registered lesion segmentation tagged multi-modality MRI comprising a computer device, characterized in that the computer device is programmed or configured to perform the steps of the method for generating registered lesion segmentation tagged multi-modality MRI of any one of claims 1 to 7, or a storage medium of the computer device has stored thereon a computer program programmed or configured to perform the method for generating registered lesion segmentation tagged multi-modality MRI of any one of claims 1 to 7.

10. A computer readable storage medium having stored thereon a computer program programmed or configured to perform the method of generating registered lesion segmentation tagged multi-modality MRI of any one of claims 1-7.