CN116630463A - Enhanced CT image generation method and system based on multitask learning - Google Patents

Enhanced CT image generation method and system based on multitask learning Download PDF

Info

Publication number
CN116630463A
CN116630463A CN202310896717.8A CN202310896717A CN116630463A CN 116630463 A CN116630463 A CN 116630463A CN 202310896717 A CN202310896717 A CN 202310896717A CN 116630463 A CN116630463 A CN 116630463A
Authority
CN
China
Prior art keywords
image
enhanced
generator
representing
segmentation
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202310896717.8A
Other languages
Chinese (zh)
Other versions
CN116630463B (en
Inventor
曲昂
张泽斌
贺树萌
王俊杰
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing Lianying Intelligent Imaging Technology Research Institute
Peking University Third Hospital Peking University Third Clinical Medical College
Original Assignee
Beijing Lianying Intelligent Imaging Technology Research Institute
Peking University Third Hospital Peking University Third Clinical Medical College
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing Lianying Intelligent Imaging Technology Research Institute, Peking University Third Hospital Peking University Third Clinical Medical College filed Critical Beijing Lianying Intelligent Imaging Technology Research Institute
Priority to CN202310896717.8A priority Critical patent/CN116630463B/en
Publication of CN116630463A publication Critical patent/CN116630463A/en
Application granted granted Critical
Publication of CN116630463B publication Critical patent/CN116630463B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T11/00—2D [Two Dimensional] image generation
    • G06T11/003—Reconstruction from projections, e.g. tomography
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/0464—Convolutional networks [CNN, ConvNet]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/0475—Generative networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/084—Backpropagation, e.g. using gradient descent
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/12—Edge-based segmentation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10072—Tomographic images
    • G06T2207/10081—Computed x-ray tomography [CT]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
    • Y02T10/00—Road transport of goods or passengers
    • Y02T10/10—Internal combustion engine [ICE] based vehicles
    • Y02T10/40—Engine management systems

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Computational Linguistics (AREA)
  • Data Mining & Analysis (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Artificial Intelligence (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • Software Systems (AREA)
  • Health & Medical Sciences (AREA)
  • Evolutionary Computation (AREA)
  • General Health & Medical Sciences (AREA)
  • Molecular Biology (AREA)
  • Computing Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Mathematical Physics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Apparatus For Radiation Diagnosis (AREA)

Abstract

The application relates to an enhanced CT image generation method and system based on multi-task learning, belongs to the technical field of CT image generation, and solves the problem that an enhanced CT image cannot be obtained quickly and safely in the prior art. The method comprises the following steps: acquiring a plain scan CT image and an enhanced CT image; respectively carrying out image segmentation on the plain CT image and the enhanced CT image to obtain corresponding organ contour segmentation images; constructing a training sample set according to the plain CT image, the enhanced CT image and the corresponding organ contour segmentation image; constructing a multi-task circulation generation countermeasure network model, and training the multi-task circulation generation countermeasure network model based on the training sample set to obtain a trained multi-task circulation generation countermeasure network model; and acquiring a plain scan CT image to be converted, generating a corresponding enhanced CT image based on the trained multi-task circulation generation countermeasure network model. The rapid generation of high-quality enhanced CT images is realized.

Description

Enhanced CT image generation method and system based on multitask learning
Technical Field
The application relates to the technical field of CT image generation, in particular to an enhanced CT image generation method and system based on multi-task learning.
Background
In radiodiagnosis and radiotherapy, enhanced computed tomography (CECT) has unique advantages over Non-enhanced computed tomography (Non-Contrast enhanced CT, NECT) and plays an important role. When CECT scanning is carried out, the iodine-containing contrast agent is injected into veins of a patient, and dynamic distribution and excretion of the contrast agent in organs and tissues are observed through scanning at different time points, so that the contrast of blood vessels and tissues is increased, and the organs and lesions are more clearly visible. This is critical for imaging diagnosis of disease, tumor localization in radiation therapy, especially for automated tumor segmentation and delineation, and for assessing tumor blood supply and therapeutic effects.
However, contrast agents may trigger allergic reactions and nephrotoxicity, and are contraindicated for patients with impaired renal function. In addition, multi-stage CECT scanning can extend scanning time and increase radiation exposure, which can be detrimental to the health of radiation-sensitive populations such as children.
Disclosure of Invention
In view of the above analysis, the present application aims to provide an enhanced CT image generating method and system based on multi-task learning, so as to solve the problem that an enhanced CT image cannot be obtained quickly and safely in the prior art.
In one aspect, an embodiment of the present application provides a method for generating an enhanced CT image based on multi-task learning, including the steps of:
acquiring a plain scan CT image and an enhanced CT image; respectively carrying out image segmentation on the plain CT image and the enhanced CT image to obtain corresponding organ contour segmentation images; constructing a training sample set according to the plain CT image, the enhanced CT image and the corresponding organ contour segmentation image;
constructing a multi-task circulation generation countermeasure network model, and training the multi-task circulation generation countermeasure network model based on the training sample set to obtain a trained multi-task circulation generation countermeasure network model;
and acquiring a plain scan CT image to be converted, generating a corresponding enhanced CT image based on the trained multi-task circulation generation countermeasure network model.
Based on further improvement of the technical scheme, the multitasking cycle generating countermeasure network model comprises an enhanced CT generator, a flat scan CT generator, an enhanced CT discriminator and a flat scan CT discriminator;
the enhanced CT generator is used for converting the input flat scan CT image into an enhanced CT image and generating an enhanced organ contour segmentation image according to the input flat scan CT image; the enhanced CT discriminator is used for judging the authenticity of the enhanced CT image obtained by conversion;
the flat scan CT generator is used for converting the input enhanced CT image into a flat scan CT image and generating a flat scan organ contour segmentation image according to the input enhanced CT image; the flat scanning CT discriminator is used for judging the authenticity of the flat scanning CT image obtained through conversion.
Based on a further improvement of the above technical solution, the multitasking loop generation is against the network model to perform consistency loss constraint at the image level, the segmentation contour level and the region of interest level.
Based on the further improvement of the technical scheme, the following formula is adopted to calculate the total loss of the multi-task circulation generation countermeasure network model：
，
wherein ,GN2C Representation enhanced CT generator, G C2N Represents a flat scan CT generator, D C Representation enhanced CT discriminator, D N Representing the horizontal scanning CT discriminator,indicating loss of antagonism of the arbiter, ++>Representing a loss of consistency constraint at the image level, +.>Representing the segmentation loss of the segmentation contour level,region of interest perceived loss, represented by a region of interest hierarchy, λ1, λ2, λ3, and λ4 represent weighting coefficients.
Based on the further improvement of the technical scheme, the consistency constraint loss of the image level is calculated by adopting the following formula:
，
wherein N represents an input plain CT image, C represents an input enhanced CT image, G C2N (G N2C (N)) represents a flat scan CT image output after the enhanced CT image obtained by the enhanced CT generator is input to the flat scan CT generator; g N2C (G C2N (C) A) represents an enhanced CT image output after the flat scan CT image obtained by the flat scan CT generator is input into the enhanced CT generator, S C Corresponding organ contour segmented image representing an input enhanced CT image, S N Corresponding organ contour segmented image, ║. ║, representing an input plain CT image 1 Representing a norm of the matrix,representing global loop consistency constraint loss, +.>Representing a region of interest cyclic uniformity constraint loss.
Based on a further improvement of the above technical solution, the segmentation loss is calculated according to the following formula:
，
wherein ,representing an enhanced organ contour segmentation image output by an enhanced CT generator, < >>Representing a flat scan organ contour segmentation image output by a flat scan CT generator, < >>Representing the organ contour segmentation image output after inputting the enhanced CT image obtained by the enhanced CT generator into the plain CT generator,/for the enhanced CT image>Representing the segmented image of the enhanced organ contour output after the flat scan CT image obtained by the flat scan CT generator is input into the enhanced CT generator,representing the Dice loss.
Based on a further improvement of the technical scheme, the region of interest perceived loss is calculated according to the following formula:
，
wherein ,representing the flat scan organ contour segmentation image output by the flat scan CT generator,representing a flat scan organ contour segmentation image output after inputting the enhanced CT image obtained by the enhanced CT generator into the flat scan CT generator, < >>Representing a perceived loss;
the perceptual loss is calculated using the following formula:
calculating the perceived loss of the two images x and x', ∈>Feature map representing output of ith layer of pre-trained neural network, N P Representing the number of feature extraction layers of a pre-trained neural network ║. ║ 1 Representing a norm of the matrix.
Based on the further improvement of the technical scheme, the antagonism loss of the discriminator is calculated according to the following formula:
，
wherein E [. Cndot.]Indicating desire, D N (G C2N (C) Representing the discrimination result of the flat scan discriminator on the flat scan CT image output by the flat scan CT generator; d (D) C (G N2C (N)) represents the result of the enhancement discriminator discriminating the enhancement CT image outputted from the enhancement CT generator.
Based on the further improvement of the technical scheme, the enhanced CT generator and the swept CT generator have the same structure and comprise a preprocessing module, an encoder, a bottleneck layer, a decoder and a post-processing module;
the preprocessing module is used for preprocessing an input image;
the encoder is used for gradually extracting shallow layer characteristics of the preprocessed image;
the bottleneck layer is used for extracting deep features from shallow features output by the encoder and outputting the deep features to the decoder;
the decoder is used for decoding according to the deep layer characteristics and the shallow layer characteristics of the corresponding layer of the encoder;
and the post-processing module is used for generating CT images and organ contour segmentation images according to the decoding characteristics.
In another aspect, an embodiment of the present application provides an enhanced CT image generation system based on multi-task learning, including the following modules:
the training sample set construction module is used for acquiring a plain scan CT image and an enhanced CT image; respectively carrying out image segmentation on the plain CT image and the enhanced CT image to obtain corresponding organ contour segmentation images; constructing a training sample set according to the plain CT image, the enhanced CT image and the corresponding organ contour segmentation image;
the model training module is used for constructing a multi-task circulation generation countermeasure network model, and training the multi-task circulation generation countermeasure network model based on the training sample set to obtain a trained multi-task circulation generation countermeasure network model;
the image generation module is used for acquiring the plain scan CT image to be converted, generating a corresponding enhanced CT image based on the trained multi-task circulation generation countermeasure network model.
Compared with the prior art, the method has the advantages that the training sample set is constructed by collecting paired flat scanning CT images, corresponding enhanced CT images and corresponding organ segmentation images, and the countermeasure network model is generated by training multitasking circulation, so that the corresponding enhanced CT images can be obtained by directly converting the flat scanning CT images by using the trained model, contrast agent enhancement imaging is not needed, and the enhanced CT images are quickly obtained; and since no contrast agent is used, adverse effects on the patient's treatment (e.g., allergic reactions and nephrotoxicity) are avoided, as well as reducing the patient's radiation exposure; through training the multi-task circulation generation network, the image generation and segmentation of the image mode conversion are realized, the multi-task is complemented, the image conversion and segmentation are carried out more efficiently, the generated CECT image can provide contrast similar to the actual CECT image, the accuracy of tumor detection and segmentation is improved, and a more reliable basis is provided for planning and evaluation of radiotherapy. In addition, the trained multitask cyclic generation countermeasure network model not only can synthesize CECT images, but also can simultaneously carry out image segmentation. In addition, by utilizing the synthesized CECT image and combining the existing advanced image segmentation algorithm, the automatic outline sketching of the tumor target area and the organs at risk can be realized, so that the efficiency and the accuracy of radiotherapy are greatly improved, the workload of a radiotherapeutic operator is reduced, and the risk of human errors is reduced.
In the application, the technical schemes can be mutually combined to realize more preferable combination schemes. Additional features and advantages of the application will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by practice of the application. The objectives and other advantages of the application may be realized and attained by the structure particularly pointed out in the written description and drawings.
Drawings
The drawings are only for purposes of illustrating particular embodiments and are not to be construed as limiting the application, like reference numerals being used to designate like parts throughout the drawings;
FIG. 1 is a flow chart of an enhanced CT image generation method based on multi-task learning according to an embodiment of the present application;
FIG. 2 is a schematic diagram of a generator according to an embodiment of the present application;
FIG. 3 is a schematic diagram of the results of different models on an internal dataset in an embodiment of the present application;
FIG. 4 is a graphical representation of the results of different models on an HCC-TACE-Seg dataset in an embodiment of the present application;
FIG. 5 is a graph showing the results of different models on a KiTS dataset in accordance with an embodiment of the present application;
FIG. 6 is a schematic diagram of a visual result of an enhanced CT image generated by different models on an organ segmentation task according to an embodiment of the present application;
FIG. 7 is a block diagram of an enhanced CT image generation system based on multitasking learning in accordance with an embodiment of the present application.
Detailed Description
The following detailed description of preferred embodiments of the application is made in connection with the accompanying drawings, which form a part hereof, and together with the description of the embodiments of the application, are used to explain the principles of the application and are not intended to limit the scope of the application.
In one embodiment of the present application, an enhanced CT image generation method based on multi-task learning is disclosed, as shown in FIG. 1, comprising the following steps:
s1, acquiring a plain scan CT image and an enhanced CT image; respectively carrying out image segmentation on the plain CT image and the enhanced CT image to obtain corresponding organ contour segmentation images; constructing a training sample set according to the plain CT image, the enhanced CT image and the corresponding organ contour segmentation image;
s2, constructing a multi-task circulation generation countermeasure network model, and training the multi-task circulation generation countermeasure network model based on the training sample set to obtain a trained multi-task circulation generation countermeasure network model;
s3, acquiring a plain scan CT image to be converted, generating a corresponding enhanced CT image based on the trained multi-task circulation generation countermeasure network model.
The multitasking includes an image conversion task and an image segmentation task.
According to the application, a training sample set is constructed by collecting paired flat-scan CT images, corresponding enhanced CT images and corresponding organ segmentation images, and an countermeasure network model is generated by training multitasking circulation, so that the flat-scan CT images are directly converted by using the trained model to obtain the corresponding enhanced CT images, contrast agent enhancement imaging is not needed, and the enhanced CT images are rapidly obtained; and since no contrast agent is used, adverse effects on the patient's treatment (e.g., allergic reactions and nephrotoxicity) are avoided, as well as reducing the patient's radiation exposure; through training the multi-task circulation generation network, the image generation and segmentation of the image mode conversion are realized, the multi-task is complemented, the image conversion and segmentation are carried out more efficiently, the generated CECT image can provide contrast similar to the actual CECT image, the accuracy of tumor detection and segmentation is improved, and a more reliable basis is provided for planning and evaluation of radiotherapy. In addition, the trained multitask cyclic generation countermeasure network model not only can synthesize CECT images, but also can simultaneously carry out image segmentation. In addition, by utilizing the synthesized CECT image and combining the existing advanced image segmentation algorithm, the automatic outline sketching of the tumor target area and the organs at risk can be realized, so that the efficiency and the accuracy of radiotherapy are greatly improved, the workload of a radiotherapeutic operator is reduced, and the risk of human errors is reduced.
In practice, the test set can be used for calculating indexes to judge the advantages and disadvantages of the multi-task circulation generation countermeasure network model.
Note that the acquired pan-scan CT image and the enhanced CT image are paired. After the plain CT image and the corresponding enhanced CT image are acquired, the CT image can be preprocessed so as to facilitate the subsequent more accurate image conversion and segmentation. For example, the segmentation model is used to obtain the skin segmentation labels, remove the bed board in the CT image, and set the window width and the window level of the CT image to be the full window (for example, the window width is 2000, the window level is 0) so as to eliminate the interference of the abnormal voxel value on the subsequent training.
The organ segmentation map of the plain CT image and the enhanced CT image can be segmented through the existing CT image segmentation model, and the outline of the target organ is marked, namely the whole organ is marked as 1, and the other organ is marked as 0. In practice, the segmentation model may employ an RTP-Net lightweight automatic segmentation network.
The paired pan and enhancement CT images and the corresponding segmented images are taken as one sample to construct a training sample set.
Specifically, the constructed multi-task cyclic generation countermeasure network model comprises an enhanced CT generator, a horizontal scanning CT generator, an enhanced CT discriminator and a horizontal scanning CT discriminator;
the enhanced CT generator is used for converting the input flat scan CT image into an enhanced CT image and generating an enhanced organ contour segmentation image according to the input flat scan CT image; the enhanced CT discriminator is used for judging the authenticity of the enhanced CT image obtained by conversion;
the flat scan CT generator is used for converting the input enhanced CT image into a flat scan CT image and generating a flat scan organ contour segmentation image according to the input enhanced CT image; the flat scanning CT discriminator is used for judging the authenticity of the flat scanning CT image obtained through conversion.
In practice, the network structure of the enhanced CT generator and the flat CT generator are the same. The generator comprises a preprocessing module, an encoder, a bottleneck layer, a decoder and a post-processing module;
the preprocessing module is used for preprocessing an input image;
the encoder is used for gradually extracting shallow layer characteristics of the preprocessed image;
the bottleneck layer is used for extracting deep features from shallow features output by the encoder and outputting the deep features to the decoder;
the decoder is used for decoding according to the deep layer characteristics and the shallow layer characteristics of the corresponding layer of the encoder;
and the post-processing module is used for generating CT images and organ contour segmentation images according to the decoding characteristics.
In implementation, as shown in fig. 2, a preprocessing module (Pre Block) is first used to perform feature extraction preprocessing on an input image, then shallow feature extraction is gradually performed through a plurality of downsampling layers and convolution modules of an encoder, and the size of a feature map is gradually reduced. The preprocessing module comprises a 2D convolution layer and a LeakyReLU activation layer. Each downsampling layer comprises a 2D convolution layer. Each convolution module contains two example normalization layers, two stride 1 convolution layers, and two leak ReLU activation functions.
In contrast, the decoder includes a plurality of upsampling layers and convolution modules, and the convolution module structure of the decoder is identical to the convolution module structure in the encoder, so that the features can be decoded step by step and the size of the feature map can be increased step by step. Deconvolution may be employed as the upsampling layer. After passing through the up-sampling layer and the convolution module, the decoder outputs two channels through a post-processing module (PostBlock), one channel generates CT images, and the other channel performs organ contour segmentation. The network structure of the post-processing module is the same as that of the pre-processing module. By adding an image segmentation task, constraints are placed on the segmentation contours and the region of interest, thereby better optimizing the parameters of the network during back propagation.
To improve CT image generation quality, an Efficient Transducer Block (ETB) connection is designed to transfer shallow features of the input CT domain to the target CT domain. Deep feature extraction from the features output by the encoder is performed using ETB in the bottleneck layer between the encoder and the decoder. As shown in fig. 2, the bottleneck layer includes 12 Efficient Transformers (ETBs) for extracting deep features from the input feature map. In implementation, the ETB adopts an efficient multi-head attention mechanism to replace an original multi-head self-attention mechanism, so that the efficient utilization of GPU resources is realized.
In order to avoid overfitting and enhance deep feature learning, as shown in fig. 2, layer normalization layers are disposed before and after the high-efficiency multi-head attention module of the high-efficiency transducer block, and layer normalization (LayerNorm) layers are disposed before and after the feedforward layer of the high-efficiency transducer block.
In implementation, the shallow layer features and the deep layer features on the corresponding layers of the corresponding encoder and the decoder are fused through jump connection, so that the decoder can acquire more high-resolution information during upsampling, further the detail information in the original image is recovered more perfectly, and the image conversion and segmentation precision is improved.
In order to further improve the accuracy of the feature image conversion and segmentation, ETB is also used in jump connection for further feature extraction from shallow features, and the extracted features are merged and fused with feature images obtained by a decoder according to channel number stacking.
The generator combines the advantages of the convolutional neural network and the transducer network, integrates the efficient attention module, remarkably improves the network performance, and can generate high-quality CT images.
In practice, the arbiter employs PatchGAN.
After the multi-task circulation generating countermeasure network is built, training the network model according to the training sample set to obtain the trained multi-task circulation generating countermeasure network model. In the training process, back propagation is carried out according to the loss of the model, and the model parameters are updated.
In practice, to improve image generation quality, the multitasking loop generation places consistency loss constraints against the network model at the image level, the segmentation contour level, and the region of interest level.
Specifically, the following formula is used to calculate the total loss of the multi-tasking loop generation countermeasure network model：
，
wherein ,GN2C Representation enhanced CT generator, G C2N Represents a flat scan CT generator, D C Representation enhanced CT discriminator, D N Representing the horizontal scanning CT discriminator,indicating the loss of antagonism of the arbiter,representing a loss of consistency constraint at the image level, +.>Segmentation loss representing the segmentation contour level, +.>Region of interest perceived loss, represented by a region of interest hierarchy, λ1, λ2, λ3, and λ4 represent weighting coefficients.
Consistency constraint loss of image layers, including global loss and region of interest loss. Specifically, the following formula is used to calculate the consistency constraint loss of the image hierarchy:
，
wherein N represents an input plain CT image, C represents an input enhanced CT image, G C2N (G N2C (N)) represents a flat scan CT image output after the enhanced CT image obtained by the enhanced CT generator is input to the flat scan CT generator; g N2C (G C2N (C) A) represents an enhanced CT image output after the flat scan CT image obtained by the flat scan CT generator is input into the enhanced CT generator, S C Corresponding organ contour segmented image representing an input enhanced CT image, S N Corresponding organ contour segmented image, ║. ║, representing an input plain CT image 1 Representation matrixIs a function of the one of the norms of (1),representing global loop consistency constraint loss, +.>Representing a region of interest cyclic uniformity constraint loss.
Specifically, the challenge loss of the arbiter is calculated according to the following formula:
，
wherein E [. Cndot.]Indicating desire, D N (G C2N (C) Representing the discrimination result of the flat scan discriminator on the flat scan CT image output by the flat scan CT generator; d (D) C (G N2C (N)) represents the result of the enhancement discriminator discriminating the enhancement CT image outputted from the enhancement CT generator.
Specifically, the segmentation loss is calculated according to the following formula:
，
wherein ,representing an enhanced organ contour segmentation image output by an enhanced CT generator, < >>Representing a flat scan organ contour segmentation image output by a flat scan CT generator, < >>Representing the organ contour segmentation image output after inputting the enhanced CT image obtained by the enhanced CT generator into the plain CT generator,/for the enhanced CT image>Representing the segmented image of the enhanced organ contour output after the flat scan CT image obtained by the flat scan CT generator is input into the enhanced CT generator,representing the Dice loss.
Wherein the Dice loss can be based onAnd (5) calculating to obtain the product. Wherein->Representing the intersection of a and B, |a| and |b| represent the number of elements thereof.
In practice, the region of interest perceived loss is calculated according to the following formula:
，
wherein ,representing the perceived loss. The perceptual loss is typically a distance measure in high dimensional space using a pre-trained neural network (e.g., VGG network) as a feature extractor. And the generalization of the model is improved by restraining the perception loss of the region of interest in a high-dimensional space.
In practice, byCalculating the perceived loss of the two images x and x', ∈>Feature map representing output of ith layer of pre-trained neural network, N P Representing the number of feature extraction layers of a pre-trained neural network ║. ║ 1 Representing a norm of the matrix.
The loss of the multi-task cyclic generation countermeasure network model not only comprises the consistency constraint loss of the image level and the countermeasure loss of the discriminator, but also comprises the contour segmentation loss and the interested region perception loss in the segmentation task, so that the image generation task is assisted by the segmentation task, and the image generation quality is improved. The application combines a multi-task learning strategy to restrict the whole at the image level, thereby ensuring the generation quality of partial areas of the CT image; learning the target ROI region on the segmentation contour level to ensure that the model can capture contour information of the enhancement region; and the interested region layer level carries out additional supervision constraint on the ROI region to ensure the authenticity and reliability of the generated region.
In implementation, the countermeasure network model can be generated according to the corresponding different-period multi-task cyclic generation of the CT images of different enhancement periods (such as arterial enhancement CT and venous enhancement CT), so that the generation of the enhanced CT images of different periods is realized (for example, the countermeasure network model is generated by the multi-task cyclic generation of the flat-scan rotational pulse enhancement CT, and the countermeasure network model is generated by the multi-task cyclic generation of the flat-scan rotational venous enhancement CT).
In practice, to illustrate the effects of the present application, the multitasking loop generation of the present application was compared against the common network model (U-Net, transUNet and PTNet) on the internal dataset and two public datasets (HCC-TACE-Seg and KiTS). In addition, segmentation experiments are performed on the generated enhanced CT images to evaluate the performance of the generated enhanced CT images in organ segmentation.
In order to evaluate the quality of the generated CT image and ensure the similarity with the real CT image, two evaluation indexes commonly used in the generation countermeasure network, namely FID and LPIPS, are used for evaluating the quality of the generated enhanced CT image, and the FID and the LPIPS are two evaluation indexes for measuring the distance between the pseudo image and the real image in a high-dimensional feature space, wherein the lower the distance is, the better the distance is. In addition, the organ segmentation performance of the generated enhanced CT image was evaluated using 4 indices commonly used in the medical image segmentation field, namely, a Dice Similarity Coefficient (DSC), a 95% hausdorff distance (HD 95), an Average Surface Distance (ASD), and a Jaccard Coefficient (JC), respectively, the DSC and JC being region-based indices, and the HD95 and ASD being boundary-based indices, thereby providing comprehensive and accurate evaluation.
Fig. 3, 4 and 5 show the results of comparing arterial phase enhanced CT images and venous phase enhanced CT generated by different network models on different data sets. Fig. 3 shows the results of the different models on the internal dataset, fig. 4 shows the results of the different models on the HCC-TACE-Seg dataset, and fig. 5 shows the results of the different models on the kit dataset. And table 1 shows the corresponding index. By comparing the differences between the generated enhancement map and the true enhancement map (GT) of these network models, we can easily assess the relative performance of each model. In the arterial phase, the main areas of enhancement are the iliac arteries and veins. MT-CTGAN enhances these small vessels while ensuring accuracy of the enhanced regions, and fig. 3 shows that our network model is superior to other models. During the venous phase we evaluate the enhancement of the kidney and aortic regions, the challenge at this stage is the difficulty in restoring the structures and vessels inside the kidney and at the aortic boundary. MT-CTGAN performs well in accurately locating aortic boundaries and recovering detailed texture information of the kidneys. In the HCC-TACE-Seg dataset, arterial, renal and spleen and venous liver are major challenges. For the results of the KiTS dataset, the main challenge is not only the enhancement of the ROI, but also the proximity of the tumor. Fig. 5 shows two types of kidney tumors and their manifestations in the enhancement map. It can be seen from the figure that the model we propose is superior to other models in generating an enhancement map that more closely approximates the true enhanced CT image. Table 1 shows quantitative measures of the results, which indicate that the MT-CTGAN generated enhanced CT image is very similar to the real enhanced CT image. Thus, both quantitative and visual results demonstrate the effectiveness of the proposed MT-CTGAN in handling domain shifts and achieving generalization.
Table 1 quantitative results of enhanced CT images generated by different network models
The performance metrics of CT images generated by different models in the organ segmentation task are given in Table 2, and the comparison of the visual results of the CT images is given in FIG. 6. In table 2, NE represents pan-scan CT; AP represents arterial phase enhanced CT; VP represents portal enhancement CT; S-AP represents pseudo-arterial phase enhancement CT generated by a network model; S-VP represents pseudo-venous phase enhancement CT generated by the network model, and p values in bold in a vs. c column and a vs. e column indicate that the segmentation performance of the plain scan CT is significantly different from that of the enhancement CT generated by the network. The bold p-values in columns b vs. c and d vs. e indicate that there is no significant difference in segmentation performance between the true enhanced CT and the generated enhanced CT. For the results of the HCC-TACE-Seg dataset, venous phase enhanced CT (whether real or network generated) is superior to other phases. This is because the contrast agent typically enters the venous phase a few minutes after injection of the contrast agent, which allows more time for the contrast agent to circulate and accumulate in the HCC lesion, further enhancing the contrast agent's visibility. Whereas in the KiTS dataset, the resulting arterial phase enhancement CT shown in FIG. 6 (b) can enhance the exact boundaries of the kidneys, assisting the segmentation model to detect tumors more accurately. The enhanced CT image generated by our proposed model yields a more accurate tumor boundary than relying on the flat scan CT image alone. These remarkable results indicate that our proposed model can assist in the segmentation of abdominal tumors by generating enhanced CT images.
Table 2 results of enhanced CT images generated by different network models on organ segmentation tasks
Furthermore, to illustrate the effectiveness of the proposed loss functions of the present application, the final performance was evaluated by deleting one of these loss functions, while retaining the other loss functions, with the results shown in table 3. Table 3 shows that each loss function contributes significantly to the overall performance of the model. These loss functions based on the MT-CTGAN split branch design further confirm the effectiveness of our proposed multitask learning mechanism. These findings provide valuable insight into the design of loss functions in image generation tasks and multitasking learning strategies.
TABLE 3 Performance results for different loss functions
In one embodiment of the present application, an enhanced CT image generation system based on multi-task learning is disclosed, as shown in FIG. 7, comprising the following modules:
the training sample set construction module is used for acquiring a plain scan CT image and an enhanced CT image; respectively carrying out image segmentation on the plain CT image and the enhanced CT image to obtain corresponding organ contour segmentation images; constructing a training sample set according to the plain CT image, the enhanced CT image and the corresponding organ contour segmentation image;
the model training module is used for constructing a multi-task circulation generation countermeasure network model, and training the multi-task circulation generation countermeasure network model based on the training sample set to obtain a trained multi-task circulation generation countermeasure network model;
the image generation module is used for acquiring the plain scan CT image to be converted, generating a corresponding enhanced CT image based on the trained multi-task circulation generation countermeasure network model.
The method embodiment and the system embodiment are based on the same principle, and the related parts can be mutually referred to and can achieve the same technical effect. The specific implementation process refers to the foregoing embodiment, and will not be described herein.
Those skilled in the art will appreciate that all or part of the flow of the methods of the embodiments described above may be accomplished by way of a computer program to instruct associated hardware, where the program may be stored on a computer readable storage medium. Wherein the computer readable storage medium is a magnetic disk, an optical disk, a read-only memory or a random access memory, etc.
The present application is not limited to the above-mentioned embodiments, and any changes or substitutions that can be easily understood by those skilled in the art within the technical scope of the present application are intended to be included in the scope of the present application.

Claims (10)

1. The enhanced CT image generation method based on the multi-task learning is characterized by comprising the following steps of:
acquiring a plain scan CT image and an enhanced CT image; respectively carrying out image segmentation on the plain CT image and the enhanced CT image to obtain corresponding organ contour segmentation images; constructing a training sample set according to the plain CT image, the enhanced CT image and the corresponding organ contour segmentation image;
constructing a multi-task circulation generation countermeasure network model, and training the multi-task circulation generation countermeasure network model based on the training sample set to obtain a trained multi-task circulation generation countermeasure network model;
and acquiring a plain scan CT image to be converted, generating a corresponding enhanced CT image based on the trained multi-task circulation generation countermeasure network model.
2. The enhanced CT image generation method for multi-task learning of claim 1, wherein the multi-task cyclic generation countermeasure network model includes an enhanced CT generator, a flat scan CT generator, an enhanced CT discriminator, and a flat scan CT discriminator;
the enhanced CT generator is used for converting the input flat scan CT image into an enhanced CT image and generating an enhanced organ contour segmentation image according to the input flat scan CT image; the enhanced CT discriminator is used for judging the authenticity of the enhanced CT image obtained by conversion;
the flat scan CT generator is used for converting the input enhanced CT image into a flat scan CT image and generating a flat scan organ contour segmentation image according to the input enhanced CT image; the flat scanning CT discriminator is used for judging the authenticity of the flat scanning CT image obtained through conversion.
3. The enhanced CT image generation method for multi-tasking according to claim 2 wherein the multi-tasking cyclic generation places a consistency loss constraint on the countermeasure network model at the image level, the segmentation contour level and the region of interest level.
4. The enhanced CT image generation method for multi-task learning of claim 3 wherein the total loss of the multi-task loop generation countermeasure network model is calculated using the formula：
，
wherein ,GN2C Representation enhanced CT generator, G C2N Represents a flat scan CT generator, D C Representation enhanced CT discriminator, D N Representing the horizontal scanning CT discriminator,indicating loss of antagonism of the arbiter, ++>Representing a loss of consistency constraint at the image level, +.>Representing the segmentation loss of the segmentation contour level,region of interest perceived loss, represented by a region of interest hierarchy, λ1, λ2, λ3, and λ4 represent weighting coefficients.
5. The enhanced CT image generation method for multi-task learning of claim 4 wherein the consistency constraint loss at the image level is calculated using the formula:
，
wherein N represents an input plain CT image, C represents an input enhanced CT image, G C2N (G N2C (N)) represents a flat scan CT image output after the enhanced CT image obtained by the enhanced CT generator is input to the flat scan CT generator; g N2C (G C2N (C) A) represents an enhanced CT image output after the flat scan CT image obtained by the flat scan CT generator is input into the enhanced CT generator, S C Corresponding organ contour segmented image representing an input enhanced CT image, S N Representing the inputCorresponding organ contour segmented image of the incoming plain CT image ║. ║ 1 Representing a norm of the matrix,representing global loop consistency constraint loss, +.>Representing a region of interest cyclic uniformity constraint loss.
6. The enhanced CT image generation method for multitasking learning of claim 4, wherein the segmentation loss is calculated according to the following formula:
，
wherein ,representing an enhanced organ contour segmentation image output by an enhanced CT generator, < >>Representing a flat scan organ contour segmentation image output by a flat scan CT generator, < >>Representing the organ contour segmentation image output after inputting the enhanced CT image obtained by the enhanced CT generator into the plain CT generator,/for the enhanced CT image>Representing the segmented image of the enhanced organ contour output after the flat scan CT image obtained by the flat scan CT generator is input into the enhanced CT generator,representing the Dice loss.
7. The enhanced CT image generation method for multitasking learning of claim 4, wherein the region of interest perceived loss is calculated according to the following formula:
，
wherein ,representing a flat scan organ contour segmentation image output by a flat scan CT generator, < >>Representing a flat scan organ contour segmentation image output after inputting the enhanced CT image obtained by the enhanced CT generator into the flat scan CT generator, < >>Representing a perceived loss;
the perceptual loss is calculated using the following formula:
the perceived loss of the two images x and x' is calculated,feature map representing output of ith layer of pre-trained neural network, N P Representing the number of feature extraction layers of a pre-trained neural network ║. ║ 1 Representing a norm of the matrix.
8. The enhanced CT image generation method for multitasking learning of claim 4, wherein the contrast loss of the discriminator is calculated according to the following formula:
，
wherein E [. Cndot.]Representing expectations，D N (G C2N (C) Representing the discrimination result of the flat scan discriminator on the flat scan CT image output by the flat scan CT generator; d (D) C (G N2C (N)) represents the result of the enhancement discriminator discriminating the enhancement CT image outputted from the enhancement CT generator.
9. The method for generating an enhanced CT image for multi-task learning according to claim 2, wherein the enhanced CT generator and the flat CT generator have the same structure, and each include a preprocessing module, an encoder, a bottleneck layer, a decoder, and a post-processing module;
the preprocessing module is used for preprocessing an input image;
the encoder is used for gradually extracting shallow layer characteristics of the preprocessed image;
the bottleneck layer is used for extracting deep features from shallow features output by the encoder and outputting the deep features to the decoder;
the decoder is used for decoding according to the deep layer characteristics and the shallow layer characteristics of the corresponding layer of the encoder;
and the post-processing module is used for generating CT images and organ contour segmentation images according to the decoding characteristics.
10. An enhanced CT image generation system based on multitasking learning, comprising the following modules:
the training sample set construction module is used for acquiring a plain scan CT image and an enhanced CT image; respectively carrying out image segmentation on the plain CT image and the enhanced CT image to obtain corresponding organ contour segmentation images; constructing a training sample set according to the plain CT image, the enhanced CT image and the corresponding organ contour segmentation image;
the model training module is used for constructing a multi-task circulation generation countermeasure network model, and training the multi-task circulation generation countermeasure network model based on the training sample set to obtain a trained multi-task circulation generation countermeasure network model;
the image generation module is used for acquiring the plain scan CT image to be converted, generating a corresponding enhanced CT image based on the trained multi-task circulation generation countermeasure network model.
CN202310896717.8A 2023-07-21 2023-07-21 Enhanced CT image generation method and system based on multitask learning Active CN116630463B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202310896717.8A CN116630463B (en) 2023-07-21 2023-07-21 Enhanced CT image generation method and system based on multitask learning

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202310896717.8A CN116630463B (en) 2023-07-21 2023-07-21 Enhanced CT image generation method and system based on multitask learning

Publications (2)

Publication Number Publication Date
CN116630463A true CN116630463A (en) 2023-08-22
CN116630463B CN116630463B (en) 2023-10-13

Family

ID=87636831

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202310896717.8A Active CN116630463B (en) 2023-07-21 2023-07-21 Enhanced CT image generation method and system based on multitask learning

Country Status (1)

Country Link
CN (1) CN116630463B (en)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116977466A (en) * 2023-07-21 2023-10-31 北京大学第三医院（北京大学第三临床医学院） Training method for enhancing CT image generation model and storage medium
CN117437514A (en) * 2023-12-22 2024-01-23 南昌航空大学 Colposcope image mode conversion method based on CycleGan

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108921851A (en) * 2018-06-06 2018-11-30 深圳市未来媒体技术研究院 A kind of medicine CT image dividing method based on 3D confrontation network
CN113222852A (en) * 2021-05-26 2021-08-06 深圳高性能医疗器械国家研究院有限公司 Reconstruction method for enhancing CT image
CN113763390A (en) * 2021-08-31 2021-12-07 山东师范大学 Brain tumor image segmentation and enhancement system based on multi-task generation countermeasure network
CN113989178A (en) * 2020-07-08 2022-01-28 北京大学 CTA image blood vessel segmentation and flat scan image prediction based on self-supervision learning
US20220208355A1 (en) * 2020-12-30 2022-06-30 London Health Sciences Centre Research Inc. Contrast-agent-free medical diagnostic imaging
CN115601352A (en) * 2022-11-04 2023-01-13 河北工业大学（Cn） Medical image segmentation method based on multi-mode self-supervision
CN116363248A (en) * 2023-03-31 2023-06-30 徐州鑫达房地产土地评估有限公司 Method, system, equipment and medium for synthesizing CT image by single plane X-Ray image

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108921851A (en) * 2018-06-06 2018-11-30 深圳市未来媒体技术研究院 A kind of medicine CT image dividing method based on 3D confrontation network
CN113989178A (en) * 2020-07-08 2022-01-28 北京大学 CTA image blood vessel segmentation and flat scan image prediction based on self-supervision learning
US20220208355A1 (en) * 2020-12-30 2022-06-30 London Health Sciences Centre Research Inc. Contrast-agent-free medical diagnostic imaging
CN113222852A (en) * 2021-05-26 2021-08-06 深圳高性能医疗器械国家研究院有限公司 Reconstruction method for enhancing CT image
CN113763390A (en) * 2021-08-31 2021-12-07 山东师范大学 Brain tumor image segmentation and enhancement system based on multi-task generation countermeasure network
CN115601352A (en) * 2022-11-04 2023-01-13 河北工业大学（Cn） Medical image segmentation method based on multi-mode self-supervision
CN116363248A (en) * 2023-03-31 2023-06-30 徐州鑫达房地产土地评估有限公司 Method, system, equipment and medium for synthesizing CT image by single plane X-Ray image

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
李伟: "基于深度学习的主动脉CT增强扫描图像合成研究", 《中国优秀硕士学位论文全文数据库 (医药卫生科技辑)》, pages 21 - 50 *

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116977466A (en) * 2023-07-21 2023-10-31 北京大学第三医院（北京大学第三临床医学院） Training method for enhancing CT image generation model and storage medium
CN117437514A (en) * 2023-12-22 2024-01-23 南昌航空大学 Colposcope image mode conversion method based on CycleGan
CN117437514B (en) * 2023-12-22 2024-04-05 南昌航空大学 Colposcope image mode conversion method based on CycleGan

Also Published As

Publication number Publication date
CN116630463B (en) 2023-10-13

Similar Documents

Publication Publication Date Title
US10909681B2 (en) Automated selection of an optimal image from a series of images
CN116630463B (en) Enhanced CT image generation method and system based on multitask learning
Liu et al. Multimodal MR image synthesis using gradient prior and adversarial learning
JP2022544229A (en) 3D Object Segmentation of Localized Medical Images Using Object Detection
Iqbal et al. Generative adversarial networks and its applications in the biomedical image segmentation: a comprehensive survey
CN107451983A (en) The three-dimensional fusion method and system of CT images
Li et al. DenseX-net: an end-to-end model for lymphoma segmentation in whole-body PET/CT images
JP2023540910A (en) Connected Machine Learning Model with Collaborative Training for Lesion Detection
Song et al. Bridging the gap between 2D and 3D contexts in CT volume for liver and tumor segmentation
CN115398555A (en) Generating a radiographic image
Al Khalil et al. On the usability of synthetic data for improving the robustness of deep learning-based segmentation of cardiac magnetic resonance images
CN112036506A (en) Image recognition method and related device and equipment
Singh et al. Medical image generation using generative adversarial networks
CN113935976A (en) Method and system for automatically segmenting blood vessels in internal organs by enhancing CT (computed tomography) image
Feng et al. MRI generated from CT for acute ischemic stroke combining radiomics and generative adversarial networks
CN116612174A (en) Three-dimensional reconstruction method and system for soft tissue and computer storage medium
Fan et al. TR-Gan: multi-session future MRI prediction with temporal recurrent generative adversarial Network
CN116563402A (en) Cross-modal MRI-CT image synthesis method, system, equipment and medium
Thaler et al. Efficient multi-organ segmentation using spatialconfiguration-net with low GPU memory requirements
CN110852993A (en) Imaging method and device under action of contrast agent
Rezaei Generative adversarial network for cardiovascular imaging
CN116977466A (en) Training method for enhancing CT image generation model and storage medium
EP2174292A1 (en) A method, apparatus, computer-readable medium and use for pharmacokinetic modeling
Soh et al. HUT: Hybrid UNet transformer for brain lesion and tumour segmentation
Peng et al. 2D brain MRI image synthesis based on lightweight denoising diffusion probabilistic model

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant