CN116894783A - Metal artifact removal method based on adversarial generative network model with time-varying constraints - Google Patents

Abstract

The application discloses a metal artifact removal method for an countermeasure generation network model based on time-varying constraint, which constructs a GAN network demetallization artifact Model (MARGANVAC) with time-varying constraint, and the model carries out self-adaptive fidelity constraint on each part of a full graph by introducing time-varying constraint items, so that a generator is more effectively trained to generate images with better detail fidelity and metal artifact removal effect. Compared with the current mainstream model, the method has better artifact removal effect and wider application scene.

Description

Metal artifact removal method based on adversarial generative network model with time-varying constraints

Technical field

The invention relates to a metal artifact removal method, specifically a metal artifact removal method based on a time-varying constraint adversarial generation network model, and belongs to the technical field of improving CT image quality.

Background technique

The presence of metal implants will cause severe metal artifacts in CT images, seriously reducing image quality. Over the past few decades, many methods have been proposed to reduce metal artifacts. Although these traditional methods have produced certain results in metal artifact removal, they are still far from the actual required results. In recent years, with the rise of deep learning, deep learning models have gradually been used in research on metal artifact removal, and have produced good results. In references [1, 2], Park et al. used U-net to learn the mapping from the geometric shape of the metal track area in the projection map to the corresponding beam hardening factor to correct the inaccurate projection data in the projection map. The CNNMAR method combines the uncorrected image, LI image and BHC image into a three-channel image and inputs it into the convolutional neural network to generate a prior image, and then uses its projection results to guide the interpolation process to produce a picture with artifacts removed. Based on the idea of conditional generative adversarial network (cGAN), Wang et al. proposed cGANMAR in Reference [3] to learn the mapping from images with artifacts to images without artifacts, and use PatchGAN as the discriminator. The above methods are all supervised methods and require good paired data as training sets. Many scholars are committed to the research of unsupervised learning, liberating the need for paired data for metal artifact removal problems. Based on the idea of CycleGAN, Lee et al. proposed Attention-guidedβ-CycleGAN, see reference [4], which uses the attention mechanism to pay attention to the unique characteristics of metal artifacts in the spatial domain and channel domain, and conducts training in an unsupervised manner. , the results are very robust. In reference [5], Liao et al. creatively proposed the ADN network. ADN uses the concept of latent space, using different encoders to encode artifact-containing images into a content space containing only image information features, and uses another encoder to The encoder encodes images containing artifacts into an artifact space containing only artifacts to achieve decoupling of artifacts and tissue details. Unsupervised methods eliminate the need for paired data, but do not cope well with the complex artifacts and tissue details produced in clinical conditions. Although the current mainstream supervised methods are more superior in performance than unsupervised methods, they still cannot meet the needs of practical applications in terms of metal artifact removal and image detail fidelity.

references:

[1]Park H S, Chung Y E, Lee S M, et al.Sinogram-consistency learning inCT for metal artifact reduction[J].arXiv preprint arXiv:00607,2017,1.

[2] Park H S, Lee S M, Kim H P, et al. CT sinogram-consistency learning for metal-induced beam hardening correction [J]. Medical physics, 2018, 45(12): 5376-84.

[3]Wang J, Zhao Y, Noble J H, et al. Conditional generative adversarial networks for metal artifact reduction in CT images of the ear[C]. International Conference on Medical Image Computing and Computer-AssistedIntervention.2018:3-11.

[4]Lee J,Gu J,Ye J C.Unsupervised CT Metal Artifact Learning UsingAttention-Guidedβ-CycleGAN[J].IEEE Transactions on Medical Imaging, 2021,40(12):3932-44.

[5]Liao H, Lin W-A, Zhou S K, et al.ADN:artifact disentanglement network for unsupervised metal artifact reduction[J].IEEE Transactions on MedicalImaging,2019,39(3):634-43.

Contents of the invention

Purpose of the invention: In view of the problems and deficiencies existing in the existing technology, the present invention provides a metal artifact removal method based on a time-varying constraint adversarial generation network model. A GAN network metal artifact removal model with time-domain variable constraints (MARGANVAC) was constructed. This model introduces time-varying constraints to implement adaptive fidelity constraints on each part of the entire image, thereby more effectively training the generator to generate Images with better detail fidelity and metallic artifact removal. Compared with the current mainstream models, this method not only has better artifact removal effects, but also has a wider range of application scenarios.

Technical solution: A metal artifact removal method based on time-varying constraints of the adversarial generative network model; a GAN network metal artifact removal model with time-domain variable constraints (MARGANVAC) is constructed, which is based on the GAN network. Introducing time-varying constraints to implement adaptive fidelity constraints on each part of the entire graph;

The GAN network metal artifact removal model with time domain variable constraints includes three modules, namely the generator G, the discriminator D and the registration network R. After the CT image to be processed undergoes random affine changes, the input The generator G is output to the registration network and the discriminator D through the generator G; the registration network R is used for random sampling, and the neighborhood of the sampled pixels gradually decreases as the number of iterations increases; the registration network R Without manual intervention, parameters can be adjusted adaptively, allowing the generator to produce more realistic images.

Further, the training process of the GAN network metal artifact removal model with time domain variable constraints is as follows:

First, ^{perform a} random affine transformation on each input ^image image after and x _T ; when the image/> After passing the generator G, we get the image with artifacts removed/> Will/> and x _T input the registration network R; the physical meaning of the registration network can be used as a function/> To express, the first part of the function G (x ^a ; θ _G ) is the output result of the generator subnetwork, θ _G represents the parameters of the generator subnetwork, j is the pixel index, and the second part φ represents an abstract The sampling function is used to sample pixels within the neighborhood of pixel x _j . σ _t is a parameter used to control the size of the neighborhood. As the iteration proceeds during the training process, the parameter σ _t gradually converges to zero. At the beginning of training, the performance of the registration network is poor, so the output There will be a certain position deviation from the ground truth image x _t after affine transformation. These deviations are variable, that is, each pixel in each input image of each iteration has an independent and different deviation. This feature It can help simulate the random sampling process of function φ(x _j ,σ _t ). Moreover, as training progresses, the registration performance of the registration network will improve, which means that the parameter σ _t will decrease. If the entire model converges well, σ _t will drop to zero.

The trained GAN network metal artifact removal model with time-domain variable constraints is used to generate the image after removing metal artifacts from the CT image with metal artifacts.

Further, residual learning is introduced in the generator G; self-reconstruction is introduced as a constraint to regularize the generator; the generator contains an encoder and a decoder; the encoder is used to map image samples from the image domain to the latent space, and its latent space The features of content information and metal artifacts in space are separated; the decoder reconstructs the separated content information into artifact-free images; between the encoder and decoder, a deep layer consisting of 21 Inception-ResNet modules is introduced sub-network to improve the separation ability of artifact features and content information in the latent space.

Furthermore, in the training process of the GAN network metal artifact removal model with time-domain variable constraints, in order to better distinguish metal artifacts and content information, self-reconstruction is introduced as a constraint to regularize the generator. During the self-reconstruction process, the artifact-free image y is spliced into [y, y] and then input to the generator. Before splicing, random affine transformation is performed on the artifact-free image y to obtain the result A ₃ (y), and the splicing is [A ₃ (y), A ₃ (y)]. Correspondingly, in the self-reconstruction, the generator output Becomes: G([A ₃ (y), A ₃ (y)]; θ _G ); the reference image that is not affected by artifacts corresponding to the artifact-free image y is also y.

Further, for the image x ^a , the linear interpolation method (LI) is used to obtain the estimated value of the metal trajectory part in the projection image in the sinogram domain, and the LI corrected reconstructed image x ^[LI]a is obtained through the FBP or FDK method; The image x ^[LI]a corrected by the LI method and the image x ^a affected by the artifact are subjected to random affine transformation respectively to obtain the affine transformation results A ₁ (x ^[LI]a ) and A ₁ (x ^a ). A ₁ (x ^[LI]a ) and A ₁ (x ^a ) are connected in the channel dimension as the input of the generator G.

Further, residual learning is introduced in the generator G, and the image after removing metal artifacts is expressed as: [x ^a , x ^[LI]a ] represents the connection operation of x ^a and x ^[LI]a ; in self-reconstruction, the generator output becomes/> Then take/> The splicing of A ₂ (x) and A 2 (x) is used as the input of the registration network R. A ₂ (x) represents the random affine transformation of the reference image x. The output of the registration network R is a deformation vector field T _x ,/> θ _R is the registration network parameter. After obtaining the deformation vector field T _x , the input image/> Apply T _x to get the resampled image/> Follow the same steps as generating T _x to obtain the deformation vector field T _y ,/> A ₄ (y) represents the random affine transformation of the reference image y; and in the case of self-reconstruction, the corresponding resampled image/> Generator output/> and A ₂ (x) as the input of the discriminator D. Similarly, in the case of self-reconstruction, the generator output/> and A ₄ (y) as the input of the discriminator D.

Further, the GAN network metal artifact removal model with time domain variable constraints includes four loss functions, namely artifact correction loss. Self-reconstruction loss/> Fight against losses/> and smooth loss blocks used to constrain the deformation vector field/> The total loss is expressed as the weighted sum of these losses/> Each λ represents the weight coefficient of the corresponding loss.

correction loss The registration network R and the generator G are trained simultaneously, and the design correction loss is:

Using the L1 norm allows the generator to maintain more detail while reducing metallic artifacts, Express expectations,/> Indicates that x comes from the data set/> The definition domain of CT images affected by metal artifacts is/>

self-reconstruction loss The training goal is to make the generator learn to remove metal artifacts, and more constraints need to be introduced to make the generator retain more content information; that is, to minimize metal artifacts when there are metal and artifacts, and to minimize metal artifacts when there are no metal artifacts. Try to retain all image content information without shadowing.

in, Express expectations,/> Indicates that y comes from the data set/> The domain of an image that is not affected by artifacts is/>

against losses Adversarial learning forces the generator G to generate more realistic artifact-free images. To achieve this goal, the generator should have the ability to distinguish artifact information and content information in the latent space. To enhance this capability, two adversarial learning strategies are introduced. One strategy is to improve the ability of the latent space to identify metal artifacts, and the other strategy is to improve the ability of the latent space to preserve content information. The input data for the two strategies are images affected by metal artifacts and images without metal artifacts. These two strategies are performed simultaneously in adversarial learning, and their respective losses can be written as:

The base of log is 2, and D(·) represents the output of the discriminator D; so the total adversarial loss is:

Smooth loss When minimizing the combined loss of correction loss and self-reconstruction loss, the resulting deformation vector field may become unsmooth and physically unrealistic. In order to solve this problem, a diffusion regular operator in the gradient direction of the deformation vector field is introduced in the original registration network to constrain the deformation vector field and make it smooth. Because the generator has two outputs and/> There are two corresponding deformation vector field regularization terms. Therefore, the total smoothing loss can be expressed as:

in is the gradient of the deformation vector field T. In practice, the differences between pixels are used to approximate the gradient of each pixel in the deformation vector field.

A computer device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the adversarial generation network model based on time-varying constraints as described above. Metal artifact removal method.

A computer-readable storage medium stores a computer program for executing the metal artifact removal method based on the time-varying constraint adversarial generative network model as described above.

Beneficial effects: Compared with the existing technology, the present invention constructs a GAN network metal artifact removal model with time-domain variable constraints (MARGANVAC), which adaptively adapts to each part of the entire image by introducing time-varying constraint terms. Fidelity constraints, thereby more efficiently training the generator to produce images with better detail fidelity and metallic artifact removal. Compared with the current mainstream models, this method not only has better artifact removal effects, but also has a wider range of application scenarios.

Description of the drawings

Figure 1 is a network diagram of a conventional GAN model;

Figure 2 is a result display diagram of the MAR method based on conventional GAN architecture;

Figure 3 is a schematic diagram of the GAN network that introduces the registration network as the sampling function;

Figure 4 is a schematic diagram of affine transformation, in which: (a) is the original image, (b) is the image after affine transformation of (a), (c) is the difference image between (a) and (b);

Figure 5 is a schematic diagram of the overall architecture of the MARGANVAC model according to the embodiment of the present invention;

Figure 6 is a schematic diagram of the generator structure;

Figure 7 is a schematic diagram of the basic component modules of the generator;

Figure 8 is a schematic diagram of the discriminator structure;

Figure 9 is a schematic diagram of qualitative comparison of different methods on the deeplesion data set;

Figure 10 is a schematic diagram of the qualitative comparison of different methods on the Micro CT synthetic artifact data set;

Figure 11 is a schematic diagram of the results of qualitative comparison of different methods on a cone beam Micro CT data set containing real metal artifacts.

Detailed ways

The present invention will be further clarified below with reference to specific examples. It should be understood that these examples are only used to illustrate the present invention and are not intended to limit the scope of the present invention. After reading the present invention, those skilled in the art will be familiar with various equivalent forms of the present invention. All modifications fall within the scope defined by the appended claims of this application.

Assume that the definition domain of CT images affected by metal artifacts is The domain of an image that is not affected by artifacts is/> The goal of the metal artifact removal network is to find the mapping function f(x ^a )=x, where/> When there is a paired data set When , a supervised network is trained using the paired data set to learn f.

GAN networks have better fitting capabilities than common convolutional neural networks (CNN) and can generate images with more anatomical details. Therefore, the GAN model is a very good basic network model that can be used for MAR. The disadvantage of traditional GAN models is that the generated details may be inconsistent with ground truth. A conventional GAN network is shown in Figure 1. It consists of three parts. The first part is the generator subnetwork G(x ^a ; θ _G ), the second part is the discriminator subnetwork D(x ^a ||x; θ _D ), and the third part is based on L1, L2 norm or some specific network Fidelity loss of extracted high-level features. Generally speaking, it is very simple and convenient to design a powerful generator G(x ^a ; θ _G ) and a matching discriminator D (x ^a ||x; θ _D ) to remove metal artifacts well. , this is because the characteristics of metal artifacts are very different from the characteristics of image content information. Figure 2 shows the artifact-removed image obtained by the traditional GAN network-based MAR model. From this, you can see that specific texture features of metal artifacts, such as streaks and shadows, are eliminated and the generator produces an image that appears to be free of metal artifacts. In other words, a powerful generator successfully The x ^a image in the domain is converted to/> /> in domain image. However, it is obvious that the image generated by the generator In terms of content information, it is not consistent with the ground truth image x. It can be seen from this preliminary experiment that the generative adversarial network has been trained well enough to achieve conversion between images in different domains, but the fidelity loss is not enough to guide the generator to optimize to generate images that are very similar to ground truth images. picture. Common fidelity losses are

L _p =||C(x ^a ; θ _G )-x|| _p , (1.1)

Where p is 1 or 2, indicating L1 or L2 norm. Equation (1.1) is a pixel-level fidelity constraint, so it is a strong constraint. From many previous studies, such as the famous ResNet model, encoding residual vectors is more efficient than encoding original vectors. This is because the residual vector contains less encoding information, thus reducing the learning burden of the network. This residual coding can be called spatial coding. This invention attempts to extend the idea of residual coding to the time domain, that is, to allow the network to gradually learn content information features. More specifically, let the network first learn the low-frequency features of the image (i.e., the relatively smooth parts of the CT image), and then capture the high-frequency features (i.e., structures such as edge information in the image). Therefore, the learning process is gradual and the network can gradually evolve. In order to achieve this goal, the present invention needs to design a time-varying loss function to replace the time-invariant loss function.

L _var =F ^t (G(x ^a ; θ _G ), x) (1.2)

where F ^t is a time-varying function about the generated image G (x ^a ; θ _G ) and the ground truth image x. The image is composed of low-frequency components and high-frequency components, which means that the image x can be represented as a combination of low-frequency components and high-frequency components. Due to its relatively smooth characteristics, the low-frequency component of the image can be regarded as a piecewise constant function, so the pixel x _j has a high probability of being similar to the adjacent pixels. If the GAN model is allowed to learn to generate low-frequency parts from the beginning, the pixel fidelity constraints are relaxed by introducing neighborhood similarity constraints, as shown below:

where j is the pixel index, where φ is a sampling function that samples pixels within the neighborhood of pixel x _j , and σ is a parameter that controls the size of the neighborhood. The physical meaning of this loss function L _p is that the pixel x _j is similar to its adjacent pixels. G(x ^a ; θ _G ) learns the low-frequency components of x. If the parameter σ can be changed dynamically to make it a time-varying function, a loss function L _var can be obtained:

The function φ(x _j , σ _t ) is the sampling function of the area near the pixel x _j . As the iteration proceeds, the parameter σ _t gradually converges to zero, that is, the loss function eventually degenerates into:

is a common fidelity loss, which means that constraints are being tightened to prompt the GAN to learn high-frequency components. Next, instantiate a function φ(x _j , σ _t ) to satisfy the above conditions, that is, the function has a sampling function and the parameter σ decreases with time; in theory, there should be many options to build such a function, and similar function extensions are Within the scope of the present invention, the deformable registration network designed by the present invention is a better solution. Its advantage is that the network parameters can be adaptively adjusted without manual intervention, so that the generator can generate more realistic images. Remove artifacts from images. Figure 3 shows a schematic diagram of the GAN network introducing the registration network. First, perform a random affine transformation on each input image x ^a , and then perform a random affine transformation on its reference image x, so that the transformed image can be obtained and x _T . When/> After passing through the generator, the image with artifacts removed is obtained Due to/> and x _T are not pixel-level corresponding images. In order to compare the differences between them, // and x _T are input into the registration network, which partially corrects the previous deformation caused by random affine transformation. As shown in Figure 4, the amplitude of the random affine changes is not large and is within a controllable range. Therefore, the registration network can correct these random deviations.

At the beginning of training, the performance of the registration network is poor, so the output There will be certain position deviations from the affine transformed groundtruth image x _T , and these deviations are variable, that is, each pixel in each input image of each epoch has an independent and different deviation. This feature can help the network simulate the random sampling process of function φ(x _j , σ _t ). Moreover, as training progresses, the registration performance of the registration network will improve, that is, the parameter σ _t will decrease. If the entire model converges well, _σt will drop to zero.

The overall architecture of the model MARGANVAC proposed by this invention is shown in Figure 5. MARGANVAC contains three modules, namely generator G, discriminator D and registration network R. In addition to the novel training mechanism with variable temporal constraints mentioned above, some effective techniques are also introduced to enable the generator to be trained more effectively. First, for the input image x ^a , the linear interpolation method is used to obtain the estimated value of the metal trajectory part in the projection image in the sinogram domain, and the LI corrected reconstructed image x ^[LI]a is obtained through the FBP or FDK method. The image x ^[LI]a corrected by the LI method and the image ^xa affected by the artifact are connected in the channel dimension as the input of the generator G. Secondly, in order to further reduce the learning burden of the generator, residual learning is introduced in the generator G. Then the image after removing metal artifacts is expressed as:

Where [m, n] represents the splicing operation of images m and n. In order to better distinguish metal artifacts and content information, self-reconstruction is introduced as a constraint to regularize the generator. In the self-reconstruction process, the artifact-free image y is spliced into [y, y], and then input to the generator, so the self-reconstructed image is expressed as:

In order for the registration network R to work properly, a random affine transformation is performed on each input artifact image x ^a and ground-truth image x. Assume that the corresponding affine transformed images are A ₁ (x ^a ) and A ₂ (x) respectively. In this way, the image with metal artifacts removed is expressed as:

Correspondingly, in self-reconstruction, the output becomes:

then take and A ₂ (x) as the input of the registration network R, and the output of the registration network R is a deformation vector field T _x ,

After obtaining the deformation vector field T _x , by comparing the input image Apply T _x to get the resampled image/>

Following the same steps as generating T _x , the deformation vector field T _y is obtained,

And in the case of self-reconstruction, the corresponding resampled image

In order for the generator to well separate content information and metal artifacts, a specially designed generator is introduced, as shown in Figure 6. Generator contains encoder and decoder. This encoder is used to map image samples from the image domain to a latent space where content information and features of metal artifacts are separated. The decoder reconstructs the separated content information into an artifact-free image. Between the encoder and decoder, a deep sub-network consisting of 21 Inception-ResNet modules is introduced to improve the separation ability of artifact features and content information in the latent space. The Inception structure proposed in GoogleNet makes full use of convolution kernels of different sizes, so the Inception structure has better performance in extracting features. The Inception-ResNet module is formed by introducing a residual network structure into the Inception structure, so it is easy to build a deep network composed of multiple Inception-ResNet modules to enhance the separation ability of content information and artifact features without worrying about whether the network Convergence issues. The structure of the Inception-ResNet module used is shown in Figure 7(c), in which a 1×1 convolution operator is added to reduce the dimensionality of the input feature map, thereby reducing the amount of calculation. In the encoder, the core module is the downsampling module as shown in Figure 7(b), which consists of a convolutional layer, an instance normalization (InstanceNormalization) layer and a ReLU activation function. The reason for choosing instance normalization over batch normalization (BatchNormalization) is that instance normalization has been proven to have better performance for image generation tasks with small batches. In the decoder, the core module is the upsampling module, as shown in Figure 7(e), which is similar to the downsampling module, the only modification is that the transposed convolution replaces the convolution operation. The discriminator D is built based on the PatchGAN discriminator. The structure of the discriminator is shown in Figure 8. The entire structure of the registration network R is based on the U-Net structure, but is deeper than the traditional U-Net structure. In this very deep U-Net, all levels of features can be obtained to effectively facilitate the registration process.

Loss function design

The learning process of the model is also a process that encourages the generator to reduce metallic artifacts while maintaining content information. In each adversarial learning iteration, the generator G will output an image with reduced metal artifacts; when the performance of both G and D improves, the output images look more like artifact-free images. The function of the introduced registration network is to help the generator G gradually learn content features, so the network weights θ _R and θ _G need to be updated simultaneously during the training process. As can be seen from Figure 5, MARGANVAC contains four forms of losses, namely artifact correction loss Self-reconstruction loss/> Fight against losses/> and a smoothing loss used to constrain the deformation vector field/> The total loss can be expressed as the weighted sum of these losses as follows:

where λ are hyperparameters that balance the importance of each loss in the training process.

correction loss Since the entire model is based on supervised learning, the most direct and effective constraint is the correction loss, which minimizes the difference between the image with metal artifacts removed and the ground truth image. However, after random affine transformation is performed on the input image, there is no strict pixel-level corresponding ground truth image anymore. In order to solve this problem, the registration network R and the generator G are trained simultaneously, and the design correction loss is:

in is the resampled image obtained by equation (1.11), and A ₂ (x) is the ground truth image after affine transformation shown in equation (1.10). In many image-to-image translation tasks, the L1 norm loss function has been proven to be more effective in recovering image details, so the L1 norm is used here to encourage the generator to maintain more details while reducing metallic artifacts.

self-reconstruction loss The training goal is to make the generator learn to remove metal artifacts. At the same time, more constraints need to be introduced to encourage the generator to keep more existing content information unchanged. The core goal is to let the network learn to identify metal artifacts and content information, that is, reduce metal artifacts in the presence of metal and artifacts, and retain all image content information in the absence of metal artifacts. Therefore, the self-reconstruction loss is introduced as follows:

in, is the resampled image obtained by equation (1.13), and A ₄ (y) is the affine transformation of the artifact-free image y.

against losses Adversarial learning can encourage the generator G to generate more realistic artifact-free images. To achieve this goal, the generator should have the ability to distinguish artifact information and content information in the latent space. To enhance this capability, two adversarial learning strategies are introduced. One strategy is to improve the ability of the latent space to identify metal artifacts, and another approach is to improve the ability of the latent space to preserve content information. The input data for the two strategies are images affected by metal artifacts and images without metal artifacts. These two strategies are performed simultaneously in adversarial learning, and their respective losses can be written as:

So the total adversarial loss is:

Smooth loss When minimizing the combined loss of correction loss and self-reconstruction loss, the resulting deformation vector field may become unsmooth and physically unrealistic. In order to solve this problem, a diffusion regular operator in the gradient direction of the deformation vector field is introduced in the original registration network to constrain the deformation vector field and make it smooth. Because the generator has two outputs and/> There are two corresponding deformation vector field regularization terms. Therefore, the total smoothing loss can be expressed as:

in is the gradient of the deformation vector field T. In practice, the differences between pixels are used to approximate the gradient of each pixel in the deformation vector field.

Obviously, those skilled in the art should understand that the above-mentioned steps of the metal artifact removal method based on the time-varying constraint adversarial generation network model of the embodiment of the present invention can be implemented with a general-purpose computing device, and they can be concentrated in a single calculation device, or distributed on a network composed of multiple computing devices, optionally, they can be implemented with program codes executable by the computing device, so that they can be stored in a storage device and executed by the computing device, and In some cases, the steps shown or described may be performed in a different order than herein, or may be fabricated separately into individual integrated circuit modules, or multiple modules or steps thereof may be fabricated into a single integrated circuit module. to fulfill. As such, embodiments of the present invention are not limited to any specific combination of hardware and software.

Preparation for the experiment

(1) Data set composition

Dataset 1: deeplesion simulated metal artifact data set

Select 4,000 artifact-free CT images from the deeplesion data set, select 90 metal masks from 100 binary metal masks of different shapes and sizes provided by CNNMAR to synthesize metal artifacts, and synthesize a total of 360,000 paired CT images as training data set. In addition, 200 artifact-free CT images and the remaining 10 binary metal masks were selected from the deeplesion data set to generate a test data set containing 2000 paired CT images. Fan-beam projection is used, the reconstruction method is FBP, and the image size is 256×256.

Dataset 2: Real Artifact Dataset

The real artifact data set is a Micro CT data set, which is a collection of real CT projection images of bone samples with and without metal implants. And the paired training data set is synthesized using a synthesis method based on real metal artifacts. After obtaining the synthetic projection data inserted into the metal trajectory, the corresponding reconstructed image is obtained through the FDK method. The training set contains 3868 images and the test set contains 537 images. Image size is 364×364.

(2)Implementation and training details

The network is built based on the Pytorch deep learning box and runs on a computer equipped with an Nvidia 2080Ti GPU. During the training process, the Adam optimizer (optimizer parameters are (β1, β2) = (0.5, 0.999)) is used to optimize the loss function. For the deeplesion data set, the batch size is set to 2, the learning rate is set to 0.0001, and the number of training epochs is 5. For the cone beam Micro CT data set, the batch sizea is set to 1, the learning rate is set to 0.0001, and a total of 70 epochs are trained. The weights of the loss function are λ _smooth =10, λ _Adv =1, λ _Corr =20, and λ _Rec =20.

(3)Evaluation indicators

On the paired data set of synthetic artifacts, structural similarity (SSIM), peak signal-to-noise ratio (PSNR), and standard deviation are used to quantitatively evaluate the performance of all MAR methods, including the method proposed by the present invention and other classic MAR method. Specifically, the higher the SSIM and PSNR values, the better the metal artifact removal effect and content information retention performance. The introduction of standard deviation is to evaluate different methods from the dimension of result stability, because in addition to SSIM and PSNR, it is also necessary to pay attention to the stability of the results of different models when encountering different data. On a real cone-beam Micro CT dataset with real metal artifacts, only a qualitative evaluation based on image visual assessment was performed due to the lack of paired artifact-free CT images.

Validity verification of MARGANVAC model

In order to prove the excellent performance of the proposed method in metal artifact removal, several representative MAR methods were experimented, including the traditional LI algorithm, FSMAR algorithm, supervised algorithm CNNMAR, cGANMAR, U-Net, dual domain Methods InDuDoNet, unsupervised ADN algorithm, CycleGAN, attention-guided β-CycleGAN algorithm. All methods are performed on the public deeplesion dataset and the private Micro CT dataset. Except for the model proposed in this invention, all methods are based on public codes or models proposed in published papers.

Because the training process of dual-domain networks involves the projection and reconstruction of CT images, the FDK of cone beam CT requires a large amount of computing resources. Therefore, in the previous training of dual-domain networks, fan-beam CT and Micro CT were used. The data is cone beam CT, so the dual-domain network experiment is only performed on the deeplesion data set, and the dual-domain experiment is not performed on the Micro CT data set.

Evaluated on the deeplesion dataset

Table 1.1 Comparison of PSNR (dB) and SSIM of different methods on the deeplesion data set

In this part of the experiments, various MAR methods are qualitatively and quantitatively analyzed on the paired deeplesion data set. The results of quantitative analysis are obtained based on 2000 test set images. As shown in Table 1.1, it can be intuitively seen that deep learning methods are better than traditional methods, and supervised methods are generally better than unsupervised methods. Among all recurring supervised methods, the dual-domain model InDuDoNet performs best. The PSNR and SSIM scores of the model proposed by the present invention are second only to InDuDoNet, while InDuDoNet is only adapted to fan-beam CT, but the model of the present invention can not only be adapted to fan-beam, but also cone-beam CT.

The image affected by artifacts, its corresponding artifact-free image, and the images after artifacts are removed by various MAR methods are shown in Figure 9. Figure 9 shows an example of a chest slice for demonstration. In the image affected by metal artifacts in Figure 9, almost no tissue/organ structure can be seen in the area adjacent to the metal implant. The stripe-like artifacts run through the entire image. The image content information in the area adjacent to the metal implant is Heavily damaged by metal artifacts. From the images after artifact removal by various MAR methods, it can be seen that most of the stripe-like artifacts away from the metal have been removed. The performance of the deep learning method in this regard is better than the traditional LI method and FSMAR method. After processing by traditional LI and FSMAR methods, the tissue structure is still blurred and there are secondary artifacts. The performance of supervised methods CNNMAR, cGANMAR, InDuDoNet and U-Net is better than traditional methods, but they cannot recover the missing content information well. In contrast, unsupervised methods tend to automatically generate rich details in images, but many of these details are inconsistent with the actual missing structure. Therefore, it is possible to see structures that resemble tissues, organs, or lesions that do not actually exist. β-CycleGAN and ADN improve on this aspect but cannot completely avoid it because unsupervised methods lack fidelity constraints. By analyzing the artifact-removed images of MARGANVAC using the method of the present invention, it can be clearly seen that several blood vessel lumens have been restored and the vertebral body boundaries are intact. These crisp details are not visible in other images. These results show that the method of the present invention shows better performance in reducing metal artifacts and retaining content information. Although the InDuDoNet method is slightly better than the method of the present invention in terms of evaluation indicators ssim and psnr, it can be seen from Figure 9 that the image texture restored by the method of the present invention is closer to the real image than the InDuDoNet method.

By calculating the standard deviation (Std) of ssim and psnr of different MAR methods, the stability of different methods can be compared. As can be seen from Table 1.1, the Std of the PSNR index of the method of the present invention ranks third, and the Std of the SSIM index ranks first. Therefore, compared with other MAR methods, the method of the present invention can obtain more stable results while ensuring good image quality.

Evaluation on Cone Beam Micro CT Artifact Migration Dataset

In order to verify the performance of the proposed MAR method in real-world applications, a data set collected from cone beam Micro CT was prepared. Paired data were generated by migrating metal trajectories affected by artifacts into projections without metal implants. The results of the quantitative analysis of all MAR methods are shown in Table 1.2. It can be seen that the quality of the original image contaminated by metal artifacts is much worse than the simulation experiment in terms of SSIM and PSNR indicators. illustrates that introducing real metal artifacts to synthesize artifact-affected images may lead to more severe image degradation. It can be seen from the quantitative analysis results of the MAR method in Table 1.2 that the performance of all MAR methods has declined, and the method of the present invention is better than other MAR methods. The reduced values may be due to the fact that the quality of the original images affected by artifacts is lower than that of the simulated experiments. The qualitative comparison results of different MAR methods are shown in Figure 10. Shown is an image of a cross-section of a bone affected by artifacts. In Figure 10, a metal implant is inserted into the bone, resulting in severe image degradation and loss of content information. Stronger shadow artifacts can be seen than in the simulation experiments, with a complete loss of some bone tissue details. Part of the trabecular bone disappears, and the intact cortical bone is divided into several parts. It can be seen from the results of artifact removal by the unsupervised methods CycleGAN and ADN that the gaps between the segmented parts of cortical bone are not well filled, and the missing trabeculae are not well restored. β-CycleGAN performs slightly better than CycleGAN and ADN in these two aspects. Compared with unsupervised methods, supervised methods, especially CNNMAR, have made great improvements in these two aspects, while the U-Net method is more inclined to smooth and blur the picture. By comparing the image obtained by the method proposed by the present invention and CNNMAR, it can be seen that the cortical bone boundary restored by the method of the present invention is clearer and closer to the ground truth image. Soft tissue areas close to metal implants are more likely to be contaminated by metal artifacts because their CT values are much smaller relative to bone tissue. But in clinical applications, correct soft tissue imaging is crucial for diagnosis. Therefore, soft tissue recovery performance should be a key indicator for reducing metal artifacts. By comparing soft tissue, it can be found that the method of the present invention has better soft tissue recovery performance while removing metal artifacts. In summary, all the above-mentioned MAR methods based on deep learning can remove most metal artifacts, but their performance in retaining content information is quite different, that is, supervised methods are better than unsupervised methods, among which the method proposed by the present invention performs best .

Table 1.2 Comparison of PSNR (dB) and SSIM of different methods on Micro CT synthetic artifact data set

Performance evaluation on cone beam Micro CT real artifact images

The method proposed in this invention was further tested on a Micro CT data set affected by real metal artifacts to test its performance in practical applications. Since there are no ground truth pictures, its performance can only be evaluated qualitatively. All MAR models are first trained on Micro CT paired training datasets generated using the artifact transfer method, and then tested on real artifact images from cone beam Micro CT. Figure 11 shows the real artifact-affected image and the corresponding artifact-removed images obtained by different MAR methods. From the real artifact image, we can see that the artifact is visually very similar to the synthesized artifact in Figure 10. By comparing the partial enlargements of the result images obtained by different MAR methods, it can be seen that the supervised method has better artifact removal performance than the unsupervised method. It can be shown that the training of supervised methods is successful. In other words, the supervised method is successfully extended to practical applications with the help of paired data sets synthesized by the artifact transfer method. Among all comparative methods, it is obvious that CNNMAR and the method of the present invention perform better in terms of the effect of removing metal artifacts and restoring image content information. Further comparison of the details in the partial enlarged images shows that the anatomical structure of the trabecular bone obtained by the method of the present invention is clearer, there is almost no residual artifact in the soft tissue area around the metal, and the soft tissue boundary is smoother. This result is consistent with the above simulation experiment.

Claims (9)

1. A metal artifact removal method based on time-varying constraints of the adversarial generative network model. It is characterized by constructing a GAN network metal artifact removal model with time-domain variable constraints, referred to as MARGANVAC. This model is in the GAN network. Basically, by introducing time-varying constraints, adaptive fidelity constraints are applied to each part of the entire graph; The GAN network metal artifact removal model with time domain variable constraints includes three modules, namely the generator G, the discriminator D and the registration network R. After the CT image to be processed undergoes random affine changes, the input The generator G is output to the registration network and the discriminator D through the generator G; the registration network R is used for random sampling, and the neighborhood of the sampled pixels gradually decreases as the number of iterations increases; The trained GAN network metal artifact removal model with time domain variable constraints is used to generate an image after removing metal artifacts from the CT image with metal artifacts. 2. The metal artifact removal method of the adversarial generation network model based on time-varying constraints according to claim 1, characterized in that the training process of the GAN network metal artifact removal model with time-domain variable constraints is as follows: First, ^{perform a} random affine transformation on each input ^image image after and x _T ; when the image/> After passing the generator G, we get the image with artifacts removed/> Will/> and x _T input the registration network R; the physical meaning of the registration network is represented by the function /> To express, the first part of the function G(x ^a ; θ _G ) is the output result of the generator subnetwork, θ _G is the parameter of the generator subnetwork, j is the pixel index, and the second part φ is a sampling function for Pixels are sampled within the neighborhood of pixel x _j . σ _t is a parameter used to control the size of the neighborhood. As the iteration proceeds during the training process, the parameter σ _t gradually converges to zero. 3. The metal artifact removal method of the adversarial generative network model based on time-varying constraints according to claim 1, characterized in that residual learning is introduced in the generator G; self-reconstruction is introduced as a constraint to regularize the generator ;The generator contains an encoder and a decoder; the encoder is used to map image samples from the image domain to the latent space, where the features of content information and metal artifacts are separated; the decoder reconstructs the separated content information into artifact-free image; between the encoder and decoder, a deep sub-network consisting of 21 Inception-ResNet modules is introduced. 4. The metal artifact removal method of the adversarial generation network model based on time-varying constraints according to claim 1, characterized in that the training process of the GAN network metal artifact removal model with time-domain variable constraints is introduced. Self-reconstruction is used as a constraint to regularize the generator. During the self-reconstruction process, the artifact-free image y is spliced into [y, y], and then input into the generator; before splicing, the artifact-free image y is subjected to random affine transformation. The result A ₃ (y) is obtained, and the splicing is [A ₃ (y),A ₃ (y)]. Correspondingly, in the self-reconstruction, the generator output becomes: G([A ₃ (y),A ₃ ( y)]; θ _G ); The artifact-free image y corresponds to a reference image that is not affected by artifacts and is also y. 5. The metal artifact removal method based on the time-varying constraint adversarial generative network model according to claim 1, characterized in that, for the image x ^a , a linear interpolation method is used to obtain the metal trajectory part in the projection image in the sinogram domain. The estimated value of , and the LI corrected reconstructed image x ^[LI]a is obtained through the FBP or FDK method; the image x ^[LI]a corrected by the LI method and the image x ^a affected by the artifact are respectively subjected to random affine transformation to obtain the affine The radial transformation results A ₁ (x ^[LI]a ) and a ₁ (x ^a ) are a ₁ (x ^[LI]a ) and A ₁ (x ^a ) are connected in the channel dimension as the input of the generator G. 6. The metal artifact removal method of the adversarial generative network model based on time-varying constraints according to claim 1, characterized in that residual learning is introduced in the generator G, and the image after removing the metal artifact is expressed as: [x ^a ,x ^[LI]a ] represents the connection operation of x ^a and x ^[LI]a ; in self-reconstruction, the generator output becomes/> Then take/> The splicing of A ₂ (x) and A 2 (x) is used as the input of the registration network R. A ₂ (x) represents the random affine transformation of the reference image x. The output of the registration network R is a deformation vector field T _x . θ _R is the registration network parameter. After obtaining the deformation vector field T _x , the input image/> Apply T _x to get the resampled image/> Follow the same steps as generating T _x to obtain the deformation vector field T _y ,/> A ₄ (y) represents the random affine transformation of the reference image y; and in the case of self-reconstruction, the corresponding resampled image/> Generator output image/> and A ₂ (x) as the input of the discriminator D. Similarly, in the case of self-reconstruction, the generator output/> and A ₄ (y) as the input of the discriminator D. 7. The metal artifact removal method based on the time-varying constraint adversarial generative network model according to claim 1, characterized in that the GAN network metal artifact removal model with time-variable constraints includes four loss functions , respectively the artifact correction loss Self-reconstruction loss/> Fight against losses/> and a smoothing loss used to constrain the deformation vector field The total loss is expressed as the weighted sum of these losses/> Each λ represents the weight coefficient of the corresponding loss; correction loss The registration network R and the generator G are trained simultaneously, and the design correction loss is: Using the L1 norm allows the generator to maintain more detail while reducing metallic artifacts, Express expectations,/> Indicates that x comes from the data set/> The definition domain of CT images affected by metal artifacts is/> self-reconstruction loss The training goal is to make the generator learn to remove metal artifacts. At the same time, more constraints need to be introduced to enable the generator to retain more content information, that is, to reduce metal artifacts in the presence of metal and artifacts, and to reduce metal artifacts in the absence of metal artifacts. retain all image content information; in, Express expectations,/> Indicates that y comes from the data set/> The domain of an image that is not affected by artifacts is/> against losses Adversarial learning encourages the generator G to generate more realistic artifact-free images; to achieve this goal, the generator should have the ability to distinguish artifact information and content information in the latent space; to enhance this ability, two types of adversarial are introduced learning strategy; one strategy is to improve the ability of the latent space to identify metal artifacts, and the other strategy is to improve the ability of the latent space to preserve content information; the input data of the two strategies are images affected by metal artifacts and images that do not contain Image of metal artifacts; these two strategies are performed simultaneously in adversarial learning, and their respective losses are: The base of log is 2, and D(.) represents the output of the discriminator D; so the total adversarial loss is: Smooth loss for: in is the gradient of the deformation vector field T. 8. A computer device, characterized in that: the computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the implementation of claims 1-7 is achieved. The metal artifact removal method based on the time-varying constraint adversarial generative network model described in any one of the above. 9. A computer-readable storage medium, characterized in that: the computer-readable storage medium stores metal artifact removal for performing the time-varying constraint-based adversarial generation network model according to any one of claims 1-7. Method computer program.