US20240041412A1 - Few-view ct image reconstruction system - Google Patents
Few-view ct image reconstruction system Download PDFInfo
- Publication number
- US20240041412A1 US20240041412A1 US18/381,214 US202318381214A US2024041412A1 US 20240041412 A1 US20240041412 A1 US 20240041412A1 US 202318381214 A US202318381214 A US 202318381214A US 2024041412 A1 US2024041412 A1 US 2024041412A1
- Authority
- US
- United States
- Prior art keywords
- image
- network
- few
- generator network
- generator
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000002591 computed tomography Methods 0.000 claims abstract description 94
- 238000012549 training Methods 0.000 claims description 77
- 238000000034 method Methods 0.000 claims description 45
- 230000008569 process Effects 0.000 claims description 17
- 238000007781 pre-processing Methods 0.000 abstract description 20
- 230000006870 function Effects 0.000 description 30
- 230000015654 memory Effects 0.000 description 26
- 238000001914 filtration Methods 0.000 description 17
- 238000012545 processing Methods 0.000 description 15
- 238000013135 deep learning Methods 0.000 description 13
- 238000013528 artificial neural network Methods 0.000 description 8
- 230000009977 dual effect Effects 0.000 description 7
- 238000005070 sampling Methods 0.000 description 5
- 230000008901 benefit Effects 0.000 description 4
- 238000013527 convolutional neural network Methods 0.000 description 4
- 238000010586 diagram Methods 0.000 description 4
- 238000013507 mapping Methods 0.000 description 4
- 238000012986 modification Methods 0.000 description 4
- 230000004048 modification Effects 0.000 description 4
- 210000002569 neuron Anatomy 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 230000004913 activation Effects 0.000 description 2
- 230000014509 gene expression Effects 0.000 description 2
- 230000036541 health Effects 0.000 description 2
- 230000005865 ionizing radiation Effects 0.000 description 2
- 230000005291 magnetic effect Effects 0.000 description 2
- 230000007246 mechanism Effects 0.000 description 2
- 239000004065 semiconductor Substances 0.000 description 2
- 206010028980 Neoplasm Diseases 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 201000011510 cancer Diseases 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000003745 diagnosis Methods 0.000 description 1
- 238000002059 diagnostic imaging Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000009499 grossing Methods 0.000 description 1
- 239000011159 matrix material Substances 0.000 description 1
- 238000005259 measurement Methods 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 229910052704 radon Inorganic materials 0.000 description 1
- SYUHGPGVQRZVTB-UHFFFAOYSA-N radon atom Chemical compound [Rn] SYUHGPGVQRZVTB-UHFFFAOYSA-N 0.000 description 1
- 238000007670 refining Methods 0.000 description 1
- 238000012216 screening Methods 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
Images
Classifications
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B6/00—Apparatus for radiation diagnosis, e.g. combined with radiation therapy equipment
- A61B6/02—Devices for diagnosis sequentially in different planes; Stereoscopic radiation diagnosis
- A61B6/03—Computerised tomographs
- A61B6/032—Transmission computed tomography [CT]
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B6/00—Apparatus for radiation diagnosis, e.g. combined with radiation therapy equipment
- A61B6/52—Devices using data or image processing specially adapted for radiation diagnosis
- A61B6/5205—Devices using data or image processing specially adapted for radiation diagnosis involving processing of raw data to produce diagnostic data
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T11/00—2D [Two Dimensional] image generation
- G06T11/003—Reconstruction from projections, e.g. tomography
- G06T11/005—Specific pre-processing for tomographic reconstruction, e.g. calibration, source positioning, rebinning, scatter correction, retrospective gating
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T11/00—2D [Two Dimensional] image generation
- G06T11/003—Reconstruction from projections, e.g. tomography
- G06T11/006—Inverse problem, transformation from projection-space into object-space, e.g. transform methods, back-projection, algebraic methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2211/00—Image generation
- G06T2211/40—Computed tomography
- G06T2211/421—Filtered back projection [FBP]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2211/00—Image generation
- G06T2211/40—Computed tomography
- G06T2211/436—Limited angle
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2211/00—Image generation
- G06T2211/40—Computed tomography
- G06T2211/441—AI-based methods, deep learning or artificial neural networks
Landscapes
- Engineering & Computer Science (AREA)
- Health & Medical Sciences (AREA)
- Physics & Mathematics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Medical Informatics (AREA)
- High Energy & Nuclear Physics (AREA)
- General Health & Medical Sciences (AREA)
- Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
- Optics & Photonics (AREA)
- Pathology (AREA)
- Radiology & Medical Imaging (AREA)
- Biomedical Technology (AREA)
- Heart & Thoracic Surgery (AREA)
- Molecular Biology (AREA)
- Surgery (AREA)
- Animal Behavior & Ethology (AREA)
- Biophysics (AREA)
- Public Health (AREA)
- Veterinary Medicine (AREA)
- Algebra (AREA)
- Mathematical Analysis (AREA)
- Mathematical Optimization (AREA)
- Mathematical Physics (AREA)
- Pure & Applied Mathematics (AREA)
- Pulmonology (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Apparatus For Radiation Diagnosis (AREA)
Abstract
A system for few-view computed tomography (CT) image reconstruction is described. The system includes a preprocessing module, a first generator network, and a discriminator network. The preprocessing module is configured to apply a ramp filter to an input sinogram to yield a filtered sinogram. The first generator network is configured to receive the filtered sinogram, to learn a filtered back-projection operation and to provide a first reconstructed image as output. The first reconstructed image corresponds to the input sinogram. The discriminator network is configured to determine whether a received image corresponds to the first reconstructed image or a corresponding ground truth image. The generator network and the discriminator network correspond to a Wasserstein generative adversarial network (WGAN). The WGAN is optimized using an objective function based, at least in part, on a Wasserstein distance and based, at least in part, on a gradient penalty.
Description
- This application is a continuation of U.S. application Ser. No. 17/642,725, filed Mar. 14, 2022, which was the National Stage of International Application No. PCT/US2020/050654, filed Sep. 14, 2020, that claims the benefit of U.S. Provisional Application No. 62/899,517, filed Sep. 12, 2019, and U.S. Provisional Application No. 63/077,745, filed Sep. 14, 2020, which are incorporated by reference as if disclosed herein in their entireties.
- This invention was made with government support under award numbers CA233888 and CA237267 awarded by the National Institutes of Health (NIH), and under award number EB026646, awarded by the National Institutes of Health (NIH). The government has certain rights in the invention.
- The present disclosure relates to few-view CT (computed tomography) image reconstruction.
- X-ray computed tomography (CT) is a popular medical imaging method for screening, diagnosis, and image guided intervention. Although CT brings overwhelming healthcare benefits to patients, it may potentially increase cancer risk due to the involved ionizing radiation. Low-dose CT and few-view CT result in a reduced exposure to the ionizing radiation but typically at a cost of reduced image quality.
- In an embodiment, there is provided a system for few-view computed tomography (CT) image reconstruction. The system includes a preprocessing module, a first generator network, and a discriminator network. The preprocessing module is configured to apply a ramp filter to an input sinogram to yield a filtered sinogram. The first generator network is configured to receive the filtered sinogram, to learn a filtered back-projection operation and to provide a first reconstructed image as output. The first reconstructed image corresponds to the input sinogram. The discriminator network is configured to determine whether a received image corresponds to the first reconstructed image or a corresponding ground truth image. The first generator network and the discriminator network correspond to a Wasserstein generative adversarial network (WGAN). The WGAN is optimized using an objective function based, at least in part, on a Wasserstein distance and based, at least in part, on a gradient penalty.
- In some embodiments, the system further includes a second generator network. The second generator network is configured to receive a concatenation of the first reconstructed image and a filtered back-projection of the input sinogram. The second generator network is further configured to provide a second reconstructed image. The discriminator network is further configured to determine whether the received image corresponds to the second reconstructed image.
- In some embodiments of the system, the first generator network is configured to learn the filtered back-projection operation in a point-wise manner.
- In some embodiments of the system, the first generator network includes a filtration portion, a back-projection portion, and a refinement portion.
- In some embodiments of the system, the WGAN is trained, initially, using image data from an image database including a plurality of images.
- In some embodiments of the system, the first generator network is configured to reconstruct the first reconstructed image using O(C×N×Nv) parameters, where N is a dimension of the first reconstructed image, Nv is a number of projections and C is an adjustable hyper-parameter in the range of 1 to N.
- In some embodiments of the system, the second generator network corresponds to a refinement portion.
- In an embodiment, there is provided a method for few-view computed tomography (CT) image reconstruction. The method includes applying, by a preprocessing module, a ramp filter to an input sinogram to yield a filtered sinogram; receiving, by a first generator network, the filtered sinogram; learning, by the first generator network, a filtered back-projection operation; and providing, by the first generator network, a first reconstructed image as output. The first reconstructed image corresponds to the input sinogram. The method further includes determining, by a discriminator network, whether a received image corresponds to the first reconstructed image or a corresponding ground truth image. The first generator network and the discriminator network correspond to a Wasserstein generative adversarial network (WGAN). The WGAN is optimized using an objective function based, at least in part, on a Wasserstein distance and based, at least in part, on a gradient penalty.
- In some embodiments, the method further includes receiving, by a second generator network, a concatenation of the first reconstructed image and a filtered back-projection of the input sinogram; providing, by the second generator network, a second reconstructed image; and determining, by the discriminator network, whether the received image corresponds to the second reconstructed image.
- In some embodiments of the method, the first generator network is configured to learn the filtered back-projection operation in a point-wise manner.
- In some embodiments of the method, the first generator network includes a filtration portion, a back-projection portion, and a refinement portion.
- In some embodiments, the method further includes learning, by the first generator network, an initial filtered back-projection operation using image data from an image database including a plurality of images.
- In some embodiments of the method, the first generator network is configured to reconstruct the first reconstructed image using O(C×N×Nv) parameters, where N is a dimension of the first reconstructed image, Nv is a number of projections and C is an adjustable hyper-parameter in the range of 1 to N.
- In some embodiments, the method further includes receiving, by a filtered back projection module, the input sinogram and providing, by the filtered back projection module, the filtered back-projection of the input sinogram.
- In an embodiment, there is provided a computer readable storage device. The device has stored thereon instructions configured for few-view computed tomography (CT) image reconstruction. The instructions that when executed by one or more processors result in the following operations including: applying a ramp filter to an input sinogram to yield a filtered sinogram; receiving the filtered sinogram; learning a filtered back-projection operation; providing a first reconstructed image as output, the first reconstructed image corresponding to the input sinogram; and determining whether a received image corresponds to the first reconstructed image or a corresponding ground truth image, the operations corresponding to a Wasserstein generative adversarial network (WGAN). The WGAN is optimized using an objective function based, at least in part, on a Wasserstein distance and based, at least in part, on a gradient penalty.
- In some embodiments of the device, the operations further include receiving a concatenation of the first reconstructed image and a filtered back-projection of the input sinogram; providing a second reconstructed image; and determining whether the received image corresponds to the second reconstructed image.
- In some embodiments of the device, the filtered back-projection operation is learned in a point-wise manner.
- In some embodiments of the device, the operations further include learning an initial filtered back-projection operation using image data from an image database including a plurality of images.
- In some embodiments of the device, the first reconstructed image is reconstructed using O(C×N×Nv) parameters, where N is a dimension of the first reconstructed image, Nv is a number of projections and C is an adjustable hyper-parameter in the range of 1 to N.
- The drawings show embodiments of the disclosed subject matter for the purpose of illustrating features and advantages of the disclosed subject matter. However, it should be understood that the present application is not limited to the precise arrangements and instrumentalities shown in the drawings, wherein:
-
FIG. 1 illustrates a functional block diagram of a system that includes a deep learning CT image reconstruction system consistent with several embodiments of the present disclosure; -
FIG. 2 illustrates a functional block diagram of a system that includes a dual network architecture (DNA) CT image reconstruction system consistent with several embodiments of the present disclosure; -
FIG. 3 is a flow chart of deep learning CT image reconstruction system training operations according to various embodiments of the present disclosure; and -
FIG. 4 is a flow chart of dual network architecture (DNA) CT image reconstruction system training operations according to various embodiments of the present disclosure. - Current commercial CT scanners typically use one or two x-ray sources that are mounted on a rotating gantry to take hundreds of projections at different angles around a patient's body. The rotating mechanism is massive and consumes substantial energy related to a net angular momentum generated during the rotation. Thus, outside major hospitals, current commercial CT scanners are largely inaccessible due to their size, weight and expense. Few-view CT may be implemented in a mechanically stationary scanner thus avoiding the rotating mechanism and associated power consumption.
- The Nyquist sampling theorem provides a lower bound on an amount of data used for image reconstruction. For example, when sufficient (i.e., above the Nyquist limit) projection data are acquired, analytic methods such as filtered back-projection (FBP) may provide relatively high quality CT image reconstruction. In few-view CT, due in part to under-sampled data, streak artifacts may be introduced in analytically reconstructed images because of incomplete projection data. Iterative techniques may incorporate prior knowledge in the image reconstruction but can be relatively time-consuming and may not produce satisfying results in some cases.
- Generally, the present disclosure relates to a few-view CT image reconstruction system. In an embodiment, the few-view CT image reconstruction system corresponds to a deep efficient end-to-end reconstruction (DEER) network for few-view CT image reconstruction. In another embodiment, the few-view CT image reconstruction system corresponds to a dual network architecture (DNA) CT image reconstruction system. A method and/or system consistent with the present disclosure may be configured to receive CT scanner projection data (i.e., sinograms) and to generate a corresponding image. A system may include at least one generator network and a discriminator network configured as a generative adversarial neural network (GAN). The generator network(s) and the discriminator network correspond to artificial neural networks. In an embodiment, the generator network(s) and the discriminator network may correspond to convolutional neural networks. The generator network(s) and discriminator network may be trained, adversarially, as will be described in more detail below. The trained generator network(s) may then be configured to receive filtered few view projection data and to provide a reconstructed image as output.
- In an embodiment, at least one generator network may correspond to a back projection network (BPN). The BPN may be configured to reconstruct a CT image directly from raw (i.e., sinogram) data using, for example, O(C×N×Nv) parameters. N corresponds to dimension of reconstructed image and Nv corresponds to number of projections. C is an adjustable hyper-parameter and is in the range of 1 to N. A BPN consistent with the present disclosure may thus be trainable on one consumer-level GPU (graphics processing unit). However, this disclosure is not limited in this regard. The BPN, similar to filtered back projection (FBP), is configured to learn a refined filtration back-projection process for reconstructing images directly from sinograms. For X-ray CT, each point in the sinogram domain relates to pixels/voxels on an X-ray path through a field of view. Thus, a plurality of line integrals acquired by a plurality of different detectors at particular angle are not related to each other. With this intuition, the reconstruction process of BPN is learned in a point-wise manner that facilitates constraining a memory burden.
- In some embodiments, the generator network may be pre-trained using natural images from a publicly available image database, e.g., ImageNet. The generator network may then be refined using actual patient data. Advantageously, the complexity of natural images may facilitate learning the back-projection process.
-
FIG. 1 illustrates a functional block diagram of asystem 100 that includes a deep learning CTimage reconstruction system 102, consistent with several embodiments of the present disclosure. CTimage reconstruction system 102 includes elements configured to implement training of a back projection network (BPN), as will be described in more detail below.System 100 further includes acomputing device 104.Computing device 104 is configured to perform the operations of deep learning CTimage reconstruction system 102. - The
computing device 104 may include, but is not limited to, a server, a workstation computer, a desktop computer, a laptop computer, a tablet computer, an ultraportable computer, an ultramobile computer, a netbook computer and/or a subnotebook computer, etc.Computing device 104 includes aprocessor 110, amemory 112, input/output (I/O)circuitry 114, a user interface (UI) 116, andstorage 118. - CT
image reconstruction system 102 includes atraining module 120, atraining data store 122, apreprocessing module 124, agenerator network 126, and adiscriminator network 128.Generator network 126 includes a filtration portion 126-1, a back-projection portion 126-2 and a refinement portion 126-3.Generator network 126, after training, corresponds to a BPN. As used herein, the terms “generator network” and “generative network” are used interchangeably. -
Processor 110 may include one or more processing units and is configured to perform operations ofsystem 100, e.g., operations oftraining module 120,preprocessing module 124,generator network 126, anddiscriminator network 128.Memory 112 may be configured to store data associated withtraining module 120,preprocessing module 124,generator network 126, anddiscriminator network 128, and/ortraining data store 122. I/O circuitry 114 may be configured to communicate wired and/or wirelessly with a source of projection data and/or a recipient of a corresponding generated image.UI 116 may include a user input device (e.g., keyboard, mouse, microphone, touch sensitive display, etc.) and/or a user output device, e.g., a display.Storage 118 is configured to store at least a portion oftraining data store 122.Training data store 122 is configured to store training data including, but not limited to, one or moreobjective functions 140, one or moretraining data sets 142,generator parameters 146 and discriminator parameters 148. -
Training module 120 is configured to manage training operations of generator network 126 (and discriminator network 128).Training module 120 may thus be configured to provide training projection data topreprocessing module 124 and ground truth image data to discriminatornetwork 128. The training projection data and ground truth image data may be stored, for example, intraining data store 122 as training data sets 142.Training module 120 may be further configured to provide an objective function, e.g.,objective function 140, todiscriminator network 128 and to receive a decision from discriminator network.Training module 120 may be further configured to provide, adjust and/or receivegenerator parameters 127 and/ordiscriminator parameters 129 during training operations. Such parameters may include, for example, neural network weights. Generator parameters may be stored in training data store asgenerator parameters 146 and discriminator parameters may be stored in training data store as discriminator parameters 148. After training, i.e., during normal operations, the generator parameters may be set, CTimage reconstruction system 102 may be configured to receive projection data in (corresponding to an actual CT sinogram) and may be configured to provide a corresponding generatedimage 121 as generated image output. - CT image reconstruction may be expressed as:
-
I FV =R −1(S SV) (1) - where IFV∈ w×w is an object image with dimension w×w, SSV∈ w×w is the sinogram with dimension v×w and R−1 corresponds to an inverse radon transform (e.g., filtered back projection (FBP)) in an instance where sufficient two dimensional (2D) projection data is available. When sufficient 2D projection data is available, CT image reconstruction can be reduced to solving a system of linear equations. If the number of linear equations is less than the number of unknown pixels as in the few-view CT setting, the image reconstruction is an underdetermined problem. Deep learning (DL) may be utilized to extract features of raw data for image reconstruction. With a deep neural network, as described herein, training data corresponds to prior knowledge configured to establish a relationship between a sinogram and the corresponding CT image. Thus, a trained deep neural network may be configured to efficiently solving this undetermined problem.
- In operation, CT
image reconstruction system 102 is configured as a Wasserstein Generative Adversarial Network (WGAN) to optimize (i.e., train)generator network 126. After optimization,generator network 126 may correspond to a back projection network (BPN). TheBPN 126 is configured to receive preprocessed, as described herein, few view CT projection data, and to reconstruct a corresponding CT image. - A WGAN generally includes a generator network, e.g.,
generator network 126, and a discriminator network, e.g.,discriminator network 128. Thegenerator network 126 aims at reconstructing images directly from a batch of few-view sinograms. Thediscriminator network 128 is configured to receive generatedimage data 121 fromgenerator network 126 or ground truth image data from, e.g.,training module 120, and intends to distinguish whether an image is real (i.e., ground truth) or fake (from generator network 126). Bothnetworks discriminator network 128 can hardly distinguish fake images from real images, thengenerator network 126 can fooldiscriminator network 128 which is the goal of WGAN. In other words, ifdiscriminator network 128 is unable to distinguish between a generated image from thegenerator network 126 and a ground truth image, thegenerator network 126 has been optimized, i.e., is trained. Thediscriminator network 128 may facilitate improving a texture of the final image and reduce occurrence of over-smoothing. - WGAN is configured to replace a cross-entropy loss function of a non-Wasserstein generative adversarial network (GAN) with the Wasserstein distance. The Wasserstein distance is configured to improve the training stability during the training process compared to the GAN. In an embodiment, an objective function used during training includes the Wasserstein distance as well as a gradient penalty term. The objective function of the
discriminator network 128 may be written as: -
- where D corresponds to operation of the
discriminator network 128, G corresponds to operation of thegenerator network 126, SSV and IFV represent sparse-view sinograms and ground-truth images, respectively. Terms of the form a[b] in Eq. 2A denote an expectation of b as a function of a. θG and θD represent the trainable parameters of thegenerator network 126 and thediscriminator network 128, respectively. Ī=α·IFV+(1−α)·G(SSV)·α is uniformly sampled from the interval [0,1]. In other words, Ī represents images between fake and real images. ∇(Ī) denotes the gradient of D with respect to Ī. λ is a parameter used to balance the Wasserstein distance term and gradient penalty term. Thegenerator network 126 and the discriminator network 128 (e.g., the generator parameters and the discriminator parameters) may be updated iteratively. - The input to the
BPN 126 is a batch of few-view sinograms. According to Fourier slice theorem, low-frequency information is sampled denser than high-frequency information. It may be appreciated that performing back-projection directly on the batch of few-view sinograms may result in blurry reconstructed images.Preprocessing module 124 is configured to implement a ramp filter to filter received projection data (i.e., sinogram) to avoid this blurry issue. The ramp filter operation may be performed on sinograms in the Fourier domain as multiplication. The filter length may be set as twice the length of the sinogram. Theoretically, the length of the filter is infinitely long for a bandlimited signal, but it is not practical in reality. Since values of the filter outside twice the length of sinograms are generally at or near zero, filter length is set as twice the length of sinograms. The filtered sinograms, i.e., output of thepreprocessing module 124, may then be provided to thegenerator network 126. -
Generator network 126 is configured to learn a revised filtration back-projection operation and output reconstructed images, i.e., generatedimage 121.Generator network 126 includes three components: a filtration portion 126-1, a back-projection portion 126-2 and a refinement portion 126-3. - In filtration portion 126-1, a plurality of one dimensional (1-D) convolutional layers are used to learn small variance to filtered sinograms. Because the filtration portion 126-1 is a multi-layer CNN, different layers can learn different parts of the filter. In one nonlimiting example, the 1-D convolutional window may be set as one quarter the length of the sinograms. The length of the 1-D convolutional window is configured to reduce computational burden. The idea of residual connection may be used to preserve high-resolution information and prevent gradient from vanishing. Inspired by the ResNeXt structure, in one nonlimiting example, a cardinality of the convolutional layers may be 3. It may be appreciated that increasing the cardinality of the network may be more effective than increasing the depth or width of the network when the network capacity is increased. In
BPN 126, the value of the cardinality may correspond to a number of branches. - The learned sinograms from the filtration portion 126-1 may then be provided to the back-projection portion 126-2. It may be appreciated that each point in the sinogram relates to pixel values on the x-ray path through corresponding object image and any other pixels do not contribute to the point. Thus, the reconstruction process may be learned in a point-wise manner using a point-wise fully-connected layer. Thus, the
generator network 126, by learning in a point-wise manner, may learn the back-projection process with relatively fewer parameters compared to other methods. Learning with relatively fewer parameters may utilize relatively fewer memory resources. In other words, for a sinogram with dimension Nv×N, there is a total of Nv×N relatively small fully-connected layers in the method, as described herein. The respective input to each of these relatively small fully-connected layers is a single point in the sinogram and the output is a line with dimension N×1. After this point-wise fully-connected layer, rotation and summation may be applied to simulate FBP, and to put all the learned lines to their appropriate positions. Bilinear interpolation may be used for rotating images and maintaining the rotated image on a Cartesian grid. - This network design is configured to allow the corresponding neural network to learn the reconstruction process using N parameters. In some situations, due to the relative complexity of medical images and incomplete projection data (due to the few-view input data), N parameters may not be sufficient for learning relatively high-quality images. Thus, in an embodiment, the number of parameters may be increased to O(C×N×Nv) in this point-wise fully-connected layer to by increasing the number of branches to C (an adjustable hyper-parameter). The increase in number of parameters is further supported by using a different set of parameters for different angles in order to compensate the negative effect introduced by bilinear interpolation. An amount of bias terms in this point-wise fully-connected layer is the same as the amount of weights in order to learn fine details in medical images. Bias terms are added along the detector direction. Then, there is one 2-D convolutional layer with 3×3 kernel and
stride 1 configured to combine all the learned mappings from sinogram domain to image domain. It should be noted that by learning in this point-wise manner, each point in the sinogram becomes a training sample instead of a whole sinogram and in order to reduce training time, a plurality of fully-connected layers may be implemented together by one piece-wise multiplication. - Images reconstructed in the back-projection portion 126-2 may then be provided to the last portion of
generator network 126, i.e., the refinement portion 126-3. Refinement portion 126-3 may be configured to remove remaining artifacts. For example, the refinement portion 126-3 may correspond to a U-net, including conveying paths, and may be built with ResNeXt structure. The conveying paths are configured to copy early feature maps and reuse them as part of the input to later layers. Concatenation is used to combine early and later feature maps along the channel dimension. Thegenerator network 126 may then be configured to preserve high-resolution features. Each layer in the U-net may be followed by a rectified linear unit (ReLU). 3×3 kernels may be used in both convolutional and transpose-convolutional layers. A stride of 2 may be used for down-sampling and up-sampling layers and stride of 1 may be used for all other layers. In order to maintain the tensor's size, zero-padding is used. - The
discriminator network 128 is configured to receive input from either generator network 126 (i.e., generated image 121) or the ground-truth dataset (e.g., ground truth image data from training module 120). As described herein, thediscriminator network 128 may be configured to distinguish whether the input is real or fake. In one nonlimiting example, thediscriminator network 128 may contain 6 convolutional layers with 64, 64, 128, 128, 256, 256 filters, respectively, and followed by 2 fully-connected layers with number ofneurons 1024 and 1, respectively. A leaky ReLU activation function may be used after each layer with a slope of 0.2, for example, in the negative part. A convolutional window of 3×3 and zero-padding may be used for all convolutional layers. Stride may be equal to 1 for odd layers and 2 for even layers. - Generally, the objective function used for optimizing a generator network may include one or more of mean square error (MSE), adversarial loss and structural similarity index (SSIM). MSE may effectively suppress back-ground noise, but may result in over-smoothed images. Generally, MSE may not be sensitive to image texture. MSE generally assumes background noise is white Gaussian noise that is independent of local image features. The formula of MSE loss may be expressed as:
-
- where Nb, W and H correspond to the number of batches, image width and image height respectively. Yi and Xi represent ground-truth image and image reconstructed by
generator network 126, respectively. In order to compensate the disadvantages of MSE and acquire visually better images, SSIM is introduced in the objective function. SSIM aims to measure structural similarity between two images. In one nonlimiting example, the convolution window used to measure SSIM is set as 11×11. The SSIM formula is expressed as: -
- where C1=(K1·R)2 and C2=(K2·R)2 are constants used to stabilize the formula if the denominator is small. R stands for the dynamic range of pixel values and, in one nonlimiting example, K1=0.01 and K2=0.03. μY, μX, σY 2, σX 2 and σYX are the means of Y and X, variances of Y and X and the covariance between Y and X, respectively. The structural loss may then be expressed as:
-
L sl=1−SSIM(Y,X) (5) - The adversarial learning technique used in BPN aims to help
generator network 126 produce sharp images that are indistinguishable by thediscriminator network 128. Referring to Eq. 1, adversarial loss may be written as: - The overall objective function of the
generator network 126 may then be written as: -
L G=λQ ·L al+λP ·L sl +L 2 (7A) - where λQ and λP are hyper-parameters used to balance different loss functions.
- Thus, a deep efficient end-to-end reconstruction (DEER) network for few-view CT image reconstruction system, consistent with the present disclosure, may include a generator network and a discriminator network. The generator network and discriminator network may be trained, adversarially, using a WGAN framework, as described herein. The DEER network for few-view CT image reconstruction system may then be configured to receive CT scanner projection data (i.e., sinograms), to filter the received projection data and to generate a corresponding image. In some embodiments, the generator network and discriminator network may be pre-trained using, for example, ImageNet data. The CT image reconstruction process of the BPN network is learned in a point-wise manner that facilitates constraining a memory burden.
-
FIG. 2 illustrates a functional block diagram ofsystem 200 that includes a dual network architecture (DNA) CTimage reconstruction system 202 consistent with several embodiments of the present disclosure.DNA system 202 includes elements configured to implement training generator network(s) and a discriminator network, as will be described in more detail below.System 200 further includescomputing device 104, as described herein.Computing device 104 is configured to perform the operations of dual network CTimage reconstruction system 202.Storage 118 may be configured to store at least a portion oftraining data store 222, as described herein. - It may be appreciated that DNA CT
image reconstruction system 202 has at least some elements and features in common with CTimage reconstruction system 102 ofFIG. 1 . In the interest of descriptive efficiency, the common elements and features will be only briefly described, with reference provided to the description herein related to the CTimage reconstruction system 102 ofFIG. 1 . - CT
image reconstruction system 202 includes atraining module 220, atraining data store 222, apreprocessing module 224, a filtered back projection (FBP)module 226, a first generator network (Gen 1) 228, anintermediate processing module 230, a second generator network (Gen 2) 232, and adiscriminator network 234.Training data store 222 is configured to store training data including, but not limited to, one or more objective function(s) 240, one or moretraining data sets 242, first generator (Gen 1)parameters 244, second generator (Gen 2)parameters 246, anddiscriminator parameters 248. - The
preprocessing module 224 corresponds to preprocessingmodule 124 ofFIG. 1 . The first generator (Gen 1)network 228 corresponds to thegenerator network 126 ofFIG. 1 . Similar to the deep learning CTimage reconstruction system 102 ofFIG. 1 , DNA CTimage reconstruction system 202 is configured to receive CT scanner projection data (i.e., sinograms) and to generate (i.e., reconstruct) a corresponding image (Final output image 233). The DNA CTimage reconstruction system 202 may be trained, adversarially, as described herein. A subsystem that includespreprocessing module 224, trainedgenerator networks intermediate processing module 230, may then be configured to receive filtered projection data and to provide a reconstructed image as output. - DNA CT
image reconstruction system 202 includes two generator networks:Gen 1network 228 andGen 2network 232. As used herein, the terms “G1”, “Gen 1” and “Gen 1 network” are used interchangeably and all refer toGen 1network 228 ofFIG. 2 . As used herein, the terms “G2”, “Gen 2” and “Gen 2 network” are used interchangeably and all refer toGen 2network 232 ofFIG. 2 .Training module 220 is configured to manage training operations ofgenerator networks discriminator network 234, similar totraining module 120. -
Training module 220 may thus be configured to provide training projection data (i.e., input sinogram) topreprocessing module 224 andFBP module 226.Training module 220 may be further configured to provide ground truth image data to discriminatornetwork 234. The training projection data and ground truth image data may be stored, for example, intraining data store 222 as training data sets 242.Training module 220 may be further configured to provide an objective function, e.g.,objective function 240, todiscriminator network 234 and to receive a decision from discriminator network.Training module 220 may be further configured to provide, adjust and/or receiveGen 1 parameters 243,Gen 2parameters 245, and/ordiscriminator parameters 247 during training operations. Such parameters may include, for example, neural network weights.Gen 1 parameters may be stored in training data store asGen 1parameters 244,Gen 2 parameters may be stored in training data store asGen 2parameters 246, and discriminator parameters may be stored in training data store asdiscriminator parameters 248. After training, i.e., during normal operations, theGen 1 andGen 2 parameters may be set, CTimage reconstruction system 202 may be configured to receive projection data in (corresponding to an actual CT sinogram) and may be configured to provide a corresponding generated image asfinal output image 233. - In operation,
preprocessing module 224 andFBP module 226 are configured to receive training projection data (e.g., a batch of few-view sinograms) from, e.g.,training module 220.Preprocessing module 224 is configured to filter the few-view sinograms to yield filteredfew view sinograms 225. In one nonlimiting example, the filter length may be twice the length of the sinogram. The filtering corresponds to a ramp filter applied to the sinograms in the Fourier domain, as described herein. - The filtered few-
view sinograms 225 may then be provided toGen 1network 228.Gen 1network 228 corresponds togenerator network 126 ofFIG. 1 .Gen 1network 228 is configured to operate on the filtered few-view sinograms (i.e., to learn a filtered back projection technique) to produce anintermediate output 229. Theintermediate output 229 may correspond to reconstructed image that may then be provided to theintermediate processing module 230. TheFBP module 226 is configured to perform filtered back-projection on the received training projection data (e.g., a batch of few-view sinograms) and to provide anFBP result 227 to theintermediate processing module 230. Theintermediate processing module 230 is configured to concatenate theintermediate output 229 with theFBP result 227 and to provide the concatenatedresult 231 to theGen 2network 232. TheGen 2network 232 is configured to operate on the concatenated result 231 (e.g., to optimize the concatenated result) to produce afinal output image 233. Theintermediate output 229 and thefinal output image 233 may be further provided to thediscriminator network 234. Thediscriminator network 234 is further configured to receive ground truth image data and to provide at least one decision indicator to, for example,training module 220. - Similar to
generator network 126 ofFIG. 1 , theGen 1network 228 may include three portions: filtration 228-1, back-projection 228-2, and refinement 228-3. The filtration portion 228-1 may correspond to a multi-layer CNN. In the filtration part 228-1, 1-D convolutional layers are used to produce filtered data. In one nonlimiting example, filter length of filtration portion 228-1 may be set to twice a length of a projection vector. It may be appreciated that the length of the projection vector may be shortened. Since the filtration is done through a multi-layer CNN, different layers can learn different parts of the filter. In one nonlimiting example, the 1-D convolutional window may be empirically set as one quarter the length of the projection vector to reduce the computational burden. Residual connections may be used to preserve high-resolution information and to prevent gradient from vanishing. - The learned sinogram from the filtration portion 228-1 may then be provided to the back-projection portion 228-2. The back-projection portion 228-2 is configured to perform back-projection operations on the received learned sinogram. Operation of the back-projection portion 228-2 is inspired by the following intuition: every point in the filtered projection vector only relates to pixel values on the x-ray path through the corresponding object image and any other data points in this vector contribute nothing to the pixels on this x-ray path. As is known a single fully-connected layer can be implemented to learn the mapping from the sinogram domain to the image domain, but relies on relatively large matrix multiplications in this layer that may tax memory. To reduce the memory burden, DNA CT image reconstruction system 202 (e.g.,
Gen 1 network 228) is configured to learn the reconstruction process in a point-wise manner using a point-wise fully-connected layer. Back-projection portion 228-2 may then learn the back-projection process. The input to the point-wise fully-connected layer corresponds to a single point in the filtered projection vector. The number of neurons may then correspond to a width of the corresponding image. After this point-wise fully-connected layer, rotation and summation operations are applied to simulate the analytical FBP method. Bilinear interpolation may be used for rotating images. In one nonlimiting example, C may be empirically set as 23, allowing the network to learn multiple mappings from the sinogram domain to the image domain. The value of C can be understood as the number of branches. Different view-angle may use different parameters. Although the proposed filtration and back-projection parts all together learn a refined FBP method, streak artifacts may not be eliminated perfectly. An image reconstructed by the back-projection part 228-2 may thus be provided to the refinement portion 228-3 ofGen 1 for refinement. - The refinement portion 228-1 may correspond to a U-net with conveying paths and may be constructed with the ResNeXt structure. In one nonlimiting example, U-net may be configured to contain 4 down-sampling and 4 up-sampling layers. Each layer may have a stride of 2 and may be followed by a rectified linear unit (ReLU). A 3×3 kernel may be included in both convolutional and transpose-convolutional layers. The number of kernels in each layer is 36. To maintain the tensors size, zero-padding is used.
-
Gen 2network 232 is configured to have a same structure as the refinement portion 228-3 inGen 1. Theinput 231 to G2 is a concatenation of FBP-result 227 andoutput 229 from G1. With the use of G2, the network becomes deep. As a result, the benefits of deep learning can be utilized in this direct mapping for CT image reconstruction. - In operation, similar to the deep learning CT
image reconstruction system 102 ofFIG. 1 , DNA CTimage reconstruction system 202 is optimized using the Wasserstein Generative Adversarial Network (WGAN) framework. As described herein, the DNA CTimage reconstruction system 202 includes three components: two generator networks:Gen 1network 228 andGen 2network 232, and adiscriminator network 234.Gen 1 andGen 2 aim at reconstructing images directly from a batch of few-view sinograms. Thediscriminator network 234 is configured to receive images fromGen 1 andGen 2 and a ground-truth dataset, and intends to distinguish whether an image is real (i.e., is from the ground-truth dataset) or fake (i.e., is from G1 or G2). The networks are configured to be optimized in the training process. If an optimized network D can hardly distinguish fake images from real images, then it is concluded that generators G1 and G2 can fool discriminator D which is the goal of GAN. The network D is configured to help to improve the texture of the final image and prevent over-smoothed issue from occurring. - Different from generative adversarial network (GAN), Wasserstein GAN (WGAN) replaces the cross-entropy loss function with the Wasserstein distance, improving the training stability during the training process, as described herein. In an embodiment, an objective function used during training of the DNA CT
image reconstruction system 202 includes the Wasserstein distance as well as a gradient penalty term. The objective function of the WGAN framework for the DNA CTimage reconstruction system 202 may be expressed as: -
- where SSV, ISV=G1(SSV), IFV represent a sparse-view sinogram, an image reconstructed by
Gen 1 from a sparse-view sinogram and the ground-truth image reconstructed from the full-view projection data, respectively. Similar to Eq. 2A, terms of the form a[b] in Eq. 2B denote an expectation of b as a function of a. θG1 , θG2 and θD represent the trainable parameters ofGen 1network 228,Gen 2network 232 andDiscriminator network 234, respectively. Ī represents images between fake (from G1 or G2) and real (from the ground-truth dataset) images. ∇(Ī) denotes the gradient of D with respect to Ī. The parameter λ balances the Wasserstein distance terms and gradient penalty terms. G1, G2 and D may be updated iteratively. - The objective function for optimizing the generator networks,
Gen 1 andGen 2, may include the mean square error (MSE), structural similarity index (SSIM) and adversarial loss. MSE is a popular choice for denoising applications, which effectively suppresses the background noise but could result in over-smoothed images. Generally, MSE may be insensitive to image texture since it assumes background noise is white Gaussian noise and is independent of local image features. The formula of MSE loss (L2) is expressed as Eq. 3, as described herein, where Nb, W and H denote the number of batches, image width and image height respectively. Yi and Xi represent ground-truth image and image reconstructed by generator networks (G1 or G2), respectively. - To compensate for the disadvantages of MSE and acquire visually better images, SSIM is introduced in the objective function. The SSIM formula may be expressed as Eq. 4, as described herein. The structural loss may then be expressed as Eq. 5, as described herein.
- The adversarial loss aims to assist the
generators discriminator network 234. Referring to Eq. 2B, adversarial loss forGen 1 may be expressed as: - and adversarial loss for G2 is expressed as:
- It may be appreciated that solving the few-view CT image reconstruction is similar to solving a set of linear equations when the number of equations is not sufficient to perfectly resolve all the unknowns. DNA CT
image reconstruction system 202 is configured to estimate the unknown by combining the information from the existing equations and the knowledge contained in the big data. MSE between the original sinogram and the synthesized sinogram from a reconstructed image (fromGen 1 or Gen 2) may be included as part of the objective function, which may be written as: -
- where Nb, V, H denote the number of batches, number of views and sinogram height, respectively. Yi sino represents the original sinogram and Xi sino represents sinogram from a reconstructed image (from
Gen 1 or Gen 2). - Both generator networks,
Gen 1,Gen 2 may be updated at the same time. The overall objective function of two generators, e.g.,generator networks -
- where the superscripts (1) and (2) indicate that the term is for measurements between ground-truth images and results reconstructed by G1 and G2, respectively. λQ, λP and λR are hyper-parameters used to balance different loss functions.
- The
discriminator network 234 is configured to receive inputs from G1 and G2, and the ground-truth dataset, and to try to distinguish whether each input is real or fake. In one nonlimiting example, thediscriminator network 234 may include 6 convolutional layers with 64, 64, 128, 128, 256, 256 filters and followed by 2 fully-connected layers with numbers of neurons 1,024 and 1, respectively. The leaky ReLU activation function may be used after each layer with a slope of 0.2, for example, in the negative part. A 3×3 kernel and zero-padding are used for all the convolutional layers, with stride equal 1 for odd layers and stride equal 2 for even layers. - Thus, a dual network architecture CT image reconstruction system, consistent with the present disclosure, may include a plurality of generator networks and a discriminator network. The generator networks and discriminator network may be trained, adversarially, using a WGAN framework, as described herein. The DNA CT image reconstruction system may then be configured to receive CT scanner projection data (i.e., sinograms), to filter the received projection data and to generate a corresponding image. The CT image reconstruction process of the generator networks is learned in a point-wise manner that facilitates constraining a memory burden. In some embodiments, the generator network(s) and discriminator network may be pre-trained using, for example, ImageNet data.
-
FIG. 3 is aflowchart 300 of deep learning CT image reconstruction training operations according to various embodiments of the present disclosure. In particular, theflowchart 300 illustrates training a deep learning CT image reconstruction system to reconstruct an image from a few-view sinogram. The operations may be performed, for example, by deep learning CT image reconstruction system 102 (e.g.,preprocessing module 124,generator network 126, and/or discriminator network 128) ofFIG. 1 . - In some embodiments, operations may include operation 302. Operation 302 includes learning an initial filtered back-projection operation using image data from an image database that includes a plurality of images. For example, the image database may correspond to ImageNet. Operation 304 may include receiving projection data (i.e., an input sinogram). A ramp filter may be applied to the input sinogram to yield a filtered sinogram at
operation 306. The filtered sinogram may be received by a first generator network atoperation 308.Operation 310 may include learning a filtered back-projection operation. A first reconstructed image corresponding to the input sinogram may be provided as output atoperation 312.Operation 314 may include determining, by a discriminator network, whether a received image corresponds to the first reconstructed image or a corresponding ground truth image. The generator network and the discriminator network correspond to a Wasserstein generative adversarial network (WGAN). The WGAN is optimized using an objective function based, at least in part, on a Wasserstein distance and based, at least in part, on a gradient penalty. - Thus, a deep learning CT image reconstruction system may be trained for few-view CT image reconstruction.
-
FIG. 4 is aflow chart 400 of dual network architecture (DNA) CT image reconstruction system training operations according to various embodiments of the present disclosure. In particular, theflowchart 400 illustrates training a DNA CT image reconstruction system to reconstruct an image from a few-view sinogram. The operations may be performed, for example, by DNA CT image reconstruction system 202 (e.g.,preprocessing module 224, filtered back projection (FBP)module 226, first generator network (Gen 1) 228,intermediate processing module 230, second generator network (Gen 2) 232, and/or discriminator network 234) ofFIG. 2 . - In some embodiments, operations may include operation 402. Operation 402 includes learning an initial filtered back-projection operation using image data from an image database that includes a plurality of images. For example, the image database may correspond to ImageNet.
Operation 404 may include receiving projection data (i.e., an input sinogram). A ramp filter may be applied to an input sinogram to yield a filtered sinogram atoperation 406. The input sinogram may be processed by a filtered back projection module to yield a filtered back projection result atoperation 408. The filtered sinogram may be received by a first generator network atoperation 410.Operation 412 may include learning a filtered back-projection operation by the first generator network. A first reconstructed image corresponding to the input sinogram may be provided as an intermediate output atoperation 414. The first reconstructed image and a filtered back projection result may be concatenated atoperation 416.Operation 418 may include refining a concatenation result by a second generator network.Operation 420 may include determining, by a discriminator network, whether a received image corresponds to the first reconstructed image, the second reconstructed image or a corresponding ground truth image. The generator networks and the discriminator network correspond to a Wasserstein generative adversarial network (WGAN). The WGAN is optimized using an objective function based, at least in part, on a Wasserstein distance and based, at least in part, on a gradient penalty. - Thus, a DNA CT image reconstruction system may be trained for few-view CT image reconstruction.
- As used in any embodiment herein, the terms “logic” and/or “module” may refer to an app, software, firmware and/or circuitry configured to perform any of the aforementioned operations. Software may be embodied as a software package, code, instructions, instruction sets and/or data recorded on non-transitory computer readable storage medium. Firmware may be embodied as code, instructions or instruction sets and/or data that are hard-coded (e.g., nonvolatile) in memory devices.
- “Circuitry”, as used in any embodiment herein, may include, for example, singly or in any combination, hardwired circuitry, programmable circuitry such as computer processors comprising one or more individual instruction processing cores, state machine circuitry, and/or firmware that stores instructions executed by programmable circuitry. The logic and/or module may, collectively or individually, be embodied as circuitry that forms part of a larger system, for example, an integrated circuit (IC), an application-specific integrated circuit (ASIC), a system on-chip (SoC), desktop computers, laptop computers, tablet computers, servers, smart phones, etc.
- The foregoing provides example system architectures and methodologies, however, modifications to the present disclosure are possible. The
processor 110 may include one or more processing units and may be configured to perform operations of one or more circuitries, modules and/or artificial neural networks. Processing units may include, but are not limited to, general-purpose processing units, graphical processing units, parallel processing units, etc. -
Memory 112 may include one or more of the following types of memory: semiconductor firmware memory, programmable memory, non-volatile memory, read only memory, electrically programmable memory, random access memory, flash memory, magnetic disk memory, and/or optical disk memory. Either additionally or alternatively system memory may include other and/or later-developed types of computer-readable memory. - Embodiments of the operations described herein may be implemented in a computer-readable storage device having stored thereon instructions that when executed by one or more processors perform the methods. The processor may include, for example, a processing unit and/or programmable circuitry. The storage device may include a machine readable storage device including any type of tangible, non-transitory storage device, for example, any type of disk including floppy disks, optical disks, compact disk read-only memories (CD-ROMs), compact disk rewritables (CD-RWs), and magneto-optical disks, semiconductor devices such as read-only memories (ROMs), random access memories (RAMs) such as dynamic and static RAMs, erasable programmable read-only memories (EPROMs), electrically erasable programmable read-only memories (EEPROMs), flash memories, magnetic or optical cards, or any type of storage devices suitable for storing electronic instructions.
- The terms and expressions which have been employed herein are used as terms of description and not of limitation, and there is no intention, in the use of such terms and expressions, of excluding any equivalents of the features shown and described (or portions thereof), and it is recognized that various modifications are possible within the scope of the claims. Accordingly, the claims are intended to cover all such equivalents.
- Various features, aspects, and embodiments have been described herein. The features, aspects, and embodiments are susceptible to combination with one another as well as to variation and modification, as will be understood by those having skill in the art. The present disclosure should, therefore, be considered to encompass such combinations, variations, and modifications.
Claims (20)
1. A few-view computed tomography (CT) image reconstruction system, the system comprising:
a generator network configured to receive a few-view sinogram, and to generate a reconstructed image corresponding to the few-view sinogram; and
a discriminator network configured to receive an input image, and to determine whether the received input image corresponds to the reconstructed image or a ground truth image,
wherein the generator network and the discriminator network correspond to a Wasserstein generative adversarial network (WGAN), the generator network is configured to learn a reconstruction process in a point-wise manner, and a trained generator network is configured to reconstruct a few-view CT image directly from a corresponding input few-view sinogram.
2. The system of claim 1 , wherein the generator network comprises a point-wise fully-connected layer.
3. The system of claim 1 , wherein the generator network is configured to reconstruct the reconstructed image using O(C×N×Nv) parameters, where N is a dimension of the reconstructed image, Nv is a number of projections and C is an adjustable hyper-parameter in the range of 1 to N.
4. The system of claim 1 , wherein the WGAN is trained, initially, using image data from an image database comprising a plurality of images.
5. The system of claim 1 , wherein an objective function used during training comprises a Wasserstein distance and a gradient penalty.
6. The system of claim 1 , wherein an objective function that is configured to optimize the generator network during training comprises an error term, and a structural similarity index term.
7. The system of claim 1 , wherein the generator network corresponds to a back propagation network.
8. A method for few-view computed tomography (CT) image reconstruction, the method comprising:
receiving, by a generator network, a few-view sinogram;
generating, by the generator network, a reconstructed image corresponding to the few-view sinogram;
receiving, by a discriminator network, an input image; and
determining, by the discriminator network, whether the received input image corresponds to the reconstructed image or a ground truth image,
wherein the generator network and the discriminator network correspond to a Wasserstein generative adversarial network (WGAN), the generator network is configured to learn a reconstruction process in a point-wise manner, and a trained generator network is configured to reconstruct a few-view CT image directly from a corresponding input few-view sinogram.
9. The method of claim 8 , wherein the generator network comprises a point-wise fully-connected layer.
10. The method of claim 8 , wherein the generator network is configured to reconstruct the reconstructed image using O(C×N×Nv) parameters, where N is a dimension of the reconstructed image, Nv is a number of projections and C is an adjustable hyper-parameter in the range of 1 to N.
11. The method of claim 8 , wherein the WGAN is trained, initially, using image data from an image database comprising a plurality of images.
12. The method of claim 8 , wherein an objective function used during training comprises a Wasserstein distance and a gradient penalty.
13. The method of claim 8 , wherein an objective function that is configured to optimize the generator network during training comprises an error term, and a structural similarity index term.
14. The method of claim 8 , wherein the generator network corresponds to a back propagation network.
15. A computer readable storage device having stored thereon instructions configured for few-view computed tomography (CT) image reconstruction, the instructions that when executed by one or more processors result in the following operations comprising:
receiving a few-view sinogram;
generating a reconstructed image corresponding to the few-view sinogram;
receiving an input image; and
determining whether the received input image corresponds to the reconstructed image or a ground truth image,
wherein the operations correspond to a Wasserstein generative adversarial network (WGAN), a reconstruction process is learned in a point-wise manner, and a trained generator network is configured to reconstruct a few-view CT image directly from a corresponding input few-view sinogram.
16. The device of claim 15 , wherein the generator network comprises a point-wise fully-connected layer.
17. The device of claim 15 , wherein the reconstructed image is reconstructed using O(C×N×Nv) parameters, where N is a dimension of the reconstructed image, Nv is a number of projections and C is an adjustable hyper-parameter in the range of 1 to N.
18. The device of claim 15 , wherein the instructions that when executed by one or more processors result in the following additional operations comprising: training the WGAN, initially, using image data from an image database comprising a plurality of images.
19. The device of claim 15 , wherein an objective function used during training comprises a Wasserstein distance and a gradient penalty.
20. The device of claim 15 , wherein an objective function that is configured to optimize the generator network during training comprises an error term, and a structural similarity index term.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US18/381,214 US20240041412A1 (en) | 2019-09-12 | 2023-10-18 | Few-view ct image reconstruction system |
Applications Claiming Priority (5)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US201962899517P | 2019-09-12 | 2019-09-12 | |
US202063077745P | 2020-09-14 | 2020-09-14 | |
PCT/US2020/050654 WO2021051049A1 (en) | 2019-09-12 | 2020-09-14 | Few-view ct image reconstruction system |
US202217642725A | 2022-03-14 | 2022-03-14 | |
US18/381,214 US20240041412A1 (en) | 2019-09-12 | 2023-10-18 | Few-view ct image reconstruction system |
Related Parent Applications (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US17/642,725 Continuation US11806175B2 (en) | 2019-09-12 | 2020-09-14 | Few-view CT image reconstruction system |
PCT/US2020/050654 Continuation WO2021051049A1 (en) | 2019-09-12 | 2020-09-14 | Few-view ct image reconstruction system |
Publications (1)
Publication Number | Publication Date |
---|---|
US20240041412A1 true US20240041412A1 (en) | 2024-02-08 |
Family
ID=74866490
Family Applications (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US17/642,725 Active 2040-10-31 US11806175B2 (en) | 2019-09-12 | 2020-09-14 | Few-view CT image reconstruction system |
US18/381,214 Pending US20240041412A1 (en) | 2019-09-12 | 2023-10-18 | Few-view ct image reconstruction system |
Family Applications Before (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US17/642,725 Active 2040-10-31 US11806175B2 (en) | 2019-09-12 | 2020-09-14 | Few-view CT image reconstruction system |
Country Status (2)
Country | Link |
---|---|
US (2) | US11806175B2 (en) |
WO (1) | WO2021051049A1 (en) |
Families Citing this family (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US11580356B2 (en) * | 2019-07-19 | 2023-02-14 | Qualcomm Incorporated | Efficient inferencing with piecewise pointwise convolution |
CN117036533B (en) * | 2023-10-10 | 2024-02-13 | 北京理工大学 | Sparse angle CT reconstruction method and system for coaxial phase contrast imaging |
Family Cites Families (14)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US10970887B2 (en) * | 2016-06-24 | 2021-04-06 | Rensselaer Polytechnic Institute | Tomographic image reconstruction via machine learning |
US10475214B2 (en) * | 2017-04-05 | 2019-11-12 | General Electric Company | Tomographic reconstruction based on deep learning |
US11257259B2 (en) * | 2017-08-15 | 2022-02-22 | Siemens Healthcare Gmbh | Topogram prediction from surface data in medical imaging |
WO2019128660A1 (en) * | 2017-12-29 | 2019-07-04 | 清华大学 | Method and device for training neural network, image processing method and device and storage medium |
US10709394B2 (en) | 2018-01-15 | 2020-07-14 | Siemens Healthcare Gmbh | Method and system for 3D reconstruction of X-ray CT volume and segmentation mask from a few X-ray radiographs |
WO2019147767A1 (en) | 2018-01-24 | 2019-08-01 | Rensselaer Polytechnic Institute | 3-d convolutional autoencoder for low-dose ct via transfer learning from a 2-d trained network |
WO2019156877A1 (en) | 2018-02-06 | 2019-08-15 | Hrl Laboratories, Llc | Domain adaption learning system |
US10832392B2 (en) * | 2018-12-19 | 2020-11-10 | Siemens Healthcare Gmbh | Method, learning apparatus, and medical imaging apparatus for registration of images |
CN109697741B (en) * | 2018-12-28 | 2023-06-16 | 上海联影智能医疗科技有限公司 | PET image reconstruction method, device, equipment and medium |
CN110097611B (en) * | 2019-04-28 | 2023-09-22 | 上海联影智能医疗科技有限公司 | Image reconstruction method, device, equipment and storage medium |
US11049223B2 (en) * | 2019-06-19 | 2021-06-29 | Siemens Healthcare Gmbh | Class-aware adversarial pulmonary nodule synthesis |
US11410374B2 (en) * | 2019-10-09 | 2022-08-09 | Siemens Healthcare Gmbh | Synthetic parameterized computed tomography from surface data in medical imaging |
WO2021120069A1 (en) * | 2019-12-18 | 2021-06-24 | 深圳先进技术研究院 | Low-dose image reconstruction method and system on basis of a priori differences between anatomical structures |
US11076824B1 (en) * | 2020-08-07 | 2021-08-03 | Shenzhen Keya Medical Technology Corporation | Method and system for diagnosis of COVID-19 using artificial intelligence |
-
2020
- 2020-09-14 US US17/642,725 patent/US11806175B2/en active Active
- 2020-09-14 WO PCT/US2020/050654 patent/WO2021051049A1/en active Application Filing
-
2023
- 2023-10-18 US US18/381,214 patent/US20240041412A1/en active Pending
Also Published As
Publication number | Publication date |
---|---|
US20220375142A1 (en) | 2022-11-24 |
US11806175B2 (en) | 2023-11-07 |
WO2021051049A1 (en) | 2021-03-18 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US10387765B2 (en) | Image correction using a deep generative machine-learning model | |
CN110462689B (en) | Tomographic reconstruction based on deep learning | |
US11580410B2 (en) | 3-D convolutional autoencoder for low-dose CT via transfer learning from a 2-D trained network | |
US11315221B2 (en) | Apparatus and method for image reconstruction using feature-aware deep learning | |
US11769277B2 (en) | Deep learning based scatter correction | |
CN110809782B (en) | Attenuation correction system and method | |
US20240041412A1 (en) | Few-view ct image reconstruction system | |
US11540798B2 (en) | Dilated convolutional neural network system and method for positron emission tomography (PET) image denoising | |
US8346011B2 (en) | Reducing noise in an image | |
US20210142476A1 (en) | Systems and methods for image optimization | |
US11589834B2 (en) | Deep neural network for CT metal artifact reduction | |
Kim et al. | Low‐dose CT reconstruction using spatially encoded nonlocal penalty | |
US20220130084A1 (en) | Systems and methods for medical image processing using deep neural network | |
US20210186439A1 (en) | Method for generating an x-ray image dataset | |
US20210097662A1 (en) | Modularized adaptive processing neural network (map-nn) for low-dose ct | |
Xie et al. | Dual network architecture for few-view CT-trained on ImageNet data and transferred for medical imaging | |
EP4179496A1 (en) | 3d-cnn processing for ct image noise removal | |
US20230079353A1 (en) | Image correction using an invertable network | |
US20230177746A1 (en) | Machine learning image reconstruction | |
CN111325695A (en) | Low-dose image enhancement method and system based on multi-dose grade and storage medium | |
US20220044452A1 (en) | Hybrid image reconstruction system | |
Srinivasan et al. | Realizing the effective detection of tumor in magnetic resonance imaging using cluster-sparse assisted super-resolution | |
US11455755B2 (en) | Methods and apparatus for neural network based image reconstruction | |
JP7362460B2 (en) | Medical image processing device, method and storage medium | |
US20220327750A1 (en) | Medical image processing method, medical image processing apparatus, and computer readable non-volatile storage medium storing medical image processing program |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: RENSSELAER POLYTECHNIC INSTITUTE, NEW YORK Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:XIE, HUIDONG;WANG, GE;SHAN, HONGMING;AND OTHERS;SIGNING DATES FROM 20220317 TO 20220318;REEL/FRAME:065319/0924 |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION |