US10909416B2 - Deep variational method for deformable image registration - Google Patents
Deep variational method for deformable image registration Download PDFInfo
- Publication number
- US10909416B2 US10909416B2 US16/440,215 US201916440215A US10909416B2 US 10909416 B2 US10909416 B2 US 10909416B2 US 201916440215 A US201916440215 A US 201916440215A US 10909416 B2 US10909416 B2 US 10909416B2
- Authority
- US
- United States
- Prior art keywords
- deformation field
- image
- deformation
- probability density
- source image
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active, expires
Links
- 238000004800 variational method Methods 0.000 title description 9
- 238000000034 method Methods 0.000 claims abstract description 77
- 238000009826 distribution Methods 0.000 claims abstract description 53
- 238000012546 transfer Methods 0.000 claims abstract description 30
- 230000008859 change Effects 0.000 claims abstract description 23
- 230000008569 process Effects 0.000 claims abstract description 22
- 230000009466 transformation Effects 0.000 claims abstract description 11
- 238000005070 sampling Methods 0.000 claims description 11
- 238000012549 training Methods 0.000 claims description 6
- 230000001902 propagating effect Effects 0.000 claims description 4
- 230000002787 reinforcement Effects 0.000 claims description 2
- 230000006870 function Effects 0.000 description 12
- 238000003384 imaging method Methods 0.000 description 10
- 230000000644 propagated effect Effects 0.000 description 10
- 238000010586 diagram Methods 0.000 description 7
- 239000011159 matrix material Substances 0.000 description 7
- 206010028980 Neoplasm Diseases 0.000 description 5
- 238000013528 artificial neural network Methods 0.000 description 5
- 238000002595 magnetic resonance imaging Methods 0.000 description 4
- 210000002569 neuron Anatomy 0.000 description 4
- 230000001105 regulatory effect Effects 0.000 description 4
- 210000003484 anatomy Anatomy 0.000 description 3
- 239000003795 chemical substances by application Substances 0.000 description 3
- 230000033001 locomotion Effects 0.000 description 3
- 238000013507 mapping Methods 0.000 description 3
- 210000000056 organ Anatomy 0.000 description 3
- 238000001959 radiotherapy Methods 0.000 description 3
- 230000008901 benefit Effects 0.000 description 2
- 238000013527 convolutional neural network Methods 0.000 description 2
- 238000012937 correction Methods 0.000 description 2
- 238000012804 iterative process Methods 0.000 description 2
- 238000005457 optimization Methods 0.000 description 2
- 238000007670 refining Methods 0.000 description 2
- 238000012706 support-vector machine Methods 0.000 description 2
- 238000003325 tomography Methods 0.000 description 2
- ORILYTVJVMAKLC-UHFFFAOYSA-N Adamantane Natural products C1C(C2)CC3CC1CC2C3 ORILYTVJVMAKLC-UHFFFAOYSA-N 0.000 description 1
- 230000005856 abnormality Effects 0.000 description 1
- 230000003466 anti-cipated effect Effects 0.000 description 1
- 238000013459 approach Methods 0.000 description 1
- 201000011510 cancer Diseases 0.000 description 1
- 238000002591 computed tomography Methods 0.000 description 1
- 238000004590 computer program Methods 0.000 description 1
- 230000001143 conditioned effect Effects 0.000 description 1
- 230000003750 conditioning effect Effects 0.000 description 1
- 125000004122 cyclic group Chemical group 0.000 description 1
- 238000013506 data mapping Methods 0.000 description 1
- 238000003745 diagnosis Methods 0.000 description 1
- 238000006073 displacement reaction Methods 0.000 description 1
- 239000003814 drug Substances 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000002594 fluoroscopy Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000029058 respiratory gaseous exchange Effects 0.000 description 1
- 238000002560 therapeutic procedure Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
- 238000012800 visualization Methods 0.000 description 1
- 230000004584 weight gain Effects 0.000 description 1
- 235000019786 weight gain Nutrition 0.000 description 1
- 230000004580 weight loss Effects 0.000 description 1
Images
Classifications
-
- G06K9/6215—
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/30—Determination of transform parameters for the alignment of images, i.e. image registration
- G06T7/35—Determination of transform parameters for the alignment of images, i.e. image registration using statistical methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F17/00—Digital computing or data processing equipment or methods, specially adapted for specific functions
- G06F17/10—Complex mathematical operations
- G06F17/18—Complex mathematical operations for evaluating statistical data, e.g. average values, frequency distributions, probability functions, regression analysis
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/22—Matching criteria, e.g. proximity measures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N20/00—Machine learning
- G06N20/20—Ensemble learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
- G06N3/084—Backpropagation, e.g. using gradient descent
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N5/00—Computing arrangements using knowledge-based models
- G06N5/04—Inference or reasoning models
- G06N5/046—Forward inferencing; Production systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/74—Image or video pattern matching; Proximity measures in feature spaces
- G06V10/75—Organisation of the matching processes, e.g. simultaneous or sequential comparisons of image or video features; Coarse-fine approaches, e.g. multi-scale approaches; using context analysis; Selection of dictionaries
- G06V10/755—Deformable models or variational models, e.g. snakes or active contours
- G06V10/7557—Deformable models or variational models, e.g. snakes or active contours based on appearance, e.g. active appearance models [AAM]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10072—Tomographic images
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
Definitions
- the present disclosure relates to a method for determining a correspondence between a source image and a reference image.
- Deformable image registration aims to find correspondence between pairs of images.
- deformable image registration has a wide range of applications both in diagnosis and in therapy.
- longitudinal images of the same patient are often acquired to monitor the progression of a tumour and to plan radiation therapy.
- a radiation therapy plan is updated automatically depending on the progression of the tumour.
- Deformable registration of the longitudinal images corrects for differences in aspect, as well as deformations of the patient's body surrounding the tumour, for example caused by weight gain/loss, breathing, and/or other movements, allowing for efficient visualisation and comparison of the images, either manually or automatically.
- deformable image registration examples include motion correction, in which a series of images of a continuously/continually moving/deforming target are captured one after another (for example, as frames of a video), in which case deformable image registration corrects for the continuous/continual moving/deforming tissue to allow residual changes to the target to be identified.
- deformable image registration of images of corresponding anatomies of multiple patients, allowing for abnormalities in one or more of the anatomies to be identified or segment organs using an atlas image.
- Deformable image registration involves determining a deformation field representing a correspondence between a source image and a reference image. Since the deformation field is typically high-dimensional and mathematically ill-defined, it needs to be heavily regularised.
- Existing methods include parametric approaches in which a model of the deformation field is controlled by a relatively small number of control points or parameters based on prior knowledge of possible deformations. Such methods are often unable to model complex deformations accurately, for example those exhibited by parts of the human anatomy.
- Existing methods further include regularization of an energy computed on, for example, a Jacobian, Hessian, or Laplacian of the deformation field.
- Such methods can result in over-smoothed/over-regularised deformation fields and/or deformations fields that do not represent realistic deformations due to a lack of high-level prior knowledge of possible deformations.
- deformable image registration typically involves the optimization of a high-dimensional functional. Iterative optimization procedure are most commonly used, leading to time-consuming algorithms that require minutes or more to compute.
- FIG. 1 shows a deformable registration of a source image to a reference image.
- FIG. 2 is a schematic diagram of a method for determining a deformation field.
- FIG. 3 is a flow diagram of a routine for determining a deformation field.
- FIG. 4 is a schematic diagram of a variational autoencoder (VAE).
- VAE variational autoencoder
- FIG. 5 is a schematic diagram of a routine for determining a deformation field and refining a generative model.
- FIG. 6 is a schematic diagram of a conditional variational autoencoder (CVAE).
- FIG. 7 is a flow diagram of a routine for determining a set of deformation fields.
- FIG. 8 is a block diagram of a computing device for registering a source image to a reference image.
- an image is, for example, a set of digital data mapping a two- or three-dimensional domain to one or more real numbers, each of the real numbers associating an intensity with a voxel/pixel within the image domain.
- image may refer either to the data itself, or to a visual representation of the data when rendered using, for example, a screen or a projector.
- Images may be captured using various imaging techniques including, for example, digital photography, magnetic resonance imaging (MRI), X-ray fluoroscopy, proton emission tomography (PET), echography, and computerised tomography (CT). Images captured using the same imaging technique may use different imaging parameters. For example, MRI may be parameterised as T1 weighted or T2 weighted. A given imaging technique using given imaging parameters is referred to as an imaging modality.
- MRI magnetic resonance imaging
- PET proton emission tomography
- CT computerised tomography
- a deformation field is a mapping that transforms a co-ordinate system of an image. Deformation fields are expressed differently depending on the deformation model being used. In some examples, a deformation field is represented by a pixel-to-vector mapping or a voxel-to-vector mapping, associating a displacement vector with each pixel/voxel of the image.
- FIG. 1 a shows a two-dimensional reference image 100 containing three objects: a triangle 102 , a circle 104 , and a die 106 .
- the reference image 100 has an associated Cartesian co-ordinate system, represented by a Cartesian grid 108 .
- the reference image 100 was captured at a first time.
- FIG. 1 b shows a two-dimensional source image 110 , similarly containing three objects: a triangle 112 , a circle 114 , and a die 116 .
- the source image 110 has an associated Cartesian co-ordinate system, represented by a Cartesian grid 118 .
- the source image 110 was captured at a second time, using the same imaging modality as that of the reference image 100 . It is observed that, while the triangle 112 and the circle 114 appear to correspond to the triangle 102 and the circle 104 , the die 116 appears warped compared with the die 106 .
- the present embodiment provides a method of determining a correspondence between the source image 110 and the reference image 100 .
- the correspondence is represented by a deformation field ⁇ , which encodes a coordinate transformation that may be applied to the Cartesian grid 118 of the source image 110 .
- FIG. 1 c shows a transformed source image 120 , which is the result of applying the deformation field ⁇ to the source image 110 .
- the transformed source image 120 contains a triangle 122 , a circle 124 , and a die 126 .
- the transformed source image 120 has a transformed co-ordinate system, represented by a transformed grid 128 .
- the deformation field ⁇ is non-uniform, and accordingly different regions of the source image 100 are deformed differently to arrive at the transformed source image 120 .
- features within the source image move as a result of the source image being transformed. For this reason, source images are sometimes referred to as moving images, whereas reference images are sometimes referred to as fixed images.
- the transformed source image 120 approximately corresponds to the reference image 100 .
- the die 126 of the transformed source image 120 appears to correspond approximately to the die 106 of the reference image 100 .
- closely comparing the reference image 100 with the transformed source image 120 reveals that a central spot on the die 126 appears in a different position to a corresponding central spot on the die 106 .
- This difference is residual to the co-ordinate transformation encoded by the deformation field ⁇ , suggesting that in the interval between the first time and the second time, the central spot has moved relative to its surrounding region. Comparing the reference image 100 with the transformed source image could be performed manually, for example by overlaying the transformed source image on top of the reference image, or alternatively could be performed automatically by a computer.
- the reference image 100 and the source image 110 have the same modality. Determining a correspondence between the source image 110 and the reference image 100 is therefore referred to as monomodal registration.
- a source image and a reference image are captured using different modalities, and determining a correspondence between the source image and the reference image is referred to as multimodal registration.
- the present embodiments are applicable both to monomodal registration and to multimodal registration, as will be described in detail hereafter.
- an exemplary correspondence determining method 200 for determining a correspondence between a source image I S and a reference image I R employs a generative model 202 , a conditional model 204 , and an update routine 206 .
- the generative model 202 corresponds to a prior probability distribution of deformation fields, each deformation field corresponding to a respective co-ordinate transformation.
- the prior probability distribution represents the expected distribution of deformation fields, given the particular application to which the deformable registration method is being applied. For example, in a medical application a particular organ may be expected to deform in a particular way when a patient breathes. Deformation fields consistent with this expected deformation would have a higher probability in the prior probability distribution than other arbitrary deformation fields.
- the conditional model 204 For a given source image and a given deformation field, the conditional model 204 generates a style transfer probability distribution of reference images.
- I S , ⁇ ) associated with the reference image I R , given the source image I S and the deformation field ⁇ , can be estimated using the conditional model 204 , as will be described hereafter with reference to two exemplary conditional models.
- I R ,I S ) can be determined in principle using Bayes' theorem in the form of Equation (1):
- the present method involves determining a deformation field that maximises the posterior probability density p( ⁇
- the determined deformation field represents a best estimate of the correspondence between the source image I S and the reference image I R . It is noted that the probability densities p(I S ) and p(I R ,I S ) are independent of the deformation field ⁇ , and hence the dependence of the posterior probability density p( ⁇
- Equation (2) log p ( ⁇
- I R ,I S ) k +log p ( ⁇ )+log p ( I R
- the update routine 206 updates the deformation field ⁇ to increase the sum of log probability densities given by Equation (2), thereby also increasing the posterior probability density given by Equation (1).
- updating the first deformation field includes determining, from the generative model 202 and the conditional model 204 , a gradient with respect to ⁇ of the log posterior probability density given by Equation (2).
- FIG. 3 shows a computing device 300 (e.g., computer or image processor) configured to implement methods in accordance with the present embodiment.
- the computing device 300 includes a power supply 304 and a system bus 304 .
- the system bus 304 is connected to: a CPU 306 ; input/output (I/O) devices 308 ; a data interface 310 ; and a memory 312 .
- the memory 312 is a non-transitory memory device and holds program code 314 , the generative model 202 , and the conditional model 204 .
- I/O devices 308 include a monitor, a keyboard, and a mouse.
- the data interface 310 is connected one or more sources of image data.
- the computing device 300 is deployed in a hospital or other medical institution and is used by medical professionals to perform deformable registration of medical images.
- computing device 300 may be used to register longitudinal images of cancer patients in order to adjust radiotherapy plans.
- computing device 300 may be used for motion correction of a series of images captured using a medical device.
- Program code 314 may be provided to the medical institution as software.
- program code for performing deformable registration may be integrated within hardware and/or software of a medical device that captures image data.
- deformable registration of medical images may be performed by a remote system or a distributed computing system.
- FIG. 4 shows a routine 400 stored within program code 314 which, when executed by the CPU 306 , causes the computing device 300 to implement the correspondence determining method 200 .
- the generative model 202 and the conditional model 204 are pre-trained prior to the routine 400 being executed, as will be described hereafter.
- the computing device 300 receives, at S 402 , first image data comprising the source image I S and second image data comprising the reference image I R via the data interface 310 .
- the first image data and the second image data are received from the same source, for example from a medical device such as a CT scanner, a PET scanner, or an MRI scanner, or from an image database.
- the first image data and the second image data are received from different sources, for example different medical devices and/or image databases.
- the computing device 300 determines, at S 404 , an initial deformation field ⁇ 0 , where ⁇ t denotes the deformation field after t iterations of an update routine, as will be described hereafter.
- the initial deformation field ⁇ 0 is determined by randomly sampling from the generative model 202 as will be described hereafter.
- an initial guess of the deformation field may be provided as ⁇ 0 , for example using an existing parametric or a non-parametric deformable registration method.
- the computing device 300 determines, at S 406 , a change in one or more characteristics of the first deformation field ⁇ 0 .
- determining the change comprises determining gradients of the log probability densities log p( ⁇ ) and log p(I R
- I R ,I S ) is determined as the sum of the determined gradients of log p( ⁇ ) and log p(I R /I S , ⁇ ), in accordance with Equation (2).
- ⁇ ⁇ t ; ⁇ t is a step size, which may be fixed or may vary with step number t; and P is a preconditioning matrix that affects the direction and the size of the change in ⁇ induced by (3).
- Different gradient methods vary from each other by using different step sizes and preconditioning matrices. For ordinary gradient ascent, P is an identity matrix.
- Other examples of gradient methods include the Broyden-Fletcher-GoldfarbShanno (BFGS) algorithm and Adam.
- the computing device 300 changes, at S 408 , characteristics of the first deformation field in accordance with the determined change.
- the optimal deformation field ⁇ * is defined as a deformation field that maximises the posterior probability density of Equation (1).
- the computing device 300 returns, at S 410 , the converged deformation field ⁇ T , which corresponds to a co-ordinate transformation representing the correspondence between the source image I S and the reference image I R .
- the converged deformation field ⁇ T is applied to the source image I S to determine a transformed source image I S ⁇ T .
- the transformed source image may then be compared with the reference image I R , either manually or by a computer program.
- a style transfer operator for translating between two imaging modalities may be applied to the transformed source image or the reference image before comparison.
- FIG. 5 shows an example of a variational autoencoder (VAE) 500 for implementing a generative model in accordance with the present embodiment.
- the VAE 500 includes an encoder network 502 and a decoder network 504 .
- the encoder network 502 is a convolutional deep neural network having convolutional layers and fully-connected layers.
- a parameter vector ⁇ e comprises connection weights between neurons in the encoder network 502 .
- the encoder network 502 is configured to receive a deformation field ⁇ as an input, and to output a distribution q ⁇ e (z
- ⁇ ) is a multivariate normal distribution with mean ⁇ ⁇ e ( ⁇ ) and covariance ⁇ ⁇ e ( ⁇ ) such that q ⁇ e (z
- ⁇ ) ( ⁇ ⁇ e ( ⁇ ), ⁇ ⁇ e ( ⁇ )), where the mean ⁇ ⁇ e ( ⁇ ) and the covariance ⁇ ⁇ e ( ⁇ ) are determined by the encoder network 504 .
- the decoder network 504 is a deconvolutional deep neural network having fully-connected layers and deconvolutional layers.
- a parameter vector ⁇ d comprises connection weights between neurons in the decoder network 504 .
- the decoder network 504 is configured to take a latent variable z as an input, and to output a deterministic function ⁇ ⁇ d (z) of the latent variable.
- a given latent variable z a given deformation field is associated with a tractable probability density p ⁇ d ( ⁇
- the prior probability density p( ⁇ ) is given by an integral ⁇ p ⁇ d ( ⁇
- the integral for determining p( ⁇ ) for a given deformation field ⁇ is generally intractable, but can be estimated with the help of the encoder network 502 , as will be described in more detail hereafter.
- z) is assumed to follow an isotropic multivariate normal distribution such that p ⁇ d ( ⁇
- z) ( ⁇ ⁇ d (z), ⁇ 2 I), where I is an identity matrix and ⁇ 2 is a hyperparameter. Therefore, the log probability density log p ⁇ d ( ⁇
- z) K ⁇ ⁇ d (z) ⁇ 2 /(2 ⁇ 2 ) for a constant K.
- Sampling z from a multivariate normal distribution p(z) (0,I), where I is an identity matrix, and passing the sampled z value through the decoder network 504 is equivalent to sampling from the prior probability distribution of deformation fields.
- the computing device 300 determines the initial deformation field ⁇ 0 by sampling from the prior distribution in this way.
- determining a change in one or more characteristics of the deformation field ⁇ t includes determining a gradient of the log prior probability density log p( ⁇ ) using the generative model 202 .
- a variational lower bound of the log prior probability density log p( ⁇ ) for a given deformation field ⁇ is given by Equation (4): log p ( ⁇ )KL[ q ⁇ e ( z
- ⁇ )] z ⁇ q ⁇ e [log p ⁇ d ( ⁇
- Equation (4) Since the KL divergence is strictly non-negative, the left hand side of Equation (4) is a lower bound to log p( ⁇ ). Furthermore, for a sufficiently well-trained encoder network 502 , the KL divergence term on the left hand side will be close to zero, and hence Equation (4) gives an approximation of the log prior probability density log p( ⁇ ).
- ⁇ ) ( ⁇ ⁇ e ( ⁇ ), ⁇ ⁇ e ( ⁇ )).
- the deformation field ⁇ t is forward propagated through the encoder network 502 , causing the encoder network 502 to output a distribution q ⁇ e (z
- One or more latent variables are sampled from the distribution q ⁇ e (z
- An estimated gradient of the variational lower bound with respect to ⁇ is determined from the determined unbiased estimates of the variational lower bounds.
- the gradient of the second term on the right hand side of Equation (4) in the present example is determined by backpropagating the estimated variational lower bound or bounds.
- ⁇ t ) is forward propagated through the decoder network 504 to determine a deterministic function ⁇ ⁇ d (z ⁇ ), from which an estimate of the variational lower bound is determined, and this estimate is used for determining an estimated gradient of the log prior probability density with respect to ⁇ .
- generative models may be implemented using other methods instead of VAEs.
- Examples of other generative models include generative adversarial networks (GANs), Gaussian process models and denoising auto-encoders.
- GANs generative adversarial networks
- Gaussian process models denoising auto-encoders.
- determining a change in one or more characteristics of the deformation field ⁇ t further includes determining a gradient of the log style transfer probability density log p(I R /I S , ⁇ ) using the conditional model 204 .
- the style transfer probability density is modelled as random noise accounting for image noise and residual changes between the reference image and the transformed source image (for example, residual changes in topology).
- the random noise is assumed to be Gaussian random noise, and hence the conditional model generates a multivariate normal distribution such that p(I R
- I S , ⁇ ) (I S ⁇ , ⁇ r 2 I), where the variance ⁇ r 2 is determined from the image data and I is an identity matrix.
- I S , ⁇ ) K r ⁇ I S ⁇ I R ⁇ 2 /(2 ⁇ r 2 ) for a constant K r .
- K r K ⁇ I S ⁇ I R ⁇ 2 /(2 ⁇ r 2 ) for a constant K r .
- the gradient of the log style transfer probability density with respect to the deformation field ⁇ is determined using this equation.
- the means by which the variance a is determined from the image data depends on the modality of the source image and the reference image.
- the variance ⁇ r 2 is matched to the signal to noise ratio (SNR) of the device or devices capturing the images. This may be included, for example, in metadata contained within the image data comprising the source image and/or the reference image. In other examples, the variance ⁇ r 2 is not known a priori and is instead learned from image data as part of a joint optimisation routine.
- SNR signal to noise ratio
- the Gaussian noise model described above may be suitable, for example, for monomodal registration of MRI images, which are known to exhibit Gaussian noise.
- the random noise is assumed to take other forms, for example Poisson noise in the case of a CT scan.
- a style transfer operator may be determined for translating the modality of the source image to the modality of the reference image.
- known methods can be used to determine a style transfer operator for translating an image between modalities associated with the different imaging techniques/parameters.
- a known style transfer operator is applied to the source image or the reference image, and the registration process proceeds as monomodal registration.
- FIG. 6 shows an example of a conditional variational autoencoder (CVAE) 600 for generating a conditional model in accordance with the present embodiment.
- the CVAE 600 includes an encoder network 602 and a decoder network 604 .
- the encoder network 602 is a convolutional deep neural network having a series of convolutional layers connected to a series of fully-connected layers.
- a parameter vector ⁇ e comprises connection weights between neurons in the encoder network 602 .
- the encoder network 602 is configured to receive a reference image I R and a transformed source image I S ⁇ as inputs, and to output a distribution ⁇ tilde over (q) ⁇ ⁇ e (z
- I R ,I S ⁇ ) is a multivariate normal distribution with mean ⁇ tilde over ( ⁇ ) ⁇ ⁇ e (I R ,I S ⁇ ) and covariance ⁇ ⁇ e (I R ,I S ⁇ ) such that ⁇ tilde over (q) ⁇ ⁇ e (z
- I R ,I S ⁇ ) ( ⁇ tilde over ( ⁇ ) ⁇ ⁇ e (I R ,I S ⁇ ), ⁇ tilde over ( ⁇ ) ⁇ ⁇ e (I R , I S ⁇ )), where the mean ⁇ tilde over ( ⁇ ) ⁇ ⁇ e (I R ,I S ⁇ ) and the covariance ⁇ tilde over ( ⁇ ) ⁇ ⁇ e (I R ,I S ⁇ ) are determined by the encoder network 604 .
- the decoder network 604 is a deconvolutional deep neural network having fully-connected layers and deconvolutional layers.
- a parameter vector ⁇ d comprises connection weights between neurons in the decoder network 504 .
- the decoder network 604 is configured to take a latent variable z, the transformed source image I S ⁇ , and the deformation field ⁇ as an input and to output a deterministic function ⁇ tilde over ( ⁇ ) ⁇ ⁇ d (z,I S ⁇ ) of the latent variable and the transformed source image.
- a given latent variable z a given reference image is associated with a tractable probability density p ⁇ d (I R
- z,I S ⁇ ) is assumed to follow an isotropic multivariate normal distribution such that p ⁇ d (I R
- z,I S ⁇ ) ( ⁇ tilde over ( ⁇ ) ⁇ ⁇ d (z,I S ⁇ ), ⁇ tilde over ( ⁇ ) ⁇ 2 I), where I is an identity matrix and ⁇ tilde over ( ⁇ ) ⁇ 2 is a hyperparameter.
- z,I S , ⁇ ) depends on the squared Euclidian distance between the reference image I R and the deterministic function ⁇ tilde over ( ⁇ ) ⁇ ⁇ d (z,I S ⁇ ), for example according to a relation log p ⁇ d ( ⁇
- z) ⁇ tilde over (K) ⁇ I R ⁇ tilde over ( ⁇ ) ⁇ ⁇ d (z,I S ⁇ ) ⁇ 2 /(2 ⁇ tilde over ( ⁇ ) ⁇ 2 ) for a constant ⁇ tilde over (K) ⁇ .
- Sampling z from a multivariate normal distribution p(z) (0,I), where I is an identity matrix, and passing the sampled z value through the decoder network 604 along with the source image I S and the deformation field ⁇ is equivalent to sampling from the style transfer probability distribution of reference images, given the source image I S and the deformation field ⁇ .
- determining a change in one or more characteristics of the deformation field ⁇ t includes determining a gradient of the log style transfer probability density log p(I R
- a variational lower bound of the style transfer prior probability density log p(I R /I S , ⁇ ) is given by Equation (5): log p ( I R
- I R ,I S ⁇ )] z ⁇ tilde over (q) ⁇ ⁇ e [log p ⁇ d ( I R
- Equation (5) Since the KL divergence is strictly nonnegative, the left hand side of Equation (5) is a lower bound to log p(I R /I S , ⁇ ). Furthermore, for a sufficiently well-trained encoder network 602 , the KL divergence term on the left hand side will be close to zero, and hence Equation (5) gives an approximation of the style transfer probability density log p(I R /I S , ⁇ ).
- I R ,I S ⁇ ) ( ⁇ tilde over ( ⁇ ) ⁇ ⁇ e (I R ,I S ⁇ ), ⁇ tilde over ( ⁇ ) ⁇ ⁇ e (I R ,I S ⁇ ).
- the reference image I R and the transformed source image I S ⁇ t are forward propagated through the encoder network 602 , causing the encoder network 602 to output a distribution ⁇ tilde over (q) ⁇ ⁇ e (z
- One or more latent variables are sampled from the distribution ⁇ tilde over (q) ⁇ ⁇ e (z
- An estimated gradient of the variational lower bound with respect to I S ⁇ and hence an estimated gradient of the log style transfer probability density with respect to I S ⁇ , is determined from the determined unbiased estimates of the variational lower bounds.
- the gradient of the second term on the right hand side of Equation (4) in the present example is determined by backpropagating the estimated variational lower bound or bounds through the CVAE 600 . Since the transformed source image I S ⁇ is a differentiable function of the deformation field ⁇ , the gradient of the variational lower bound with respect to the deformation field is readily determined from the gradient of the variational lower bound with respect to the transformed source image (this may, for example, involve further backpropagation of the variational lower bound).
- I R ,I S ⁇ ) is forward propagated through the decoder network 604 to determine a deterministic function ⁇ tilde over ( ⁇ ) ⁇ ⁇ d (z ⁇ ,I S ⁇ ), from which an estimate of the variational lower bound is determined, and this estimate is used for determining an estimated gradient of the log style transfer probability density with respect to ⁇ .
- the ordering of the inputs and conditioning in the CVAE is reversed from that shown in FIG. 6 .
- the encoder network takes the deformation field as an input, and both the encoder and decoder network are conditioned on the source image and the reference image.
- Other arrangements are envisaged.
- the generative model 202 and the conditional model 204 are pre-trained prior to the correspondence determining method 200 being implemented.
- synthetic deformation fields are used to pretrain the VAE 500 .
- the synthetic deformation fields are chosen to cover, as best as possible, the expected deformations of a target object of interest. Synthetic deformation fields may be generated, for example, using a parametric model (such as a spline model or a mesh-based model) of a target object (such as an organ in a medical application).
- examples of the present embodiment further include refining the generative model using real data, as will be described in detail hereafter, such that even for a small and/or inaccurate initial set of synthetic deformation fields, the prior distribution will improve as more real image data is received and eventually yield accurate results.
- Pre-training the VAE 500 using synthetic deformation fields is performed using stochastic gradient ascent, in which each synthetic deformation field is forward propagated through the VAE 500 in order to determine one or more (unbiased) estimated variational lower bounds as given by Equation (4) for each synthetic deformation field.
- Each estimated variational lower bound is backpropagated through the VAE 500 to determine a gradient of the estimated variational lower bound with respect to the parameters ⁇ e , ⁇ d .
- Gradient ascent is performed with respect to each estimated variational lower bound, and performing this for a large number of estimated variational bounds and a large number of synthetic deformation fields results in stochastic optimisation of the variational lower bound, and thus causes the decoder network 502 of the VAE 500 to approximate the distribution of synthetic deformation fields, which is then used to determine the prior probability density p( ⁇ ) for a given deformation field.
- the conditional model 204 is implemented using the CVAE 600
- the CVAE 600 is pre-trained using pairs of registered images having different modalities.
- a generative model corresponding to a prior probability distribution of deformation fields is refined using the results of the deformable registration process. In this way, the generative model more accurately approximates the prior distribution of deformation fields as more image pairs are registered.
- FIG. 7 shows a correspondence determining method 700 , which is an example of the correspondence determining method 200 described above, and employs, in addition to the generative model 202 , the conditional model 204 , and the deformation field update routine 206 , a parameter update routine 208 for updating parameters of the generative model 202 .
- the image data includes a set of pairs (I S (i) ,I R (i) ) of source images and reference images for such that each reference image in the set is associated with one source image in the set.
- a converged deformation field, ⁇ T (i) is determined, for example using the routine 400 , and the parameter update routine updates parameters of the generative model 202 using the determined converged deformation field ⁇ T (i) .
- One such method involves, for each converged deformation field ⁇ T (i) , updating parameters of the generative model 202 , to optimise the prior probability density p( ⁇ T (i) ) with respect to the parameters of the generative model 202 .
- FIG. 8 shows an exemplary routine 800 , which, when executed by a computing device, causes the computing device to perform the correspondence determining method 700 .
- the computing device receives, at S 802 , the image data comprising the set of pairs (I S (i) ,I R (i) ) of source images and reference images. For each image pair, the computing device determines, at S 804 , a respective initial deformation field ⁇ 0 (i) . As described above, determining an initial deformation field may involve sampling from the generative model 202 , or may include receiving an initial guess of the deformation field.
- the computing device determines, at S 806 , a change in one or more characteristics of the respective deformation field ⁇ 0 (i) .
- the change in characteristics is determined using an equivalent process to that described above with reference to FIG. 4 .
- the computing device changes, at S 808 , characteristics of the respective deformation field in accordance with the determined change.
- the converged deformation field ⁇ T (i) represents a first estimate of the correspondence between the source image I S (i) and the reference image I R (i) .
- the computing device passes, at S 810 , each converged deformation field ⁇ T (i) through the generative model 202 , and updates, at S 812 , parameters of the generative model 202 , to optimise the prior probability density with the respect to the parameters of the generative model.
- parameters of the VAE 500 are updated to optimise the variational lower bound of the log prior probability density given by Equation (4) above.
- the converged deformation field ⁇ T (i) is forward propagated through the encoder network 502 , causing the encoder network 502 to output a distribution q ⁇ e (z
- One or more latent variables are sampled from the distribution q ⁇ e (z
- An estimated gradient of the variational lower bound with respect to ⁇ e , ⁇ d , and hence an estimated gradient of the log prior probability density log p( ⁇ T (i) ) with respect to ⁇ e , ⁇ d is determined by backpropagating the estimated variational lower bound or bounds through the VAE 500 .
- a gradient-based update rule is applied to update the parameters ⁇ e , ⁇ d .
- the variational lower bound is optimised with respect to the parameters ⁇ e , ⁇ d , and hence the prior probability density p( ⁇ T (i) ) is optimised (indirectly) with respect to the parameters ⁇ e , ⁇ d .
- ⁇ T (i) ) is forward propagated through the decoder network 504 to determine a deterministic function ⁇ ⁇ d (z ⁇ ), from which an estimate of the variational lower bound is determined.
- the estimated variational bound is then backpropagated to determine the approximate gradient of log p( ⁇ T (i) ) with respect to ⁇ e , ⁇ d and a gradient-based update rule is used to update the parameters ⁇ e , ⁇ d based on this approximate gradient.
- the routine 800 Having updated the parameters of the generative model 202 using each first estimate ⁇ T (i) corresponding to each image pair, the routine 800 returns to S 804 .
- S 804 -S 812 are performed iteratively.
- the computing device determines, for each image pair, a new estimate of the deformation field, and uses the new estimate to update the parameters of the generative model. This iterative process continues until predefined convergence conditions are satisfied.
- the predefined convergence conditions may be based on the estimated deformation fields, the parameters of the generative model, or a combination of both. Suitable convergence criteria for gradient-based optimisation methods will be well-known to those skilled in the art.
- the computing device returns, at S 814 , the converged deformation fields, which each encodes co-ordinate transformation representing a correspondence between each source images I S (i) and corresponding reference image I R (i) .
- the same computing device is used to register pairs of images and to update parameters of the generative model 202 using the registered pairs.
- the computing device updates parameters in parallel with registering images.
- updating the parameters of the generative model 202 is performed by a separate computing device.
- a deployed computing device/system may perform deformable registration on a set of images, and send the resulting deformation fields to a training device/system, which uses the images to determine updated parameters for the deployed computing device/system. This arrangement may be particularly relevant for medical applications, for reason explained hereafter.
- a similar iterative process to that described above for updating the parameters of the generative model 202 is also applied to update parameters of the conditional model 204 .
- each converged deformation field is passed through the conditional model 204 , and the parameters of the conditional model are updated to optimise the style transfer probability density with respect to the parameters of the conditional model 204 .
- updating the parameters of the conditional model 204 is performed after each iteration of updating the deformation field and the parameters of the generative model 202 .
- the parameters and the deformation field are updated in a different order.
- a single joint optimisation step is performed in which the deformation field, the parameters of the generative model 202 , and the parameters of the conditional model 204 , are updated together to optimise the posterior probability density p( ⁇ T (i)
- Some examples of the present embodiments include alternative or further steps to refine the generative model and/or the conditional model, or to otherwise improve the accuracy of the determined correspondence.
- a source image I S is registered to a reference image I R using the method described herein, yielding a first converged deformation field ⁇ .
- the same reference image I R is registered to the same source image I S , yielding a second converged deformation field g.
- the second deformation field g should be the inverse of the first deformation field ⁇ .
- an error associated with the composition ⁇ g, and an identity field Id (corresponding to an identity co-ordinate transformation) is backpropagated through the generative model and/or the conditional model, and a gradient-based optimisation method is then used to update parameters of the generative model and/or the conditional model such that the error is minimised.
- the error associated with the composition ⁇ g and an identity field Id is given by an L 2 norm ⁇ g ⁇ Id ⁇ L 2 .
- backpropagating a loss given by ⁇ v ⁇ +v g ⁇ L 2 through the generative model and/or the conditional model, and updating the parameters of the generative model and/or the conditional model to minimise the loss has an inertial effect that encourages the method to generate diffeomorphic deformation fields.
- Updating parameters of the generative model and/or the conditional model using image data as described above results in the prior probability distribution and/or the conditional probability distribution more accurately reflecting the data as the method is performed.
- the models may become highly accurate even in cases where the generative model and/or the conditional model were poorly pretrained, for example due to a lack of ground-truth data or sufficiently accurate synthetic data.
- devices are subject to regulatory approval and may only be deployed after such approval is granted.
- the performance of a device implementing the deformable registration process would be tested for a fixed set of model parameters, and that updating these model parameters in a deployed device (or equivalently, updating model parameters in a remote system in the case of a network-based implementation) may result in the regulatory approval no longer being valid. It is therefore suggested that parameters of a deployed system/device would be frozen, such that the deployed system/device performs image registration without updating the parameters.
- Image data sent to the deployed system/device would simultaneously be sent to a secondary system/device that “mirrors” the deployed system/device.
- Parameters of the secondary system/device would be updated using the methods described above, and hence the performance of the secondary system/device would be expected to improve over time, but the output of the secondary device would never be used for the intended purpose of the deployed device (for example, making clinical decisions in the case of a medical application).
- regulatory approval would be sought for the secondary system/device, and if granted, the updated parameters of the secondary system/device would be imported to the deployed system/device. This process of updating and seeking regulatory approval could be performed periodically, for example.
- the present embodiment provides an accurate and widely-applicable method of performing deformable registration.
- Some examples of the present embodiment include an additional process in which supervised learning model is used to train a one-shot model, such that the one-shot model may subsequently be used to determine approximations of converged deformation fields resulting from the variational method described above.
- the one-shot model is a regression network.
- other one-shot models may be used, for example support vector machines (SVMs).
- SVMs support vector machines
- a one-shot method takes a source image and a reference image as inputs, and outputs an approximation of the converged deformation field that would result from the application of the variational method described above to the same source image and reference image.
- Determining an approximation of a converged deformation field using the one-shot model may result in a significant time saving compared with determining a converged deformation field using the variational method, and if one-shot model is sufficiently well-trained, may suffer only a very slight loss of accuracy compared with the variational method.
- the regression network may be any suitable neural network, for example a convolutional neural network (CNN) configured for regression.
- CNN convolutional neural network
- a method of training the regression network includes forward propagating a source image and a reference image through the network, to determine a first approximation of the converged deformation field corresponding to the source image and the reference image. The method continues by determining an error associated with the first approximation and the converged deformation field determined using the variational method.
- the error is a mean-squared difference between the first approximation and the converged deformation field, but other suitable metrics for quantifying a difference between two deformation fields may be used in other examples.
- the determined error is backpropagated through the regression network, to determine gradients of the determined error with respect to parameters of the regression network.
- the parameters of the regression network are updated using a gradient-based update rule, whereby to reduce the error associated with the first approximation and the converged deformation field.
- the parameters are iteratively updated in this way until predetermined convergence criteria are satisfied.
- the training process is repeated for a large number of source images, reference images, and corresponding converged deformation fields, such that the regression network learns how to determine deformation fields for unregistered pairs of source images and reference images.
- the one-shot model may be trained in parallel to the updating of parameters of the generative and/or conditional models used in the variational method.
- partially-converged deformation field as opposed to, or as well as, converged deformation fields, may be used to train the one-shot model.
- reinforcement learning may be used to train an autonomous agent to determine approximations of converged deformation fields resulting from the variational method.
- an approximation of a deformation field is associated with a policy of the autonomous agent.
- the policy is updated according to a predetermined reward structure, such that higher (conventionally, though not necessarily, more positive) rewards are associated with actions that reduce an error associated with the approximation and the converged deformation field resulting from the variational method.
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Data Mining & Analysis (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Artificial Intelligence (AREA)
- Mathematical Physics (AREA)
- Evolutionary Computation (AREA)
- General Engineering & Computer Science (AREA)
- Computing Systems (AREA)
- Life Sciences & Earth Sciences (AREA)
- Evolutionary Biology (AREA)
- Bioinformatics & Computational Biology (AREA)
- Bioinformatics & Cheminformatics (AREA)
- General Health & Medical Sciences (AREA)
- Databases & Information Systems (AREA)
- Health & Medical Sciences (AREA)
- Medical Informatics (AREA)
- Probability & Statistics with Applications (AREA)
- Mathematical Analysis (AREA)
- Pure & Applied Mathematics (AREA)
- Computational Linguistics (AREA)
- Mathematical Optimization (AREA)
- Computational Mathematics (AREA)
- Multimedia (AREA)
- Biomedical Technology (AREA)
- Molecular Biology (AREA)
- Operations Research (AREA)
- Biophysics (AREA)
- Algebra (AREA)
- Image Analysis (AREA)
- Image Processing (AREA)
Abstract
Description
log p(ϕ|I R ,I S)=k+log p(ϕ)+log p(I R |I S,ϕ), (2)
where k is independent of the deformation field ϕ. Since the logarithm is a strictly increasing function, it follows that maximising the sum of the log probability densities on the right hand side of Equation (2) is equivalent to maximising the posterior probability density of Equation (1).
ϕt+1=ϕt+γt P −1 g t, (3)
in which: gt=∇ϕlog p(ϕ|IR,IS)|ϕ=ϕt; γt is a step size, which may be fixed or may vary with step number t; and P is a preconditioning matrix that affects the direction and the size of the change in ϕ induced by (3). Different gradient methods vary from each other by using different step sizes and preconditioning matrices. For ordinary gradient ascent, P is an identity matrix. Other examples of gradient methods include the Broyden-Fletcher-GoldfarbShanno (BFGS) algorithm and Adam.
log p(ϕ)KL[q θ
where KL denotes the Kullback-Leibler divergence between two distributions. Since the KL divergence is strictly non-negative, the left hand side of Equation (4) is a lower bound to log p(ϕ). Furthermore, for a sufficiently well-trained
log p(I R |I S,ϕ)−KL[{tilde over (q)} ρ
Claims (18)
Applications Claiming Priority (3)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
EP18186293.9 | 2018-07-30 | ||
EP18186293.9A EP3605465B1 (en) | 2018-07-30 | 2018-07-30 | A method for determining a correspondence between a source image and a reference image |
EP18186293 | 2018-07-30 |
Publications (2)
Publication Number | Publication Date |
---|---|
US20200034654A1 US20200034654A1 (en) | 2020-01-30 |
US10909416B2 true US10909416B2 (en) | 2021-02-02 |
Family
ID=63103817
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US16/440,215 Active 2039-07-05 US10909416B2 (en) | 2018-07-30 | 2019-06-13 | Deep variational method for deformable image registration |
Country Status (2)
Country | Link |
---|---|
US (1) | US10909416B2 (en) |
EP (1) | EP3605465B1 (en) |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20200293497A1 (en) * | 2019-03-13 | 2020-09-17 | Deepmind Technologies Limited | Compressed sensing using neural networks |
Families Citing this family (15)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US10909671B2 (en) * | 2018-10-02 | 2021-02-02 | International Business Machines Corporation | Region of interest weighted anomaly detection |
US11126915B2 (en) * | 2018-10-15 | 2021-09-21 | Sony Corporation | Information processing apparatus and information processing method for volume data visualization |
CN111127304B (en) * | 2018-10-31 | 2024-02-20 | 微软技术许可有限责任公司 | Cross-domain image conversion |
US11158069B2 (en) * | 2018-12-11 | 2021-10-26 | Siemens Healthcare Gmbh | Unsupervised deformable registration for multi-modal images |
US11388416B2 (en) * | 2019-03-21 | 2022-07-12 | Qualcomm Incorporated | Video compression using deep generative models |
JP7426261B2 (en) * | 2020-03-10 | 2024-02-01 | 株式会社Screenホールディングス | Learning device, image inspection device, learned parameters, learning method, and image inspection method |
US20210304457A1 (en) * | 2020-03-31 | 2021-09-30 | The Regents Of The University Of California | Using neural networks to estimate motion vectors for motion corrected pet image reconstruction |
WO2022146727A1 (en) * | 2020-12-29 | 2022-07-07 | Snap Inc. | Generative adversarial network manipulated image effects |
US11816793B2 (en) | 2021-01-06 | 2023-11-14 | Eagle Technology, Llc | Geospatial modeling system providing 3D geospatial model update based upon iterative predictive image registration and related methods |
US11636649B2 (en) * | 2021-01-06 | 2023-04-25 | Eagle Technology, Llc | Geospatial modeling system providing 3D geospatial model update based upon predictively registered image and related methods |
CN112991406B (en) * | 2021-02-07 | 2023-05-23 | 清华大学深圳国际研究生院 | Method for constructing brain map based on differential geometry technology |
CN113592927B (en) * | 2021-07-26 | 2023-12-15 | 国网安徽省电力有限公司电力科学研究院 | Cross-domain image geometric registration method guided by structural information |
CN113643339B (en) * | 2021-08-13 | 2024-02-02 | 上海应用技术大学 | Near infrared and visible light remote sensing image registration method based on reinforcement learning |
CN114359356A (en) * | 2021-12-28 | 2022-04-15 | 上海联影智能医疗科技有限公司 | Training method of image registration model, image registration method, device and medium |
CN116433730B (en) * | 2023-06-15 | 2023-08-29 | 南昌航空大学 | Image registration method combining deformable convolution and modal conversion |
Citations (14)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US8600131B2 (en) * | 2008-07-07 | 2013-12-03 | The Johns Hopkins University | Advanced cost functions for image registration for automated image analysis: multi-channel, hypertemplate and atlas with built-in variability |
US8972253B2 (en) * | 2010-09-15 | 2015-03-03 | Microsoft Technology Licensing, Llc | Deep belief network for large vocabulary continuous speech recognition |
US9031844B2 (en) * | 2010-09-21 | 2015-05-12 | Microsoft Technology Licensing, Llc | Full-sequence training of deep structures for speech recognition |
US9569843B1 (en) | 2015-09-09 | 2017-02-14 | Siemens Healthcare Gmbh | Parameter-free denoising of complex MR images by iterative multi-wavelet thresholding |
US20170372193A1 (en) | 2016-06-23 | 2017-12-28 | Siemens Healthcare Gmbh | Image Correction Using A Deep Generative Machine-Learning Model |
US9858689B1 (en) | 2016-09-15 | 2018-01-02 | Siemens Healthcare Gmbh | Fast and memory efficient redundant wavelet regularization with sequential cycle spinning |
US10133964B2 (en) | 2017-03-28 | 2018-11-20 | Siemens Healthcare Gmbh | Magnetic resonance image reconstruction system and method |
US10529318B2 (en) * | 2015-07-31 | 2020-01-07 | International Business Machines Corporation | Implementing a classification model for recognition processing |
US10540798B1 (en) * | 2019-01-10 | 2020-01-21 | Capital One Services, Llc | Methods and arrangements to create images |
US10595727B2 (en) * | 2018-01-25 | 2020-03-24 | Siemens Healthcare Gmbh | Machine learning-based segmentation for cardiac medical imaging |
US10624558B2 (en) * | 2017-08-10 | 2020-04-21 | Siemens Healthcare Gmbh | Protocol independent image processing with adversarial networks |
US10650492B2 (en) * | 2017-09-08 | 2020-05-12 | Baidu Online Network Technology (Beijing) Co., Ltd. | Method and apparatus for generating image |
US10733788B2 (en) * | 2018-03-15 | 2020-08-04 | Siemens Healthcare Gmbh | Deep reinforcement learning for recursive segmentation |
US10753997B2 (en) * | 2017-08-10 | 2020-08-25 | Siemens Healthcare Gmbh | Image standardization using generative adversarial networks |
-
2018
- 2018-07-30 EP EP18186293.9A patent/EP3605465B1/en active Active
-
2019
- 2019-06-13 US US16/440,215 patent/US10909416B2/en active Active
Patent Citations (14)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US8600131B2 (en) * | 2008-07-07 | 2013-12-03 | The Johns Hopkins University | Advanced cost functions for image registration for automated image analysis: multi-channel, hypertemplate and atlas with built-in variability |
US8972253B2 (en) * | 2010-09-15 | 2015-03-03 | Microsoft Technology Licensing, Llc | Deep belief network for large vocabulary continuous speech recognition |
US9031844B2 (en) * | 2010-09-21 | 2015-05-12 | Microsoft Technology Licensing, Llc | Full-sequence training of deep structures for speech recognition |
US10529318B2 (en) * | 2015-07-31 | 2020-01-07 | International Business Machines Corporation | Implementing a classification model for recognition processing |
US9569843B1 (en) | 2015-09-09 | 2017-02-14 | Siemens Healthcare Gmbh | Parameter-free denoising of complex MR images by iterative multi-wavelet thresholding |
US20170372193A1 (en) | 2016-06-23 | 2017-12-28 | Siemens Healthcare Gmbh | Image Correction Using A Deep Generative Machine-Learning Model |
US9858689B1 (en) | 2016-09-15 | 2018-01-02 | Siemens Healthcare Gmbh | Fast and memory efficient redundant wavelet regularization with sequential cycle spinning |
US10133964B2 (en) | 2017-03-28 | 2018-11-20 | Siemens Healthcare Gmbh | Magnetic resonance image reconstruction system and method |
US10624558B2 (en) * | 2017-08-10 | 2020-04-21 | Siemens Healthcare Gmbh | Protocol independent image processing with adversarial networks |
US10753997B2 (en) * | 2017-08-10 | 2020-08-25 | Siemens Healthcare Gmbh | Image standardization using generative adversarial networks |
US10650492B2 (en) * | 2017-09-08 | 2020-05-12 | Baidu Online Network Technology (Beijing) Co., Ltd. | Method and apparatus for generating image |
US10595727B2 (en) * | 2018-01-25 | 2020-03-24 | Siemens Healthcare Gmbh | Machine learning-based segmentation for cardiac medical imaging |
US10733788B2 (en) * | 2018-03-15 | 2020-08-04 | Siemens Healthcare Gmbh | Deep reinforcement learning for recursive segmentation |
US10540798B1 (en) * | 2019-01-10 | 2020-01-21 | Capital One Services, Llc | Methods and arrangements to create images |
Non-Patent Citations (28)
Title |
---|
A. V. Dalca, G. Balakrishnan, J. Guttag, and M. Sabuncu, "Unsupervised learning for fast probabilistic diffeomorphic registration," in International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, 2018, pp. 729-738. |
B. D. de Vos, F. F. Berendsen, M. A. Viergever, M. Staring, and I. Isgum, "End-to-end unsupervised deformable image registration with a convolutional neural network," in Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support. Springer, 2017, pp. 204-212. |
Bhushan Manav et al: "Motion Correction and Parameter Estimation in dceMRI Sequences: Application to Colorectal Cancer", Sep. 18, 2011 (Sep. 18, 2011), International Conference on Simulation, Modeling, and Programming for Autonomous Robots,SIMPAR 2010; [Lecture Notes in Computer Science; Lect.Notes Computer], Springer, Berlin, Heidelberg, pp. 476-483. |
Bora, Ashish, et al. "Compressed Sensing using Generative Models." arXiv preprint arXiv:1703.03208 (2017). |
C. Tanner, F. Ozdemir, R. Profanter, V. Vishnevsky, E. Konukoglu, andO. Goksel, "Generative adversarial networks for MR-CT deformableimage registration," arXiv preprint arXiv:1807.07349, 2018. |
Chang, S. Grace, Bin Yu, and Martin Vetterli. "Adaptive wavelet thresholding for image denoising and compression." IEEE Transactions on image processing 9.9 (2000): 1532-1546. |
D. Mahapatra, B. Antony, S. Sedai, and R. Garnavi, "Deformablemedical image registration using generative adversarial networks," in Biomedical Imaging (ISBI 2018), 2018 IEEE 15th International Symposium on. IEEE, 2018, pp. 1449-1453. |
Dalca, Adrian V., et al. "Unsupervised learning for fast probabilistic diffeomorphic registration.", 2018, arXiv, pp. 1-10 (Year: 2018). * |
Extended European Search Report (EESR) dated Apr. 12, 2019 in corresponding European Patent Application No. 18186293.9. |
G. Balakrishnan, A. Zhao, M. R. Sabuncu, J. Guttag, and A. V.Dalca, "An unsupervised learning model for deformable medical image registration," in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp. 9252-9260. |
H. Sokooti, B. de Vos et al., "Nonrigid image registration using multiscale 3D convolutional neural networks," in International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, 2017, pp. 232-239. |
H. Uzunova, M. Wilms, H. Handels, and J. Ehrhardt, "Training cnns for image registration from few samples with model-based data augmentation," in International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, 2017, pp. 223-231. |
J. Fan, X. Cao, P.-T. Yap, and D. Shen, "Birnet: Brain image registration using dual-supervised fully convolutional networks," arXiv preprint arXiv:1802.04692, 2018. |
J. Krebs, T. Mansi, H. Delingette et al., "Robust non-rigid registration through agent-based action learning," in International Conferenceon Medical Image Computing and Computer-Assisted Intervention.Springer, 2017, pp. 344-352. |
Jain, Viren, et al. "Supervised learning of image restoration with convolutional networks." Computer Vision, 2007. ICCV 2007. IEEE 11th International Conference on. IEEE, 2007. |
Jason, J. Yu, Adam W. Harley, and Konstantinos G. Derpanis. "Back to basics: Unsupervised learning of optical flow via brightness constancy and motion smoothness." In European Conference on Computer Vision, pp. 3-10. Springer, Cham, 2016. |
K. A. Eppenhof, M. W. Lafarge, P. Moeskops, M. Veta, and J. P. Pluim,"Deformable image registration using convolutional neural networks," in Medical Imaging 2018: Image Processing, vol. 10574. International Society for Optics and Photonics, 2018, p. 105740S. |
Krebs, Julian, et al. "Learning Structured Deformations using Diffeomorphic Registration." arXiv preprint arXiv:1804.07172 (Sep. 2018). |
Krebs, Julian, et al. "Unsupervised probabilistic deformation modeling for robust diffeomorphic registration.", 2018, arXiv, pp. 1-8 (Year: 2018). * |
Lexing Xie, et al. "Image Restoration" LectureColumbia University of New York; Mar. 23, 2009. |
M. Jaderberg, K. Simonyan, A. Zisserman et al., "Spatial transformernetworks," in Advances in neural information processing systems, 2015, pp. 2017-2025. |
M.-M. Rohe, M. Datar, T. Heimann, M. Sermesant, and X. Pennec, "SVF-net: Learning deformable image registration using shape matching," in International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, 2017, pp. 266-274. |
Salakhutdinov, Ruslan. Learning deep generative models. Diss. University of Toronto, 2009. |
Venkatakrishnan, Singanallur V., Charles A. Bouman, and Brendt Wohlberg. "Plug-and-play priors for model based reconstruction." Global Conference on Signal and Information Processing (GlobalSIP), 2013 IEEE. IEEE, 2013. |
X. Liang, L. Lee, W. Dai et al., "Dual motion GAN for future-flow embedded video prediction," in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 1744-1752. |
X. Yang, R. Kwitt, M. Styner, and M. Niethammer, "Quicksilver: Fastpredictive image registration-a deep learning approach," NeuroImage, vol. 158, pp. 378-396, 2017. |
X. Yang, R. Kwitt, M. Styner, and M. Niethammer, "Quicksilver: Fastpredictive image registration—a deep learning approach," NeuroImage, vol. 158, pp. 378-396, 2017. |
Y. Hu, M. Modat, E. Gibson, W. Li, N. Ghavami, E. Bonmati, G. Wang,S. Bandula, C. M. Moore, M. Emberton et al., "Weakly-supervised convolutional neural networks for multimodal image registration," Medical Image Analysis, 2018. |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20200293497A1 (en) * | 2019-03-13 | 2020-09-17 | Deepmind Technologies Limited | Compressed sensing using neural networks |
Also Published As
Publication number | Publication date |
---|---|
EP3605465A1 (en) | 2020-02-05 |
US20200034654A1 (en) | 2020-01-30 |
EP3605465B1 (en) | 2020-12-30 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US10909416B2 (en) | Deep variational method for deformable image registration | |
US11257259B2 (en) | Topogram prediction from surface data in medical imaging | |
US11449759B2 (en) | Medical imaging diffeomorphic registration based on machine learning | |
Würfl et al. | Deep learning computed tomography: Learning projection-domain weights from image domain in limited angle problems | |
US9892361B2 (en) | Method and system for cross-domain synthesis of medical images using contextual deep network | |
US20190197662A1 (en) | Registration method and apparatus | |
US9299145B2 (en) | Image segmentation techniques | |
CN110363797B (en) | PET and CT image registration method based on excessive deformation inhibition | |
US10134142B2 (en) | Optimization of parameters for segmenting an image | |
US20230046321A1 (en) | Medical image analysis using machine learning and an anatomical vector | |
EP4148745A1 (en) | Systems and methods for image evaluation | |
Qiao et al. | An efficient preconditioner for stochastic gradient descent optimization of image registration | |
Bône et al. | Learning the spatiotemporal variability in longitudinal shape data sets | |
US20210383565A1 (en) | Training a machine learning algorithm using digitally reconstructed radiographs | |
US20210330274A1 (en) | Computer-implemented method, computer program, systems and x-ray facility for correction of x-ray image data with regard to noise effects | |
US11861846B2 (en) | Correcting segmentation of medical images using a statistical analysis of historic corrections | |
CN114511642A (en) | Method and system for predicting virtual anchor sheet flow | |
US20230177706A1 (en) | Multi-layer image registration | |
EP4053752A1 (en) | Cnn-based image processing | |
US20230298136A1 (en) | Deep learning multi-planar reformatting of medical images | |
CN112069725B (en) | High-precision slice acquisition method and device for 3D printer | |
US20230060113A1 (en) | Editing presegmented images and volumes using deep learning | |
EP4386669A1 (en) | Motion correction of image data | |
US20220270256A1 (en) | Compensation of organ deformation for medical image registration | |
Jeong | Estimation of probability distribution on multiple anatomical objects and evaluation of statistical shape models |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: SIEMENS MEDICAL SOLUTIONS USA, INC., PENNSYLVANIA Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:MANSI, TOMMASO;MAILHE, BORIS;LIAO, RUI;AND OTHERS;SIGNING DATES FROM 20190607 TO 20190613;REEL/FRAME:049461/0053 |
|
FEPP | Fee payment procedure |
Free format text: ENTITY STATUS SET TO UNDISCOUNTED (ORIGINAL EVENT CODE: BIG.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY |
|
AS | Assignment |
Owner name: SIEMENS HEALTHCARE GMBH, GERMANY Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:SIEMENS MEDICAL SOLUTIONS USA, INC.;REEL/FRAME:049683/0114 Effective date: 20190613 |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NON FINAL ACTION MAILED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: RESPONSE TO NON-FINAL OFFICE ACTION ENTERED AND FORWARDED TO EXAMINER |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NOTICE OF ALLOWANCE MAILED -- APPLICATION RECEIVED IN OFFICE OF PUBLICATIONS |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: PUBLICATIONS -- ISSUE FEE PAYMENT VERIFIED |
|
STCF | Information on status: patent grant |
Free format text: PATENTED CASE |
|
AS | Assignment |
Owner name: SIEMENS HEALTHINEERS AG, GERMANY Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:SIEMENS HEALTHCARE GMBH;REEL/FRAME:066267/0346 Effective date: 20231219 |