Classifications

• • G06T3/153—
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T7/00—Image analysis
  • G06T7/30—Determination of transform parameters for the alignment of images, i.e. image registration
  • G06T7/33—Determination of transform parameters for the alignment of images, i.e. image registration using feature-based methods
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
  • G06V10/00—Arrangements for image or video recognition or understanding
  • G06V10/20—Image preprocessing
  • G06V10/24—Aligning, centring, orientation detection or correction of the image
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
  • G06V10/00—Arrangements for image or video recognition or understanding
  • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
  • G06V10/74—Image or video pattern matching; Proximity measures in feature spaces
  • G06V10/75—Organisation of the matching processes, e.g. simultaneous or sequential comparisons of image or video features; Coarse-fine approaches, e.g. multi-scale approaches; using context analysis; Selection of dictionaries
  • G06V10/754—Organisation of the matching processes, e.g. simultaneous or sequential comparisons of image or video features; Coarse-fine approaches, e.g. multi-scale approaches; using context analysis; Selection of dictionaries involving a deformation of the sample pattern or of the reference pattern; Elastic matching

Definitions

• the present invention relates to image processing systems and methods, and more particularly to image registration systems that combine two or more images into a composite image Background Art
• Image registration involves combining two or more images, or selected points from the images, to produce a composite image containing data from each of the registered images During registration, a transformation is computed that maps related points among the combined images so that points defining the same structure in each of the combined images are correlated in the composite image
• mapping relationship u(x) is extended from the set of N landmark points to the continuum using a linear quadratic form regula ⁇ zation optimization of the equation
• the second currently-practiced technique for image registration uses the mathematics of small deformation multi-target registration and is purely image data driven
• volume based imagery is generated of the two targets from which a coordinate system transformation is constructed
• a distance measure represented by the expression D(uJ, represents the distance between a template T(x) and a target image S(x)
• D(uJ) represents the distance between a template T(x) and a target image S(x)
• the distance measure D(u) measuring the disparity between imagery has various forms, e.g., the Gaussian squared error distance J
• the second technique is limited by the computational complexity presented by the number of data points in most images
• the second technique is further limited by the fact that the process produces many local minima that confuse proper registration This is because when registering two images according to the second technique, many possible orientations of the images produce subregions in the images that are properly matched but the images as a whole are improperly registered
• the present invention overcomes the limitations of the conventional techniques by providing a methodology which combines, or fuses, some aspects of both Techniques Specifically, the present invention uses landmark manifolds to produce a coarse registration, and subsequently incorporates image data to complete a fine registration of the template and target images
• a method for registering a template image and a target image comprises several steps, including defining manifold landmark points in the template image and identifying points in the target image corresponding to the defined manifold landmark points Once these points have been identified, the method includes the steps of computing a transform relating the defined manifold landmark points in the template image to corresponding points in the target image, fusing the first transform with a distance measure to determine a second transform relating all points within a region of interest in the target image to the corresponding points in the template image, and registering the template image with the target image using this second transform
• Fig 1 is a target and template image of an axial section of a human head with 0- dimensional manifolds
• FIG. 1 is schematic diagram illustrating an apparatus for registering images in accordance with the present invention
• Fig 3 is a flow diagram illustrating the method of image registration according to the present invention.
• Fig 4 is a target and a template image with 1 -dimensional manifolds
• Fig 5 is a target and a template image with 2-d ⁇ mens ⁇ onal manifolds
• Fig 6 is a target and a template image with 3-d ⁇ mens ⁇ onal manifolds
• Fig 7 is sequence of images illustrating registration of a template and target image
• Fig 8 is a flow diagram illustrating the computation of a fusing transform BEST MODE FOR CARRYING OUT THE INVENTION
• Fig 1 shows two axial views of a human head
• template image 100 contains points 102, 104, and 1 14 ldentifying structural points (0-d ⁇ mens ⁇ onal landmark manifolds) of interest in the template image
• Target image 120 contains points 108, 1 10, 1 16, corresponding respectively to template image points 102, 104, 1 14 via vectors 106, 1 12, 1 18, respectively
• FIG 2 shows apparatus to carry out the preferred embodiment of this invention
• a medical imaging scanner 214 obtains the images show in Fig 1 and stores them on a computer memory 206 which is connected to a computer central processing unit (CPU) 204
• CPU central processing unit
• a parallel computer platform having multiple CPUs is also a suitable hardware platform for the present invention, including, but not limited to, massively parallel machines and workstations with multiple processors
• Computer memorv 206 can be directly connected to CPU 204, or this memory can be remotely connected through a communications network
• Registering images 100 120 unifies registration based on landmark deformations and image data transformation using a coarse-to-fine approach
• the highest dimensional transformation required during registration is computed from the solution of a sequence of lower dimensional problems driven by successive refinements
• the method is based on information either provided bv an operator, stored as defaults or determined automatically about the various substructures of the template and the target, and varying degrees of knowledge about these substructures derived from anatomical imagery, acquired from modalities like CT MRl, functional MRl, PET ultrasound, SPECT. MEG, EEG, or cryosection
• an operator using pointing device 208, moves cursor 210 to select points 102, 104, 1 14 m Fig 1 , which are then displayed on a computer monitor 202 along with images 100, 120 Selected image points 102, 104, and 1 14 are 0-d ⁇ mens ⁇ onal manifold landmarks
• CPU 204 computes a first transform relating the manifold landmark points in template image 100 to their corresponding image points in target image 120
• a second CPU 204 transform is computed by fusing the first transform relating selected manifold landmark points with a distance measure relating all image points in both template image 100 and target image 120
• the operator can select an equation for the distance measure several ways including, but not limited to, selecting an equation from a list using pointing device 208, entering into CPU 204 an equation using keyboard 212, or reading a default equation from memory 206 Registration is completed by CPU 204 applying the second computed transform to all points in the template image 100
• the transforms, boundary values, region of interest, and distance measure can be defaults read from memory or determined automatically
• Fig 3 illustrates the method of this invention in operation
• each landmark point, x aboard in the template image is a corresponding point _y, in the target image
• the operator therefore next identifies the corresponding points, y, , in the target image are identified (step 310)
• the nature of this process means that the corresponding points can only be identified within some degree of accuracy
• This mapping between the template and target points can be specified with a resolution having a Gaussian error of variance ⁇ 2
• the operator may select a region of interest in the target image Restricting the computation to a relatively small region of interest reduces both computation and storage requirements because transformation is computed only over a subregion of interest It is also possible that in some applications the entire image is the desired region of interest In other applications there may be default regions of interest that are automatically identified
• the number of computations required is proportional to the number of points in the region of interest, so the computational savings equals the ratio of the total number of points in the image to the number of points in the region of interest
• the data storage savings for an image with N points with a region of interest having M points is a factor of N/M
• the computation time and the data storage are reduced by a factor of eight
• CPU 204 computes a transform that embodies the mapping relationship between these two sets of points (step 350) This transform can be estimated using Bayesian optimization, using the following equation
• A is a 3 x 3 matrix
• b [/>,
• b 2 , b ⁇ is a 3 x 1 vector
• L ⁇ ( / . ⁇ . ⁇ , j ] is a 3 x 1
• the foregoing steps of the image registration method provide a coarse matching of a template and a target image
• Fine matching of the images requires using the full image data and the landmark information and involves selecting a distance measure by solving a synthesis equation that simultaneously maps selected image landmarks in the template and target images and matches all image points within a region of interest
• a synthesis equation that simultaneously maps selected image landmarks in the template and target images and matches all image points within a region of interest
• the operator L in equation (6) may be the same operator used in equation (4), or alternatively, another operator may be used with a different set of boundary conditions
• the distance measure in the first term measures the relative position of points in the target image with respect to points in the template image
• this equation uses a quadratic distance measure, one of ordinary skill in the art will recognize that there are other suitable distance measures
• CPU 204 then computes a second or fusing transformation (Step 370) using the synthesis equation relating all points within a region of interest in the target image to all corresponding points in the template image
• the synthesis equation is defined so that the resulting transform incorporates or fuses, the mapping of manifold landmarks to corresponding target image points determined when calculating the first transform
• the computation using the synthesis equation is accomplished by solving a sequence of optimization problems from coarse to fine scale via estimation of the basis coefficients ⁇ k
• This is analogous to multi-g ⁇ d methods, but here the notion of refinement from coarse to fine is accomplished by increasing the number of basis components d As the number of basis functions increases, smaller and smaller variabilities between the template and target are accommodated
• the basis coefficients are determined by gradient descent, I e ,
• ⁇ is a fixed step size and ⁇ are the eigenvalues of the eigenvectors ⁇
• Equation (7) is used to initialize the value of the displacement field u(x) - ⁇ t m (x) (step 800)
• the basis coefficients ⁇ k ⁇ k ' 0> are set equal to zero and the variables ⁇ ; , A, and b are set equal to the solution of equation (6) (step 802)
• Equation (8) is then used to estimate the new values of the basis coefficients //, (n '" given the current estimate of the displacement field u ⁇ nJ (x) (step 804)
• Equation (10) is then used to compute the new estimate of the displacement field u (n> (x) given the current estimate of the basis coefficients ⁇ k ' n> (step 806)
• the next part of the computation is to decide whether or not to increase the number d of basis functions ⁇ k used to represent the transformation (step 808) Increasing the number of basis functions allows more deformation Normally, the algorithm is started
• CPU 204 uses this transform to register the template image with the target image (step 380)
• the spectrum of the second transformation, h is highly concentrated around zero This means that the spectrum mostly contains low frequency components
• the transformation can be represented by a subsampled version provided that the sampling frequency is greater than the Nyquist frequency of the transformation
• the computation may be accelerated by computing the transformation on a coarse g ⁇ d and extending it to the full voxel lattice e g , in the case of 3D images, by interpolation
• the computational complexity of the algorithm is proportional to the dimension of the lattice on which the transformation is computed Therefore, the computation acceleration equals the ratio of the full voxel lattice to the coarse computational lattice
• Another wav to increase the efficiency of the algorithm is to precompute the Green's functions and eigen functions of the operator / and store these precomputed values in a lookup table These tables replace the computation of these functions at each iteration with a table lookup
• This approach exploits the symmetry of Green ' s functions and eigen functions of the operator / so that very little computer memory is required
• the radial symmetry is exploited by precomputing the Green's function only along a radial direction
• the method described for fusing landmark information with the image data transformation can be extended from landmarks that are individual points (0-d ⁇ mens ⁇ onal manifolds) to manifolds of dimensions 1 , 2 and 3 corresponding to curves (1- dimensional), surfaces (2-d ⁇ mens ⁇ onal) and subvolumes (3 -dimensional)
• Fig 4 shows a template image 400 of a section of a brain with 1 - dimensional manifolds 402 and 404 corresponding to target image 406 1 -dimensional manifolds 408 and 410 respectively
• Fig 5 shows a template image 500 of a section of a brain with 2-d ⁇ mens ⁇ onal manifold 502 corresponding to target image 504 2-d ⁇ mens ⁇ onal manifold 506
• Fig 6 shows a template image 600 of a section of a brain with 3- dimensional manifold 602 corresponding to target image 604 3-d ⁇ mens ⁇ onal manifold 606
• M(3),dS is the Lebesgue measure on M 3 .
• dS is the surface measure on M(2)
• dS is the line measure on M(l)
• M(0), dS is the atomic measure.
• the Fredholm integral equation degenerates into a summation given by equation (5).
• step 370 It is also possible to compute the transform (step 370) with rapid convergence by solving a series of linear minimization problems where the solution to the series of linear problems converges to the solution of the nonlinear problem This avoids needing to solve the nonlinear minimization problem directly Using a conjugate gradient method, the computation converges faster than a direct solution of the synthesis equation because the basis coefficients ⁇ are updated with optimal step sizes
• the displacement field is assumed to have the form d u ( ⁇ ) ⁇ k ⁇ k W ' f ( ⁇ ) (16)
• I e , fig ⁇ , 3 , /, g, for /, g e M'
• equation ( 18) results Computation of equation ( 18) repeats until all ⁇ fall below a predetermined threshold solving for each ⁇ , in sequence of increasing j, and ⁇ y is computed using the values of ⁇ k for 0 ⁇ k ⁇ j
• step 370 of Fig 3 computing the registration transform fusing landmark and image data, is implemented using the conjugate gradient method, the computation will involve a series of inner products Using the FFT exploits the structure of the eigen functions and the computational efficiency of the FFT to compute these inner-products
• the integral in the above summation for all k can be computed by Fourier transforming the elements of the 3 x 3 matrix
• V7 ' V7Y (25)
• a distance function used to measure the disparity between images is the Gaussian square error distance
• j [(x-u(x)J - S(x) ⁇ 2 dx
• distance functions such as the correlation distance, or the Kullback Liebler distance, can be written in the form f O(l(x-u(x)) , S(x))dx
• D( , ) is a distance function relating points in the template and target images
• the basis coefficients ⁇ ⁇ k ⁇ are determined by gradient descent, i.e.
• Modifying boundary conditions requires modifying the butterflies of the FFT from complex exponentials to appropriate sines and cosines
• template image 700 illustrates the sequence of registering a template image and a target image
• Template image 700 has 0-dimensional landmark manifolds 702. Applying the landmark manifold transform computed at step 350 in Fig 3 to image 700 produces image 704 Applying a second transform computed using the synthesis equation combining landmark manifolds and image data to image 700 produces image 706
• Image 706 is the final result of registering template image 700 with target image 708 Landmark manifold 710 in image 708 corresponds to landmark manifold 702 in template image 700
• CT computed tomography
• MRl magnetic resonance imaging
• this invention can also be used on images acquired from other imaging modalities
• application of the present invention is not limited to anatomical images
• This invention also applies to non-anatomical images, including, but not limited to, satellite imagery, photographs, radar images, and images acquired from multiple sources
• the present invention overcame the limitation of the conventional technique by using some aspects of both techniques
• the principal advantage of the present invention is an image registration method and apparatus that fuses the techniques of registration using selected landmarks and image data
• Other advantages of the invention include 1 ) allowing for experts to insert knowledge directly into the imagery while at the same time allowing for the imagery itself to drive registration, 2) extending the classical framework for landmark point information (0-d ⁇ mens ⁇ onal landmarks) used in manual assisted deformation to arbitrary manifold information in the form of 0, 1 , 2, and 3 dimensional landmarks, I e .

Priority Applications (5)

Applications Claiming Priority (2)

Related Child Applications (1)

Publications (1)

Family

ID=24723602

Family Applications (1)

Country Status (7)

Cited By (8)

Families Citing this family (114)

Citations (10)

Family Cites Families (66)

• 1996
  • 1996-07-10 US US08/678,628 patent/US6009212A/en not_active Expired - Lifetime
• 1997
  • 1997-07-08 AU AU35911/97A patent/AU3591197A/en not_active Abandoned
  • 1997-07-08 WO PCT/US1997/011563 patent/WO1998001818A1/en active IP Right Grant
  • 1997-07-08 AT AT97932453T patent/ATE265070T1/de not_active IP Right Cessation
  • 1997-07-08 CA CA002260085A patent/CA2260085A1/en not_active Abandoned
  • 1997-07-08 EP EP97932453A patent/EP0910832B1/en not_active Expired - Lifetime
  • 1997-07-08 DE DE69728765T patent/DE69728765T2/de not_active Expired - Lifetime
• 1999
  • 1999-04-27 US US09/299,567 patent/US6553152B1/en not_active Expired - Lifetime

Patent Citations (10)

Non-Patent Citations (4)

Also Published As

Similar Documents

Legal Events