WO2009077938A1

WO2009077938A1 - Successive landmark-based image registration

Info

Publication number: WO2009077938A1
Application number: PCT/IB2008/055202
Authority: WO
Inventors: Jens Von Berg; Torbjørn VIK
Original assignee: Koninklijke Philips Electronics N.V.; Philips Intellectual Property & Standards Gmbh
Priority date: 2007-12-14
Filing date: 2008-12-10
Publication date: 2009-06-25

Abstract

This invention relates to successive landmark-based image registration. A user is presented with a first and second image. The first image is computed from reference image data. The second image is computed from image data to be registered with the reference image data. Registration is carried out in stages. At each stage, the set of landmarks is modified to improve the result of image registration. The modification of the set of landmark pairs is determined by the user, based on a comparison of the first and the second image presented to the user. When a satisfactory result is achieved, registration is stopped.

Description

Successive landmark-based image registration

FIELD OF THE INVENTION

The present invention relates to successive landmark-based image registration.

BACKGROUND OF THE INVENTION Landmark-based image registration is useful in many medical applications both for diagnostics and treatment planning. Some prominent application fields where image registration plays an important role are: estimation of respiratory motion in multi-phase cardiac CT or MR images, determining a dose of radiation in radiotherapy planning of liver cancer treatment, alignment of follow-up images, and registration of pre-operatively acquired images with intra- and/or post-operatively acquired images. Robust registration of such medical images is a difficult problem.

Landmark-based image registration comprises the steps of defining a set of pairs of corresponding landmarks, the first landmark of each pair in a first image and the second landmark of each pair in the second image. One image, say the first image, is a reference image. The locations of landmarks in the first image define the target locations of the corresponding landmarks in the second image. Landmark-based registration involves deforming the second image to move landmarks in the second image as close as possible to the corresponding landmarks in the first image. The deformation may be determined by a registration transformation for deforming the second image so that the deformed second image becomes identical or similar to the first image. The transformations are limited to a class of transformations, e.g. to the class of affine transformations. If each transformation from the class is defined by a set of parameter values, registering may be implemented as a search for a set of parameter values for which an error function attains a minimum. The error function may be based on the distances between the landmarks in the first image and the corresponding landmarks in the second image. A person skilled in the art will understand the term "landmark" is used hereinafter as equivalent to a "reference point", "point", or "landmark location". Thus, landmark-based image registration is also known as point-based image registration. The set of pairs of corresponding landmarks is determined on the basis of the prominent features of anatomical or pathological structures comprised in the first and second image. These prominent features and landmarks may be predetermined using feature/landmark detection. Alternatively, pairs of corresponding landmarks may be selected ad hoc by a radiologist before registering two images. The latter task is tedious and error- prone.

BRIEF DESCRIPTION OF THE INVENTION

The object of the present invention is to provide for improving landmark- based image registration by making the registration task easier and less error-prone. According to one aspect, the present invention relates to a method of successive landmark-based image data registration, comprising: presenting to a user a first image computed from first image data, the first image data being reference image data, and a second image computed from second image data, the second image data being registered with the first image data, based on a set of pairs of corresponding landmarks, each pair of corresponding landmarks comprising a first landmark in the first image data and a corresponding second landmark in the second image data; receiving a user input for identifying at least one pair of corresponding landmarks for adding to the set of pairs of corresponding landmarks; updating the set of pairs of corresponding landmarks, based on the user input; and registering the second image data with the first image data, using the landmark-based image registration, based on the updated set of pairs of corresponding landmarks.

Comparing the first image and the second image enables the user to visually evaluate the result of image registration. Based on this evaluation, the user may further navigate through the first and second image data and use the first and second image to add more pairs of corresponding landmarks to further improve the result of image registration. Modifying the set of pairs of corresponding landmarks may be repeated until the result of image registration is satisfactory to the user. Then the user may end said image registration. Advantageously, the method allows the user to select meaningful landmarks. Also, the fact that the second image becomes, after each successive landmark-based registration, more and more similar to the first image makes it easier for the user to identify relevant landmarks required to improve the registrations. This approach is also less error-prone. Additionally, in this method, the user may interrupt the task of registering two image data sets, save incomplete registration results, and continue later to complete or improve the image registration. The invention is especially useful for medical image registration and may be used for registering 2D or 3D image data.

In an embodiment of the method, the user input comprises information on removing one or more pairs of corresponding landmarks from the set of pairs of corresponding landmarks. Hence, the method further allows removing pairs of corresponding landmarks which have been erroneously chosen or have turned out to be not well suited for registering the two images.

In an embodiment of the method, the user input comprises information on repositioning one or more pairs of corresponding landmarks of the set of pairs of corresponding landmarks. This feature of the method is particularly useful for correcting small errors in landmark placement and saves user's time. In an embodiment of the invention, registering the second image data with the first image data is further based only on the at least one pair of corresponding landmarks added to the set of corresponding landmarks. Positions of other landmarks in the first and second image data are fixed. Limiting the number of pairs of corresponding landmarks used for registering the first and second image data to the at least one pair of corresponding landmarks may have at least two positive effects: first, it may limit deformation of the second image data to a region surrounding second image data landmarks of the at least one pair of corresponding landmarks without spoiling results of image registration in other regions of the second image data; second, it may also require less time for registering the two images. In an embodiment of the invention, the method further comprises computing a value of a measure of similarity of the first and second image data and presenting the computed measure value to the user. The user thus receives additional information on the similarity of the first and second image data, which he can use to arrive at a decision to end or to continue the method or to keep or to remove a pair of landmarks. In an embodiment of the invention, receiving the user input for identifying the at least one pair of corresponding landmarks from the user involves indicating the at least one pair of corresponding landmarks in a set of candidate pairs of corresponding landmarks in the first and second image data. The candidate pairs of corresponding landmarks in the first and second image data may be predetermined, e.g. by the user, pre-computed, or may be computed in an initialization step of the method or at any other time before receiving the user input. The computation of candidate pairs of corresponding landmarks in the first and second image data may be based, for example, on a search for a pair of corresponding features in the first and second image. Accordingly, the user input does not need to manually identify landmarks in the first and second image. Instead, the user may accept, reject or correct the predetermined or computed candidate pairs of corresponding landmarks. Thus, the method becomes even more user-friendly and faster.

According to another aspect, the present invention relates to a computer program product for instructing a processing unit to execute the steps of the method when the product is run on a computer.

According to yet another aspect, the present invention relates to a system for successive landmark-based image data registration, comprising: a display for presenting to a user a first image computed from first image data, the first image data being reference image data, and a second image computed from second image data, the second image data being registered with the first image data, based on a set of pairs of corresponding landmarks, each pair of corresponding landmarks comprising a first landmark in the first image data and a corresponding second landmark in the second image data; an input unit for receiving a user input for identifying at least one pair of corresponding landmarks for adding to the set of pairs of corresponding landmarks; and a processor for:

- updating the set of pairs of corresponding landmarks, based on the user input; and

- registering the second image data with the first image data, using the landmark-based image registration, based on the updated set of pairs of corresponding landmarks.

According to yet another aspect, the present invention relates to an image acquisition apparatus comprising the system of the invention.

Each aspect or embodiment of the present invention may be combined with any of the other aspects or embodiments of the invention.

These and other aspects of the invention will be apparent from and elucidated with reference to the embodiments described hereinafter.

BRIEF DESCRIPTION OF THE DRAWINGS Embodiments of the invention will be described, by way of example only, with reference to the drawings, in which

Figure 1 shows a flowchart of the method according to the invention; Figure 2 illustrates the successive landmark-based image registration method of the invention;

Figure 3 schematically shows a block diagram of an exemplary embodiment of the system; and

Figure 4 schematically shows an exemplary embodiment of the image acquisition apparatus.

DESCRIPTION OF EMBODIMENTS

Figure 1 shows a flowchart of the method according to the invention. The method begins with an initialization step 1000. In this step, the set of pairs of corresponding landmarks is initialized. The set may initially be empty. Alternatively, the set may comprise a number of predefined or pre-computed pairs of corresponding landmarks, each pair of corresponding landmarks comprising a first landmark in first image data and a corresponding second landmark in second image data. If the set of pairs of corresponding landmarks is not empty, the initialization will further comprise registering the second image data with the first image data on the basis of the initial set of pairs of corresponding landmarks.

In a first step 101, a first and a second image are presented to a user, e.g. a radiologist, the first image being computed from the first image data which is reference image data, and the second image being computed from second image data which has been registered with the first image data, based on a set of pairs of corresponding landmarks. The images may be medical images depicting, e.g., the lung or the abdomen of a patient. By presenting the images to the user in that way, the user can see features in both images. Using the observed features, the user may select pairs of corresponding landmarks in the first and second image data.

Viewing the first and second image may require navigating through the first and second image data. Navigation may involve selecting a rendering method and/or views computed from the image data. In the case of planar views, navigation may comprise e.g. moving up or down a stack of images, rotating or translating viewing planes, re-scaling, i.e. magnifying or reducing images. For 3D renditions, navigation operations may include translating, rotating and zooming the viewed scene. A navigation operation may be applied to both images concurrently. Alternatively, a navigation operation may be applied to one image data. Navigation is used to help users to view details comprised in displayed images, especially to see image features for selecting landmarks suitable for image registration. Also, a visual inspection and comparison of the first and second image also allows the user to decide whether to continue or end image registration.

After the first step 101, the method 100 continues to the second step 102 for receiving a user input for identifying the at least one pair of corresponding landmarks in the first and second image data to be added to or removed from the set of pairs of corresponding landmarks. If the user decides to add a new pair of corresponding landmarks, he may select two new landmarks, one in each of the first and second image, using a user input device such as a mouse or trackball, for example. If the user decides to remove a pair of corresponding landmarks from the set of landmarks, he may select a pair of corresponding landmarks for removing these landmarks from the set of pairs of corresponding landmarks. Optionally, the user may select a pair of corresponding landmarks from the set of pairs of corresponding landmarks and drag each landmark to a new location. Optionally, dragging a landmark in the first image may result in a respective repositioning of the corresponding landmark in the second image. Next, the user may refine positions of each of these two landmarks by moving each landmark individually. The option of repositioning pairs of corresponding landmarks offers a quick way of removing an existing pair of corresponding landmarks from the set and adding a new pair of corresponding landmarks to the set.

The user input may also comprise an input for navigating through the first and second image data as described above. The step of receiving user input may be combined with or followed by the first step 101 for presenting the first and second image to the user. Moreover, the user input may comprise a user decision to end executing the steps of the method 100. If the user, after visually comparing a number of first and second images, decides to end the registration, the registration ends.

After the second step 102, the method 100 continues to a third step 103 for updating the set of pairs of corresponding landmarks. This step may include adding or removing the at least one pair of corresponding landmarks to or from the set of pairs of corresponding landmarks, or updating the locations of the at least one pair of corresponding landmarks, based on the user input.

After the third step 103, the method 100 continues to a fourth step 104 for registering the second image data with the first image data, using the landmark-based image registration, based on the updated set of pairs of corresponding landmarks. A registration transformation based on the set of pairs of corresponding landmarks is applied to the second image data. The registration transformation may be described by a deformation field, which assigns new locations to voxels or pixels of the second image data. The transformation or, equivalently, the deformation field may be fully determined by pairs of corresponding landmarks defining said transformation. For one pair of corresponding landmarks, the transformation or deformation field is defined by a translation (i.e. no deformation sensu stricto occurs). For two pairs of corresponding landmarks, the transformation or the deformation field may be defined, for example, by a combination of a translation, rotation and scaling operation. Other possible transformations or deformation fields may be defined by similarity transformations, affine transformations, or thin plate splines for interpolating the field of displacements defined by pairs of corresponding landmarks, for example. A person skilled in the art will know many suitable landmark-based image registration techniques.

In an embodiment, the fourth 104 step may comprise presenting the result of the landmark-based image registration of the first and second image data to the user. The user may have an option to accept or reject the result based on a visual inspection of the first image computed from the first image data and the second image computed on the basis of the registered second image data. If the user rejects the result, the update of the set of corresponding landmarks carried out in the third step 103 is annulled. After the fourth step 104, the method returns to the first step 101.

In an embodiment of the invention, the method further comprises computing a value of a measure of similarity of the first and second image data and presenting the computed value to the user. The measure of similarity may be based on the set of pairs of corresponding landmarks. Alternatively or additionally, the measure of similarity may be based on the differences of voxel/pixel values of the corresponding voxels/pixels in the first and second image data. For example, the measure of similarity may be defined as the root of the mean of the sum of squares of differences of image intensities in corresponding locations. Those skilled in the art will know many useful measures of similarity which can be employed by the method of the invention. Presenting a computed value of a measure of similarity to the user provides the user with an additional clue about the quality of the image registration. In particular, when the progress of the image registration is difficult to evaluate by a visual inspection of the first and second image, the value of the similarity measure may provide an important hint as to whether the at least one pair of corresponding landmarks improved the image registration, i.e. made the first and second image data more similar. In an embodiment of the invention, receiving the user input for identifying the at least one pair of corresponding landmarks from the user involves indicating the at least one pair of corresponding landmarks in a set of candidate pairs of corresponding landmarks in the first and second image data. The candidate pairs of corresponding landmarks may be predetermined, e.g. by the user, pre-computed, or may be computed in an initialization step 1000 of the method 100 or at any other time before receiving the user input. The computation of candidate pairs of corresponding landmarks may be based, for example, on a search for a pair of corresponding features in the first and second image. In this embodiment, the user input involves accepting, rejecting or correcting the predetermined or computed landmarks. Thus, the method becomes even more user- friendly and faster.

In an embodiment, the candidate landmark pairs are determined by a local search in the first and second image data. The local search algorithm can be, for example, a block-matching algorithm. A block of voxels/pixels in the second image data is matched with a block of voxels/pixels in the first image data. The algorithm chooses image locations with salient features, e.g. Scale-Invariant Feature Transform (SIFT) features or bifurcations found in the lung with a vessel tracking algorithm, that do not yet correspond well given the current registration of the second image data with the first image data, but still have a reasonable probability of improving their correspondence. The candidate pairs of corresponding landmarks may be ordered in descending order based on the Euclidean distance between landmarks of each pair. The search may involve only those landmark pairs for which the distance is greater than a threshold.

Figure 2 illustrates the landmark-based image registration method of the invention. Figure 2 shows two axial views 21 and 22 of a lung CT image data obtained from the same patient in two different respiratory states. The first image 21 is based on the reference image data. Due to the through plane motion, the bronchial structure 25 selected by the cross lines 1 and 2 in the first image 21 is not visible at the corresponding position in the second image 22.

In order to register the first image data and the second image data, a set of corresponding landmark pairs have been determined by the user and used for landmark-based registration of the first and second image data. In the bottom row, Figure 2 shows the first image 21 and a third image 23, which third image is computed from the third image data. The third image data is the second image data registered with the first image data, using the set of corresponding landmark pairs determined by the user. After registration of the second image data with the first image data, the structure 25 is also visible in the third image. However, the registration is still not perfect. An additional landmark pair may further improve the registration accuracy.

Figure 3 shows a system 400 for successive landmark-based image data registration, comprising a display 403, an input unit 401 and a processor 402. The display 403 is adapted for presenting to a user 405 a first image 406 computed from first image data, the first image data being reference image data, and a second image 407 computed from second image data, the second image data being registered with the first image data, based on a set of pairs of corresponding landmarks, each pair of corresponding landmarks comprising a first landmark in the first image data and a corresponding second landmark in the second image data. Thus, the user can identify further anatomical landmarks in both images. The input unit 401 is adapted for receiving a user input 404 for identifying at least one pair of corresponding landmarks for adding to the set of pairs of corresponding landmarks. The input unit 401 may be a computer mouse or a button on a touch screen integrated into the display 403, for example. The processor 402 is adapted for updating the set of pairs of corresponding landmarks, based on the user input 404, and for registering the second image data with the first image data, using the landmark-based image registration, based on the updated set of pairs of corresponding landmarks.

Fig. 4 schematically shows an exemplary embodiment of the image acquisition apparatus 500 employing the system 400, said image acquisition apparatus 500 comprising a CT image acquisition unit 510 connected via an internal connection with the system 400, an input connector 501, and an output connector 502. This arrangement advantageously increases the capabilities of the image acquisition apparatus 500, providing said image acquisition apparatus 500 with advantageous capabilities of the system 400.

Hence, this invention relates to successive landmark-based image registration. A user is presented with a first and a second image. The first image is computed from reference image data. The second image is computed from image data to be registered with the reference image data. Registration is carried out in stages. At each stage, the set of landmarks is modified to improve the result of image registration. The modification of the set of landmark pairs is determined by the user, based on a comparison of the first and second image presented to the user. When a satisfactory result is achieved, registration is stopped.

Certain specific details of the disclosed embodiment are set forth for purposes of explanation rather than limitation, so as to provide a clear and thorough understanding of the present invention. However, it should be understood by those skilled in this art, that the present invention might be practiced in other embodiments that do not conform exactly to the details set forth herein, without departing significantly from the spirit and scope of this disclosure. Further, in this context, and for the purposes of brevity and clarity, detailed descriptions of well-known apparatuses, circuits and methodologies have been omitted so as to avoid unnecessary detail and possible confusion.

Reference signs are included in the claims, however, the inclusion of the reference signs is only for clarity reasons and should not be construed as limiting the scope of the claims.

Claims

CLAIMS:

1. A method of successive landmark-based image data registration, comprising: presenting (101) to a user a first image computed from first image data, the first image data being reference image data, and a second image computed from second image data, the second image data being registered with the first image data, based on a set of pairs of corresponding landmarks, each pair of corresponding landmarks comprising a first landmark in the first image data and a corresponding second landmark in the second image data; receiving (102) a user input for identifying at least one pair of corresponding landmarks for adding to the set of pairs of corresponding landmarks; - updating (103) the set of pairs of corresponding landmarks, based on the user input; and registering (104) the second image data with the first image data, using the landmark-based image registration, based on the updated set of pairs of corresponding landmarks.

2. A method as claimed in claim 1, wherein the user input comprises information on removing one or more pairs of corresponding landmarks from the set of pairs of corresponding landmarks.

3. A method as claimed in claim 1, wherein the user input comprises information on repositioning one or more pairs of corresponding landmarks of the set of pairs of corresponding landmarks.

4. A method according to claim 1, wherein registering the second image data with the first image data is further based only on the at least one pair of corresponding landmarks added to the set of corresponding landmarks.

5. A method according to claim 1, further comprising computing a value of a measure of similarity of the first and second image data and presenting the computed measure value to the user.

6. A method according to claim 1, wherein receiving the user input for identifying the at least one pair of corresponding landmarks from the user involves indicating the at least one pair of corresponding landmarks in a set of candidate pairs of corresponding landmarks in the first and second image data.

7. A computer program product for instructing a processing unit to execute the steps of the method of claim 1 when the product is run on a computer.

8. A system (400) for successive landmark-based image data registration, comprising: - a display (403) for presenting to a user a first image computed from first image data, the first image data being reference image data, and a second image computed from second image data, the second image data being registered with the first image data, based on a set of pairs of corresponding landmarks, each pair of corresponding landmarks comprising a first landmark in the first image data and a corresponding second landmark in the second image data; an input unit (401) for receiving a user input for identifying at least one pair of corresponding landmarks for adding to the set of pairs of corresponding landmarks; a processor (402) for

9. An image acquisition apparatus (500) comprising a system (400) as claimed in claim 8.