EP1851722B1 - Image processing device and method - Google Patents
Image processing device and method Download PDFInfo
- Publication number
- EP1851722B1 EP1851722B1 EP06710848A EP06710848A EP1851722B1 EP 1851722 B1 EP1851722 B1 EP 1851722B1 EP 06710848 A EP06710848 A EP 06710848A EP 06710848 A EP06710848 A EP 06710848A EP 1851722 B1 EP1851722 B1 EP 1851722B1
- Authority
- EP
- European Patent Office
- Prior art keywords
- segmentation
- model
- data
- subset
- image
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000012545 processing Methods 0.000 title claims abstract description 52
- 238000000034 method Methods 0.000 title claims description 59
- 230000011218 segmentation Effects 0.000 claims abstract description 205
- 238000003672 processing method Methods 0.000 claims abstract description 16
- 238000004590 computer program Methods 0.000 claims abstract description 6
- 230000009466 transformation Effects 0.000 claims description 20
- 238000005457 optimization Methods 0.000 claims description 12
- 238000012937 correction Methods 0.000 claims description 11
- 238000004364 calculation method Methods 0.000 claims description 9
- 238000009795 derivation Methods 0.000 claims description 2
- 210000005246 left atrium Anatomy 0.000 description 14
- 230000000747 cardiac effect Effects 0.000 description 12
- 230000008569 process Effects 0.000 description 10
- 210000003492 pulmonary vein Anatomy 0.000 description 10
- 241001088162 Primula auricula Species 0.000 description 9
- 235000006894 Primula auricula Nutrition 0.000 description 9
- 210000002837 heart atrium Anatomy 0.000 description 9
- 238000013459 approach Methods 0.000 description 8
- 238000001514 detection method Methods 0.000 description 7
- 238000010586 diagram Methods 0.000 description 6
- 238000012549 training Methods 0.000 description 6
- 230000006978 adaptation Effects 0.000 description 5
- 238000002372 labelling Methods 0.000 description 5
- 238000009877 rendering Methods 0.000 description 5
- 230000001746 atrial effect Effects 0.000 description 4
- 238000002591 computed tomography Methods 0.000 description 4
- 238000002059 diagnostic imaging Methods 0.000 description 4
- 238000003709 image segmentation Methods 0.000 description 4
- 210000005240 left ventricle Anatomy 0.000 description 4
- 238000012805 post-processing Methods 0.000 description 4
- 230000008901 benefit Effects 0.000 description 3
- 239000008280 blood Substances 0.000 description 3
- 210000004369 blood Anatomy 0.000 description 3
- 230000006870 function Effects 0.000 description 3
- 210000002216 heart Anatomy 0.000 description 3
- 230000006872 improvement Effects 0.000 description 3
- 238000007781 pre-processing Methods 0.000 description 3
- 210000001147 pulmonary artery Anatomy 0.000 description 3
- 230000002685 pulmonary effect Effects 0.000 description 3
- 238000012360 testing method Methods 0.000 description 3
- 238000000844 transformation Methods 0.000 description 3
- 210000002376 aorta thoracic Anatomy 0.000 description 2
- 238000004422 calculation algorithm Methods 0.000 description 2
- 230000000295 complement effect Effects 0.000 description 2
- 150000001875 compounds Chemical class 0.000 description 2
- 239000002872 contrast media Substances 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 238000002595 magnetic resonance imaging Methods 0.000 description 2
- 239000011159 matrix material Substances 0.000 description 2
- 210000000056 organ Anatomy 0.000 description 2
- 230000004044 response Effects 0.000 description 2
- 230000002123 temporal effect Effects 0.000 description 2
- 239000013598 vector Substances 0.000 description 2
- PXFBZOLANLWPMH-UHFFFAOYSA-N 16-Epiaffinine Natural products C1C(C2=CC=CC=C2N2)=C2C(=O)CC2C(=CC)CN(C)C1C2CO PXFBZOLANLWPMH-UHFFFAOYSA-N 0.000 description 1
- 108010028984 3-isopropylmalate dehydratase Proteins 0.000 description 1
- 101100129500 Caenorhabditis elegans max-2 gene Proteins 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 210000003484 anatomy Anatomy 0.000 description 1
- 210000000709 aorta Anatomy 0.000 description 1
- 210000001008 atrial appendage Anatomy 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 210000005242 cardiac chamber Anatomy 0.000 description 1
- 238000013184 cardiac magnetic resonance imaging Methods 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 210000004351 coronary vessel Anatomy 0.000 description 1
- 238000013144 data compression Methods 0.000 description 1
- 230000007717 exclusion Effects 0.000 description 1
- 238000010191 image analysis Methods 0.000 description 1
- 238000003384 imaging method Methods 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 230000002452 interceptive effect Effects 0.000 description 1
- 210000004072 lung Anatomy 0.000 description 1
- 238000007674 radiofrequency ablation Methods 0.000 description 1
- 210000005245 right atrium Anatomy 0.000 description 1
- 210000005241 right ventricle Anatomy 0.000 description 1
- 238000005070 sampling Methods 0.000 description 1
- 238000011524 similarity measure Methods 0.000 description 1
- 238000013519 translation Methods 0.000 description 1
- 238000002604 ultrasonography Methods 0.000 description 1
- 210000005166 vasculature Anatomy 0.000 description 1
- 210000003462 vein Anatomy 0.000 description 1
- 238000012795 verification Methods 0.000 description 1
- 238000012800 visualization Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/149—Segmentation; Edge detection involving deformable models, e.g. active contour models
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/12—Edge-based segmentation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10072—Tomographic images
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30004—Biomedical image processing
- G06T2207/30048—Heart; Cardiac
Definitions
- the present invention relates to an image processing device and a corresponding image processing method for processing a multi-dimensional data set of image data, in particular of three-dimensional medical volume image data, including a segmentation unit for a segmentation of an image object in said data set. Further, the present invention relates to a computer program for implementing said image processing method on a computer.
- X-ray, computer tomography and magnetic resonance imaging are available to assess the condition of the coronary vasculature.
- the imaged region of interest and other requirements it is often needed to segment a target organ or, more generally, a target image object from the medical image data, in particular from medical volume image data, wherein it is desired that the segmentation may be performed automatically by a computer without a need for extensive supervision or assistance by a human user.
- any kind of multi-dimensional image data may be segmented according to the invention, including medical and non-medical image data, two-dimensional image data like common pictures, three-dimensional image data like volume image data or a temporal sequence of two-dimensional image data, four-dimensional image data like a temporal sequence of volume image data, and even higher-dimensional image data.
- model-based segmentation Several different methods have been proposed for segmentation. These methods may in general be grouped into two categories: model-based segmentation and data-driven segmentation.
- Statistical shape models typically capture the shape properties of a training set of target objects. Active shape models that make use of these priors have been used already to segment cardiac chambers in MR and echocardiographic image volumes. Comparably less work was done for multislice computed tomography (MSCT) images. Labelling of a number of target objects was done by non-rigid image-based registration of a labelled image (atlas) to the image of interest (cf. Lorenzo-Valdes, G. Sanchez-Ortiz, R. Mohiaddin, D. Ruckert. Atlas-based segmentation and tracking of 3D cardiac images using non-rigid registration. In: Proc.
- MMCT contrast-medium enhanced multi-slice computed tomography
- This method reaches a resolution as fine as the voxel spacing, which is about 0.5mm in cardiac MSCT images.
- this method is not capable of reliably separating it from nearby structures that are also above the Hounsfield threshold.
- Fig. 1 gives an impression on such a segmentation.
- these other structures are mainly the blood pool of the left ventricle and the coronaries.
- the pulmonary veins also come close to the pulmonary arteries, the descending aorta, and the spine.
- Data-driven methods are not necessarily restricted by a number of parameters. Usually, either local boundary candidates are detected, or regions of similar voxels are pooled together. Thus, a high accuracy may potentially be reached in the data-driven segmentation result, but there is a vulnerability to image artifacts like noise, partial volume effects etc. Usually, data-driven methods find more boundary candidates than desired for a given object.
- a corresponding image processing method is defined in Claim 16.
- a computer program for causing a computer to carry out the steps of the image processing method according to the invention when said computer program is executed on a computer is defined in claim 17.
- subset is used herein to describe the result of the data-driven segmentation in general.
- the term “subset” does not only refer to a subset of said data set having substantially the same data structure as the data set, i.e. being a subset in a mathematical or literal sense, but also to any other result or output of the data-driven segmentation representing the image object or an approximation thereof, e.g. a iso-surface triangulation.
- the term "optimizing a similarity value” herein has the meaning of adapting said model and/or said subset in such a way that a desired similarity is achieved or that the similarity value is above or below a predetermined threshold value.
- the optimum of the similarity value does not have to be a similarity value corresponding to a sameness.
- the present invention is based in the idea that a model-based segmentation and a data-driven segmentation may be performed on the same image data and the results of these segmentations may be used as first approximations of the image object to be segmented.
- the drawbacks described above are overcome by using both approximations wherein the specific advantages of either method are used to achieve a proper segmentation of the image object.
- the information derived from the data set of image data by either of the two methods may be used to improve the results of the respective other method.
- Said model-based segmentation may include a mesh-based segmentation, a functional-approximation-based segmentation and/or combinations thereof and said data-driven segmentation may include a pixel-based segmentation, a boundary-based segmentation, a region-based segmentation, a texture-based segmentation, a watershed-segmentation and/or combinations thereof.
- Other known methods for segmentation may also be used as long as a comparison or measure of similarity between the respective results is possible.
- said data-driven segmentation means is adapted for using information derived by said model-based segmentation means from said model-based segmentation in said data-driven segmentation and/or said model-based segmentation means is adapted for using information derived by said data-driven segmentation means from said data-driven segmentation in said model-based segmentation.
- Said information derived from said model-based segmentation may preferably include starting values for said data-driven segmentation, threshold values for said data-driven segmentation and/or a region of interest for selecting a subsection of said data set for which said data-driven segmentation is to be performed.
- said information derived from said data-driven segmentation may preferably include a region of interest for selecting a subsection of said data set for which said model-based segmentation is to be performed and/or estimations for position, orientation and/or size of said object as starting values for said model-based segmentation. Since the segmentation methods are applied to the same data set of image data the result of one kind of segmentation or approximation can be used as supporting data for the other kind. For example, once a model-based segmentation is completed and a model representing the image object is obtained, the model may be used for a coarse distinction between voxels of the object and voxels of the background. According to this distinction a threshold value, e.g. a gray value, may be chosen or calculated which can be used for the data-driven segmentation.
- the estimation of a region of interest permits a omitting of a part of the data set which leads to improved processing times.
- said optimizing means is adapted for adapting said model and/or said subset by re-modeling of said model and/or of a part of said model, excluding a part of said subset from said subset and/or including a part of said data set into said subset.
- said optimizing means may be necessary to repeat one segmentation or both using information derived from the other approximation. Further, it may be decided upon the similarity value between the two approximations to omit some parts of an approximation.
- all parts of the subset of image data which have a distance to a surface of the model greater than a given value may be deleted from the subset. This will clear the subset from artifacts and objects in the image data which are no part of the object which is to be segmented.
- the image processing device further comprises display means for displaying said multi-dimensional model, said multi-dimensional subset, said similarity value, said adapted model, said adapted subset and/or said derived information; user input means for reading a correction input from a user; and correction means for correcting and/or altering said model-based segmentation, said data-driven segmentation, said model, said subset, said calculation of said similarity value, said optimization of said similarity value, said adapted model, said adapted subset, said derivation of said information and/or said derived information according to said correction input.
- the display of important data related to the segmentations allows a user to decide on corrections or improvements to the segmentations and to effect them by changing the data which is used for the segmentations or which is obtained from the segmentations via the input means.
- the image processing device includes gap detecting means for detecting gaps in said multi-dimensional subset and completion means for closing of detected gaps in said subset.
- the subset of image data includes gaps, i.e. regions in which there should be a closed and continuous boundary. Such gaps could be caused by an incorrect segmentation or by a subsequent optimization in which parts of the subset are deleted because there was a large distance to the surface of the model. If these gaps are closed - for example by interpolation of the surrounding subset - one obtains a subset of image data which gives a better approximation of the image object.
- said completion means is adapted for using closing information derived from said multi-dimensional model, in particular topological closing information.
- closing information derived from said multi-dimensional model, in particular topological closing information.
- the model includes information on the expected shape of the object, it is therefore possible and advantageous to close the gaps with a filling which corresponds to the respective area of the model, for example to include a convex filling if the gap corresponds to a part of the model which is convex.
- more complex filling structures are also possible, too.
- the image processing device comprises display means for displaying said multi-dimensional model, said multi-dimensional subset, said similarity value, said adapted model, said adapted subset and/or said derived information and user input means for reading a selecting input from a user, wherein said selecting means is adapted for selecting said adapted model or said adapted subset according to said selecting input.
- the image processing device of another embodiment comprises calculation means for calculating a transformation relation between said model and a predetermined reference model and deformation means for deforming said adapted subset by applying said transformation relation to said adapted subset.
- a model is adapted to the present data set of image data by - for example - changing of positions of vertices of a mesh. This change of positions is equivalent to a transformation of a starting model to the model which represents a first approximation of the image object. If this transformation is also applied to the adapted subset the obtained transformed adapted subset may be compared to a reference subset or a reference model in order to map corresponding details, for example anatomical details, to each other. This permits an easier classification of anatomical variations and may also help designing new reference or starting models.
- Fig. 1 shows an iso-surface triangulation of the blood-pool from an MSCT image sub-volume. This iso-surface triangulation has been segmented by the application of a threshold for the Hounsfield values.
- the resolution which can be achieved may be as fine as the voxel spacing, which is about 0.5 mm in cardiac MSCT images.
- the atrium in this case, but also nearby structures which are also above the Hounsfield threshold are included in this subset of image data.
- the left auricula is clearly visible, below it the coronaries, also both left pulmonary vein branches and the left branch of pulmonary arteries.
- a bootstrap approach for consecutive model building and refinement may be followed.
- the basic cardiac structures of interest the atrium body, the atrial appendage (atrial auricula), the left upper pulmonary vein trunk (LUPVT), the left lower PVT, the right upper PVT, and the right lower PVT
- LUPVT left upper pulmonary vein trunk
- i triangular surface models starting from geometric primitives (spheres and tubes) using the methods explained below.
- the resulting surface model is shown in Fig.
- P n ( pose ) has a translation, rotation, and anisotropic scaling component and was approximated interactively until P n v 0 visually agreed well with the image.
- A is an automatically performed free form deformation as explained below.
- additional local deformations D n were required performed with a method described in H. Timinger, V. Pekar, J. von Berg, K. Dietmeyer, M. Kaus. Integration of interactive corrections to model-based segmentation algorithms. In: Proc. Schmtechnik fuer die Medizin 11-15, 2003 .
- This mean model is however ideally positioned in the image by use of prior knowledge.
- a second, fully automatic fitting A A : v ⁇ m ⁇ ⁇ v m * was performed yielding the fitted model a * m with its vertices v * m and triangles t * m .
- a mean model v ⁇ * was calculated in a common co-ordinate system with the similarity transformation . This mean model is shown in Fig. 2b .
- N(j) gives the indices of all vertices that are neighbours of v j .
- R is a similarity transformation matrix that is determined by a closed-form point-based registration method prior to the energy calculation to make the energy calculation invariant to pose and scaling.
- e v ⁇ I is the unit vector in the direction of the image gradient at the triangle barycentre c i .
- the parameter D controls the trade-off between feature strength and distance.
- c ⁇ i is the best feature point for c i in the image.
- the used sample images were acquired at different hospitals, but all with Philips MX8000 IDT 16-line scanners. They were contrast enhanced as they were acquired for the purpose of coronary artery assessment on patients with cardiac complaints.
- the thresholding was applied to the whole image volume. In order to speed up the procedure the image was sub-sampled by a factor of two.
- the surface of the inner voxels (above threshold) was triangulated with the method from W.E. Lorensen, H.E. Cline.
- Marching Cubes A high resolution 3D surface reconstruction algorithm.
- a median filter (3 3 voxels in size) previously applied to the image volume makes the resulting iso-surface appear smoother and slightly reduces its area.
- FIG. 3 schematically shows an object 1, for instance a patient, from which image data, in particular volume image data, can be acquired by a data acquisition unit 2, which can generally be any medical imaging device, such as a computer tomograph, a magnetic resonance imaging unit, an X-ray device or an ultrasound unit.
- the acquired data are generally stored in a memory 3, such as a hard disc of a workstation or a server in a medical computer network.
- an image processing device includes a reconstruction unit 4 for reconstructing images from the acquired image data for display on a display means 5.
- a segmentation unit 6 is provided for segmentation of an (target) object in the image data.
- the segmentation unit 6 comprises a model-based segmentation means 61 which provides a model-based segmentation of the acquired data set stored in the memory 3.
- a model-based segmentation utilizes a priori knowledge on structures of the image data in the data set, for example the knowledge that an object like a atrium commonly has a special, characteristic shape and size. There may be aberrances or anomalies or even a range of possible variations which have to be included into the modeling.
- a modeling unit 7 is included into the image processing device.
- model-based segmentation methods including mesh-based segmentation and functional-approximation-based segmentation and there is no need for a further description of the model-based segmentation as such herein (cf. Model-Based Recognition Methods for LV Modeling, Section 3.4 in: Suri, Setarehdan, Singh (Eds.) Advanced Algorithmic Approaches to Medical Image Segmentation, Springer 2002 ).
- a data-driven segmentation means 62 is provided for a data-driven segmentation of the acquired data set stored in the memory 3.
- An example of such a data-driven segmentation method is a method applying a threshold for the Hounsfield values and a subsequent iso-surface triangulation as described above. It is not necessary to perform such a triangulation since there are other suitable known methods to permit a comparison between a model representing an image object and a data subset representing the same object.
- the surfaces of the object may also be approximated by any other mathematical means that allows for a similarity measure. Spherical harmonics are used for this purpose or spline-based approximations. Both allow for a distance measure between two such surfaces.
- subset is not limited to a subset of the data set of image data in a literal sense but also includes other representations of the image object or of such a subset which are obtained by a data-driven segmentation.
- the segmentation unit 6 further comprises an optimization means 63 for an optimization of a similarity value between the two approximations of the image object obtained by the model-based segmentation means 61 and the data-driven segmentation means 62.
- an optimization means 63 for an optimization of a similarity value between the two approximations of the image object obtained by the model-based segmentation means 61 and the data-driven segmentation means 62.
- Either the model or the subset representing these approximations or both of them are adapted or changed in order to obtain a better similarity. It is preferred to adapt the subset.
- a selection means 64 is comprised by the segmentation unit 6 which selects either the (adapted) model or the (adapted) subset as the final segmentation of the image object which will be outputted by the segmentation unit 6.
- This selection may be a predetermined one, as for example always the adapted subset is selected as the final representation of the image object.
- the model will preferably remain unchanged during the processing of the optimization means 63.
- the selection may also be based on properties of the model and the subset as for example the amount of data which is necessary to store or display the model or subset or the extent of adapting of either of them during the optimization.
- Fig. 4 shows a flow process chart illustrating the operation of the embodiment shown in Fig. 3 and a first embodiment of an image processing method according to the present invention.
- a step of pre-processing 31 which may include model training, data acquisition, data improvement (e.g. applying of filters or data compression) and other steps which may be necessary or beneficial to the image processing
- a model-based segmentation 32 and a data-driven-segmentation 33 are preformed.
- the steps 32 and 33 may be performed simultaneously or sequentially.
- a model-based segmentation 32 of the image data is done wherein information is derived from the result of the model-based segmentation and used for or during the data-driven segmentation 33.
- An example of such a deriving of information is the calculation of a threshold value for the data-driven segmentation from the data set of image data and the model representing a first approximation of the image object. It is also possible to perform the data-driven segmentation 33 prior to the model-based segmentation 32 wherein - for example - the obtained subset may be used to estimate orientation, position and size of the image object to be segmented and to use this estimations during the model-based segmentation. Further, a sequence comprising a plurality of segmentations may be performed wherein information derived from results of a model-based segmentation are used for a data-driven segmentation and vice versa.
- a post-processing step 36 follows which may include storing of the subset and/or the model, storing of additional information, displaying or outputting the results of the segmentation and other steps of a image processing method.
- FIG. 5 A schematic block diagram of a second embodiment of an image processing device according to the present invention is shown in Fig. 5 .
- This embodiment corresponds to the embodiment shown in Fig. 3 but comprises further elements.
- User input means 8 are provided so that a user can input data in response to a display of (intermediary) results of the segmentation of the image data by display means 5.
- a segmentation unit 16 of the image processing device comprises model-based segmentation means 161, data-driven segmentation means 162, optimizing means 163 and selecting means 164 similar to the segmentation unit 6 of the embodiment shown in Fig. 3 .
- the segmentation unit 16 comprises a correction means 165 for modifying the methods and/or results of the segmentation according to the user input given via the input means 8.
- the input means 8 may further be utilized for inputting a selecting input according to which the selecting means 164 selects an adapted approximation of the image object as a final segmentation.
- a flow process chart in Fig. 6 illustrates the operation of the embodiment shown in Fig. 5 and a second embodiment of an image processing method according to the present invention.
- the operation and method are similar to the operation and method shown in Fig. 4 .
- a model-based segmentation 32 and a data-driven-segmentation 33 are performed.
- the results of the steps 32 and 33 are processed during an optimizing step 34 during which a similarity value is optimized and the results obtained by the model-based segmentation 32 and the data-driven segmentation 33 are adapted.
- the adapted results are displayed (step 37), so a user can perform a user input 38 as a response to said displaying 36.
- a correction 39 is performed which may induce a re-do of the process starting at a point prior to the display 36.
- the display 36 allows a user to select one of the representations of the image object as the final segmentation.
- a corresponding user input 40 causes the selection 35 of the final segmentation which is followed by a post-processing 36.
- Fig. 7 a schematic block diagram of a third embodiment of an image processing device according to the present invention is shown which comprises elements corresponding to the elements of the embodiment shown in Fig. 3 .
- the segmentation unit 26 of the image processing device comprises means for a model-based segmentation, for a data-driven segmentation, for an optimization and a selection 261 to 264.
- the segmentation unit 26 includes gap detection means 265 and completion means 266. Depending on the used data-driven segmentation method and the optimization there may be gaps in the subset representing an approximation of the image object.
- gaps may -for example - result from artifacts and disturbing elements in the image data or from an exclusion of a part of the subset during an optimization.
- the gap detections means 265 detects these gaps and localizes thus areas which have to be filled in order to obtain a subset of the image data which represents a closed surface.
- the detected gaps are closed by the completion means 266 which for this purpose may utilize information derived the model, in particular topological information. Even without such topological information a completion is possible, for example by an interpolation of the regions of the subset surrounding gap.
- FIG. 8 shows again steps of flow process chart shown in Fig. 4 .
- a model-based segmentation 32 and a data-driven segmentation 33 are performed after a pre-processing step 31 and the results obtained thereby are optimized during an optimizing step 34 followed by a step of gap detection 41.
- the gap detection finds regions of the subset of image data in which the surface or boundary is interrupted. Methods for this gap detection as well as methods for the closing of gaps are well known and will therefore no be further described here in detail.
- Information which can be derived from a model-based segmentation may additionally be used for detecting and closing of gaps in the surface of an object represented by the subset of image data. Once the gaps are closed, a final segmentation is selected during a selecting step 35 from the adapted model and subset.
- a post-processing step 36 follows.
- model-based segmentation and data-driven segmentation i.e. model and subset
- a similarity value can be calculated as follows.
- i n a i n , i : ⁇ j
- the mean Hounsfield value of the adapted model surface ⁇ n differed for the images due to different procedures and timing of contrast medium donation. It ranged between 74 HU and 229 HU (average for all seven images: 134 HU, standard deviation within image was 131 HU on average).
- the iso-surface triangulation of the whole image volume yielded between 0.7 and 8.9 million faces per image at a sampling grid of 2x2x2 voxels. This procedure takes about 10s on a common 2.6GHz PC.
- Fig. 10 shows the resulting surfaces i n a of the adapted subsets with anatomic labels.
- Fig. 11a shows a segmentation of image #1 using ⁇ 1 wherein included coronary fractions are coming close to the auricula.
- Fig. 11 b shows a value of ⁇ 1 + 50 HU.
- Fig. 11c shows the sagittal cut-plane from previous images with both iso-surfaces (light: ⁇ 1 , dark: ⁇ 1 + 50 HU ).
- Making the distance criterion corresponding to a similarity value more loose will include coronaries or arterial structures (especially the right pulmonary artery close to the RUVT and the descending aorta). Also the blood of left ventricle and the ascending aorta will be partially included then. Making the criterion more strict reduces the extent of distal branches of the vein trunks and of the auricula. Reducing it too much will produce holes (see Fig. 9b ). Due to the similarity transformations acquired for the model, also these surfaces could be co-registered with each other by applying T n * - 1 to it. This allows for an overlay of all labelled surface T n * - 1 ⁇ i n i shown in Figs. 13a and 13b . A mean iso-surface i ⁇ i cannot be calculated similarly to v ⁇ and v ⁇ *, because there does not exist a direct correspondence between vertices.
- Figs. 12a and 12b give a rendering from inside the atrium for one of them. Rendering of two patients' surfaces from inside the atrium with a co-registered perspective.
- Fig. 12a shows a common left pulmonary vein trunk that splits up distally into upper and lower trunk wherein Fig. 12b shows two distinct ostia (common type).
- the deformable surface model was however able to adapt also to these images properly in the leave-one-out test, because there is no topological difference between both variants.
- the method succeeded in segmenting and labeling of the four main pulmonary vein trunks in all cases despite the observed anatomical variability.
- Figs. 13 and 13b show, the variability in the present small sample is such that once the surface models are co-registered, all five attached entities (the four main venous trunks as well as the auricula) are spatially separated from each other.
- a shape model was built for the left atrium from seven multi slice computed tomography images by semi-automatic fitting of an initial deformable surface.
- the four main pulmonary vein trunks as well as the auricula were separately labelled in this surface model.
- This model was applied for automatic fitting to the images and labelling of the named anatomical sub-regions in a leave-one-out test. It reached an accuracy of about 1mm.
- the mean Hounsfield value of the fitted model surface was then used for an iso-surface triangulation of each image.
- the results of these two complementary segmentation approaches were then combined with each other by selecting only those iso-surface triangles that are close to the fitted model.
- the final result was a fine representation of the left atrial surface for each image.
- the straight forward iso-surface detection is a powerful means to complement model-based methods with some uncommitted ⁇ ground truth". Especially in case of high resolution images of complicated compound structures like the heart it is expensive to get reliable expert reference segmentations for training and verification of model based methods. As the variability of pulmonary drainage patterns beyond the main four proximal trunks is still too high for such a surface model adaptation approach, the combination with the iso-surface allows a visualisation of these more distal branches, although there was no explicit model for them. Unlike a pure model based approach, anatomical variations and misadaptations become immediately visible in presentations like those in Fig. 10 .
- Fig. 14 a shows a rendering of the iso-surface of a cardiac MSCT image including a left atrium which is covered by and confused with other structures like the ribs (top right), spine (left) and lung vessels.
- Fig. 14b shows a model of the left atrium which is coarsely fitted to the image shown Fig. 14a.
- Fig. 14c shows an overlay of the Figs. 14a and 14b and
- Fig. 14d shows a segmentation of the left atrium according to the present invention wherein the results of the model-based segmentation shown in Fig. 14b and the data-driven segmentation shown in Fig. 14a are combined into a single segmentation of the image object, i.e. of the atrium.
- Fig. 15 shows the embodiment of an image processing device according to the present invention shown in Fig. 3 wherein calculation means 9 for calculating a transformation relation and deformation means 10 for deforming a segmentation by applying said transformation relation to said segmentation are added.
- the starting model which is fitted to data set of image data is compared with the fitted or adjusted model and a transformation is computed which transforms the adjusted or fitted model to the starting model which has the function of a reference model.
- a corresponding transformation is applied to the segmentation or the adjusted subset of image data which allows a comparison of the structure, the shape or the topology of the segmented image object to those of other data sets of image data, i.e. to other images.
- a transformation may be calculated or estimated (rigid, affine or free deformation) and applied to the vertices of a iso-surface triangulation in order to map corresponding anatomic details to each other.
- FIG. 16 An operation flow process chart similar to the one shown in Fig. 4 is shown in Fig. 16 and illustrates the steps of a calculation 43 of the transformation and of a deformation 44 of a segmentation according to this transformation following the selecting step 35 and preceding the step of post-processing 36.
- An image processing device for segmenting a data set of image data and a corresponding image processing method which give better results in terms of accuracy, reliability and robustness than known devices or methods for segmenting image data.
- the advantage of a high robustness of model-based segmentations using global information on the image data like general shapes and the advantage of a high accuracy of data-driven segmentations using local information on the image data like pixel- or voxel-values are combined whereas the disadvantages of both segmentations are avoided.
Abstract
Description
- The present invention relates to an image processing device and a corresponding image processing method for processing a multi-dimensional data set of image data, in particular of three-dimensional medical volume image data, including a segmentation unit for a segmentation of an image object in said data set. Further, the present invention relates to a computer program for implementing said image processing method on a computer.
- In medical imaging different medical imaging modalities are available to provide image data of a region of interest of a patient's body. For instance, X-ray, computer tomography and magnetic resonance imaging are available to assess the condition of the coronary vasculature. Depending on the desired use of such image data, the imaged region of interest and other requirements it is often needed to segment a target organ or, more generally, a target image object from the medical image data, in particular from medical volume image data, wherein it is desired that the segmentation may be performed automatically by a computer without a need for extensive supervision or assistance by a human user.
- In the following - for the purpose of a better understandability - the description will be focused on the processing of cardiac structures. However, it has to be noted that the invention is not limited to processing of only cardiac structures. Any kind of multi-dimensional image data may be segmented according to the invention, including medical and non-medical image data, two-dimensional image data like common pictures, three-dimensional image data like volume image data or a temporal sequence of two-dimensional image data, four-dimensional image data like a temporal sequence of volume image data, and even higher-dimensional image data.
- Several different methods have been proposed for segmentation. These methods may in general be grouped into two categories: model-based segmentation and data-driven segmentation.
- For a model-based segmentation and labeling of cardiac structures prior knowledge is mandatory. Statistical shape models typically capture the shape properties of a training set of target objects. Active shape models that make use of these priors have been used already to segment cardiac chambers in MR and echocardiographic image volumes. Comparably less work was done for multislice computed tomography (MSCT) images. Labelling of a number of target objects was done by non-rigid image-based registration of a labelled image (atlas) to the image of interest (cf. Lorenzo-Valdes, G. Sanchez-Ortiz, R. Mohiaddin, D. Ruckert. Atlas-based segmentation and tracking of 3D cardiac images using non-rigid registration. In: Proc. of MICCAI, 642-650, 2002), by joined adaptation of several models (cf. R. Pilgram, K.D. Fritscher, R. Schubert. Modeling of the geometric variation and analysis of the right atrium and right ventricle motion of the human heart using PCA. In: Proc. of CARS 1108-1113, 2004) or by an explicit compound model.
- All these segmentation methods are not mainly intended to give an accurate border detection reflecting anatomical details, but rather capture the coarse extent of the target organ part. For this purpose only a small number of parameters (some tens to some thousands) are sufficient. This helps to keep this iterative adaptation process fast, but on the other hand restricts the accuracy achievable by all possible shape configurations. When choosing the right number of parameters for the model there is a trade-off between accuracy and generality. Often, a low number of parameters is favored in order to achieve shorter processing times.
- In contrast to the (shape) model-based segmentation methods, there also exist methods less restricted by shape constraints. These data-driven methods utilize solely information included in the data itself. Examples for data-driven segmentation methods may be taken from "Low- and Medium-Level LV Segmentation Techniques", Section 3.3 in Suri, Setarehdan, Singh (Eds.) Advanced Algorithmic Approaches to Medical Image Segmentation, Springer, 2002.
- For example, in contrast-medium enhanced multi-slice computed tomography (MSCT) images of the blood pool in the left atrium and its pulmonary veins can easily be segmented by the application of a threshold for the Hounsfield values. This method reaches a resolution as fine as the voxel spacing, which is about 0.5mm in cardiac MSCT images. Having no prior knowledge about the expected position or shape of the target, this method is not capable of reliably separating it from nearby structures that are also above the Hounsfield threshold.
Fig. 1 gives an impression on such a segmentation. For the atrium these other structures are mainly the blood pool of the left ventricle and the coronaries. The pulmonary veins also come close to the pulmonary arteries, the descending aorta, and the spine. - Data-driven methods are not necessarily restricted by a number of parameters. Usually, either local boundary candidates are detected, or regions of similar voxels are pooled together. Thus, a high accuracy may potentially be reached in the data-driven segmentation result, but there is a vulnerability to image artifacts like noise, partial volume effects etc. Usually, data-driven methods find more boundary candidates than desired for a given object.
- Chen C. W., Luo J., Parker K. J., Huang T. S.: "CT volumetric data-based left ventricle motion estimation: an integrated approach", Computerized Medical Imaging And Graphics, vol. 19, no. 1, 1995, pages 85-100, discloses segmenting 3-dimensional medical volume image data, in which a data-driven segmentation and a model-based segmentation are mutually adapted and iterated until a common, total energy function is minimal.
- It is an object of the present invention to provide an image processing device and a corresponding image processing method which overcome the drawbacks of the known methods described above and allow for a segmentation which is robust in terms of vulnerability to image artifacts and accurate in terms of reflecting anatomical details, wherein the results of the segmentation are achieved within a reasonable, short processing time.
- The object is achieved according to the present invention by an image processing device as claimed in claim 1, comprising:
- model-based segmentation means for performing a model-based segmentation of said data set for obtaining a multi-dimensional model representing a first approximation of said image object;
- data-driven segmentation means for performing a data-driven segmentation of said multi-dimensional data set for obtaining a multi-dimensional subset of said multi-dimensional data set representing a second approximation of said image object;
- optimizing means for adapting said model and/or said subset for optimizing a similarity value between said first and said second approximation; and
- selecting means for selecting said adapted model or said adapted subset as a segmentation of said image object.
- A corresponding image processing method is defined in
Claim 16. A computer program for causing a computer to carry out the steps of the image processing method according to the invention when said computer program is executed on a computer is defined in claim 17. - It has to be noted that the term "subset" is used herein to describe the result of the data-driven segmentation in general. The term "subset" does not only refer to a subset of said data set having substantially the same data structure as the data set, i.e. being a subset in a mathematical or literal sense, but also to any other result or output of the data-driven segmentation representing the image object or an approximation thereof, e.g. a iso-surface triangulation.
- Further, the term "optimizing a similarity value" herein has the meaning of adapting said model and/or said subset in such a way that a desired similarity is achieved or that the similarity value is above or below a predetermined threshold value. The optimum of the similarity value does not have to be a similarity value corresponding to a sameness.
- The present invention is based in the idea that a model-based segmentation and a data-driven segmentation may be performed on the same image data and the results of these segmentations may be used as first approximations of the image object to be segmented. The drawbacks described above are overcome by using both approximations wherein the specific advantages of either method are used to achieve a proper segmentation of the image object. The information derived from the data set of image data by either of the two methods may be used to improve the results of the respective other method.
- Said model-based segmentation may include a mesh-based segmentation, a functional-approximation-based segmentation and/or combinations thereof and said data-driven segmentation may include a pixel-based segmentation, a boundary-based segmentation, a region-based segmentation, a texture-based segmentation, a watershed-segmentation and/or combinations thereof. Other known methods for segmentation may also be used as long as a comparison or measure of similarity between the respective results is possible.
- In a preferred embodiment of the image processing device said data-driven segmentation means is adapted for using information derived by said model-based segmentation means from said model-based segmentation in said data-driven segmentation and/or said model-based segmentation means is adapted for using information derived by said data-driven segmentation means from said data-driven segmentation in said model-based segmentation. Said information derived from said model-based segmentation may preferably include starting values for said data-driven segmentation, threshold values for said data-driven segmentation and/or a region of interest for selecting a subsection of said data set for which said data-driven segmentation is to be performed. Similarly, said information derived from said data-driven segmentation may preferably include a region of interest for selecting a subsection of said data set for which said model-based segmentation is to be performed and/or estimations for position, orientation and/or size of said object as starting values for said model-based segmentation. Since the segmentation methods are applied to the same data set of image data the result of one kind of segmentation or approximation can be used as supporting data for the other kind. For example, once a model-based segmentation is completed and a model representing the image object is obtained, the model may be used for a coarse distinction between voxels of the object and voxels of the background. According to this distinction a threshold value, e.g. a gray value, may be chosen or calculated which can be used for the data-driven segmentation. The estimation of a region of interest permits a omitting of a part of the data set which leads to improved processing times.
- Further, it is possible to derive said information locally. For different areas or regions of the data set of image data different values may be used which allows a more versatile use of the derived information.
- It is advantageous to calculate said similarity value from proximities between said approximations and/or differences between orientations of partial faces of said approximations. In areas in which the model obtained by the model-based segmentation and the subset obtained by the data-driven segmentation are close together, i.e. the distance between the surface of the model and the boundary of the subset is small, and/or comprise partial faces which are oriented similarly both approximations may be considered as rather good whereas in areas in which the model and the subset are far apart and/or have partial faces which are oriented differently at least one of the segmentations may have given wrong results which need correction.
- According to another embodiment of the image processing device according to the present invention said optimizing means is adapted for adapting said model and/or said subset by re-modeling of said model and/or of a part of said model, excluding a part of said subset from said subset and/or including a part of said data set into said subset. In order to improve the similarity value between the two approximations obtained by the different segmentations it may be necessary to repeat one segmentation or both using information derived from the other approximation. Further, it may be decided upon the similarity value between the two approximations to omit some parts of an approximation. For example, all parts of the subset of image data which have a distance to a surface of the model greater than a given value may be deleted from the subset. This will clear the subset from artifacts and objects in the image data which are no part of the object which is to be segmented.
- In a further embodiment of the present invention the image processing device further comprises display means for displaying said multi-dimensional model, said multi-dimensional subset, said similarity value, said adapted model, said adapted subset and/or said derived information; user input means for reading a correction input from a user; and correction means for correcting and/or altering said model-based segmentation, said data-driven segmentation, said model, said subset, said calculation of said similarity value, said optimization of said similarity value, said adapted model, said adapted subset, said derivation of said information and/or said derived information according to said correction input. The display of important data related to the segmentations allows a user to decide on corrections or improvements to the segmentations and to effect them by changing the data which is used for the segmentations or which is obtained from the segmentations via the input means.
- In yet another embodiment the image processing device includes gap detecting means for detecting gaps in said multi-dimensional subset and completion means for closing of detected gaps in said subset. It is possible that the subset of image data includes gaps, i.e. regions in which there should be a closed and continuous boundary. Such gaps could be caused by an incorrect segmentation or by a subsequent optimization in which parts of the subset are deleted because there was a large distance to the surface of the model. If these gaps are closed - for example by interpolation of the surrounding subset - one obtains a subset of image data which gives a better approximation of the image object.
- In a preferred embodiment of the image processing device said completion means is adapted for using closing information derived from said multi-dimensional model, in particular topological closing information. There are many possible ways to close such a gap, for example by including a plane, a convex or a concave boundary. The model includes information on the expected shape of the object, it is therefore possible and advantageous to close the gaps with a filling which corresponds to the respective area of the model, for example to include a convex filling if the gap corresponds to a part of the model which is convex. Of course, more complex filling structures are also possible, too.
- In another preferred embodiment the image processing device comprises display means for displaying said multi-dimensional model, said multi-dimensional subset, said similarity value, said adapted model, said adapted subset and/or said derived information and user input means for reading a selecting input from a user, wherein said selecting means is adapted for selecting said adapted model or said adapted subset according to said selecting input. Thus, a user can select either the adapted model or the adapted subset as the final segmentation of the image object.
- The image processing device of another embodiment comprises calculation means for calculating a transformation relation between said model and a predetermined reference model and deformation means for deforming said adapted subset by applying said transformation relation to said adapted subset. By said model-based segmentation a model is adapted to the present data set of image data by - for example - changing of positions of vertices of a mesh. This change of positions is equivalent to a transformation of a starting model to the model which represents a first approximation of the image object. If this transformation is also applied to the adapted subset the obtained transformed adapted subset may be compared to a reference subset or a reference model in order to map corresponding details, for example anatomical details, to each other. This permits an easier classification of anatomical variations and may also help designing new reference or starting models.
- The invention will now be explained in more detail with reference to the drawings in which
- Fig. 1
- shows an iso-surface triangulation of the blood-pool form an MSCT image sub-volume;
- Fig. 2
- a shows a left atrium surface model with anatomical labels;
- Fig. 2b
- shows the surface model of
Fig. 2a averaged over a number of sample objects; - Fig. 3
- shows a schematic block diagram of a first embodiment of an image processing device according to the present invention;
- Fig. 4
- shows a flow process chart illustrating the operation of the embodiment shown in
Fig. 3 and a first embodiment of an image processing method according to the present invention; - Fig. 5
- shows a schematic block diagram of a second embodiment of an image processing device according to the present invention;
- Fig. 6
- shows a flow process chart illustrating the operation of the embodiment shown in
Fig. 5 and a second embodiment of an image processing method according to the present invention; - Fig. 7
- shows a schematic block diagram of a third embodiment of an image processing device according to the present invention;
- Fig. 8
- shows a flow process chart illustrating the operation of the embodiment shown in
Fig. 7 and a third embodiment of an image processing method according to the present invention; - Figs. 9a, 9b
- show an iso-surface with a two different strict distance measures;
- Fig. 10
- shows labeled surfaces of the left atrium with pulmonary vein trunks and auricula of seven patients;
- Fig. 11a, 11b
- show two segmentations of image #1 of
Fig. 10 ; - Fig. 11c
- shows the sagittal cut-plane form the segmentations of
Figs. 11a, 11b ; - Fig. 12a, 12b
- show two renderings of two patients' surfaces from inside the atrium with a co-registered perspective;
- Fig. 13a, 13b
- show an overlay of the labeled and co-registered surfaces of the images of
Fig. 10 from two different perspectives; - Fig. 14a
- shows a rendering of the iso-surface of a cardiac MSCT image including a left atrium;
- Fig. 14b
- shows a model of the left atrium;
- Fig. 14c
- shows an overlay of the
Figs. 14a and 14b ; and - Fig. 14d
- shows a segmentation of the left atrium according to the present invention.
-
Fig. 1 shows an iso-surface triangulation of the blood-pool from an MSCT image sub-volume. This iso-surface triangulation has been segmented by the application of a threshold for the Hounsfield values. The resolution which can be achieved may be as fine as the voxel spacing, which is about 0.5 mm in cardiac MSCT images. As can be seen fromFig. 1 not only the object of interest, the atrium in this case, but also nearby structures which are also above the Hounsfield threshold are included in this subset of image data. In the center ofFig. 1 the left auricula is clearly visible, below it the coronaries, also both left pulmonary vein branches and the left branch of pulmonary arteries. - In the following procedures for an exemplary model-based segmentation and an exemplary data-driven segmentation are described.
- If no accurately manually segmented data is available for the model building, a bootstrap approach for consecutive model building and refinement may be followed. For a dedicated sample image the basic cardiac structures of interest (the atrium body, the atrial appendage (atrial auricula), the left upper pulmonary vein trunk (LUPVT), the left lower PVT, the right upper PVT, and the right lower PVT) were fitted with triangular surface models starting from geometric primitives (spheres and tubes) using the methods explained below. A combined surface model a 0 having about V=2500 vertices v 0,i and T=4800 triangles t 0, i was generated from them where each triangle t 0,i is attached a label li . The resulting surface model is shown in
Fig. 2a . The combination of the fitted primitives to a single connected surface is achieved by a method taken from J. von Berg, C. Lorenz. Multi-surface cardiac modelling, segmentation, and tracking. Functional Imaging and Modeling of the Heart, 3rd International Workshop 2005, submitted. - The vertices v 0 of this model a 0 were adapted to a set of N=7 further training images (cf.
Fig. 10 ). As only the vertex positions are changed by the fitting, it may be denoted as An : v 0 → vn , n =1...N yielding the corresponding triangles tn . - An may be subdivided into three single transformations An = DnAPn. Pn (pose) has a translation, rotation, and anisotropic scaling component and was approximated interactively until Pnv 0 visually agreed well with the image. A is an automatically performed free form deformation as explained below. In some cases, additional local deformations Dn were required performed with a method described in H. Timinger, V. Pekar, J. von Berg, K. Dietmeyer, M. Kaus. Integration of interactive corrections to model-based segmentation algorithms. In: Proc. Bildverarbeitung fuer die Medizin 11-15, 2003. A mean surface model v̅ was calculated in the co-ordinate system of v 0 by
- The similarity transformation Tn : vn → v 0 (rigid transformation plus isotropic scaling, given in a homogeneous co-ordinate system) was estimated for each image based on the distance vectors between all corresponding vertices (cf. G. Golub, C. van Loan. Matrix computation 3rd edition. John's Hopkins University Press, Baltimore USA, 1996).
- For a leave-one-out test, also a set of dedicated mean models
were generated. EachFig. 2b . - The automatic free form shape deformation A : v 0 →vn from the previous sections uses an iterative shape-based deformable model method (cf. T. McInerney, D. Terzopoulos. Deformable models in medical image analysis: A survey. Medical Image Analysis 1(2), 91-108, 1996; or J. Weese, M. Kaus, C. Lorenz, S. Lobregt, R. Truyen, V. Pekar. Shape constrained deformable models for 3D medical image segmentation. In: Proc. IPMI, 380-387, 2001). Each iteration is performed by minimisation of the energy term
- The internal energy E int maintains consistency of the deformable surface with the initial configuration v0 by
where N(j) gives the indices of all vertices that are neighbours of vj. R is a similarity transformation matrix that is determined by a closed-form point-based registration method prior to the energy calculation to make the energy calculation invariant to pose and scaling. - The external energy is given by
where ev̅ I , is the unit vector in the direction of the image gradient at the triangle barycentre ci . The parameter D controls the trade-off between feature strength and distance. c̃i is the best feature point for ci in the image. It is determined by
where l is the maximum search distance along the triangle normal ni and δ is the search step size. The feature function
damps the influence of the gradient strength beyond g max and restricts search to regions of a given grey value range. More details about this automatic shape deformation method can be found in J. Weese, M. Kaus, C. Lorenz, S. Lobregt, R. Truyen, V. Pekar. Shape constrained deformable models for 3D medical image segmentation. In: Proc. IPMI, 380-387, 2001 or M.R. Kaus, J. von Berg, J. Weese, W. Niessen, V. Pekar. Automated segmentation of the left ventricle in cardiac MRI. Med. Img. Anal. 8 245-254, 2004. The parameter setting was - The used sample images were acquired at different hospitals, but all with Philips MX8000 IDT 16-line scanners. They were contrast enhanced as they were acquired for the purpose of coronary artery assessment on patients with cardiac complaints. The threshold value was automatically set to the average Hounsfield value of the fitted surface model given by
where - The block diagram of the image processing device according to the invention illustrated in
Fig. 3 schematically shows an object 1, for instance a patient, from which image data, in particular volume image data, can be acquired by adata acquisition unit 2, which can generally be any medical imaging device, such as a computer tomograph, a magnetic resonance imaging unit, an X-ray device or an ultrasound unit. The acquired data are generally stored in amemory 3, such as a hard disc of a workstation or a server in a medical computer network. Generally an image processing device includes areconstruction unit 4 for reconstructing images from the acquired image data for display on a display means 5. Furthermore, for segmentation of an (target) object in the image data asegmentation unit 6 is provided. - The
segmentation unit 6 comprises a model-based segmentation means 61 which provides a model-based segmentation of the acquired data set stored in thememory 3. A model-based segmentation utilizes a priori knowledge on structures of the image data in the data set, for example the knowledge that an object like a atrium commonly has a special, characteristic shape and size. There may be aberrances or anomalies or even a range of possible variations which have to be included into the modeling. In order to permit a training of the model using well segmented training image sets and an improvement of the information and knowledge used for the model-based segmentation amodeling unit 7 is included into the image processing device. - A number of suitable model-based segmentation methods is commonly known including mesh-based segmentation and functional-approximation-based segmentation and there is no need for a further description of the model-based segmentation as such herein (cf. Model-Based Recognition Methods for LV Modeling, Section 3.4 in: Suri, Setarehdan, Singh (Eds.) Advanced Algorithmic Approaches to Medical Image Segmentation, Springer 2002).
- Further, a data-driven segmentation means 62 is provided for a data-driven segmentation of the acquired data set stored in the
memory 3. An example of such a data-driven segmentation method is a method applying a threshold for the Hounsfield values and a subsequent iso-surface triangulation as described above. It is not necessary to perform such a triangulation since there are other suitable known methods to permit a comparison between a model representing an image object and a data subset representing the same object. The surfaces of the object may also be approximated by any other mathematical means that allows for a similarity measure. Spherical harmonics are used for this purpose or spline-based approximations. Both allow for a distance measure between two such surfaces. - Again, it has to be noted that the term "subset" used herein is not limited to a subset of the data set of image data in a literal sense but also includes other representations of the image object or of such a subset which are obtained by a data-driven segmentation.
- The
segmentation unit 6 further comprises an optimization means 63 for an optimization of a similarity value between the two approximations of the image object obtained by the model-based segmentation means 61 and the data-driven segmentation means 62. Either the model or the subset representing these approximations or both of them are adapted or changed in order to obtain a better similarity. It is preferred to adapt the subset. - Also, a selection means 64 is comprised by the
segmentation unit 6 which selects either the (adapted) model or the (adapted) subset as the final segmentation of the image object which will be outputted by thesegmentation unit 6. This selection may be a predetermined one, as for example always the adapted subset is selected as the final representation of the image object. In this case the model will preferably remain unchanged during the processing of the optimization means 63. The selection may also be based on properties of the model and the subset as for example the amount of data which is necessary to store or display the model or subset or the extent of adapting of either of them during the optimization. -
Fig. 4 shows a flow process chart illustrating the operation of the embodiment shown inFig. 3 and a first embodiment of an image processing method according to the present invention. After a step of pre-processing 31 which may include model training, data acquisition, data improvement (e.g. applying of filters or data compression) and other steps which may be necessary or beneficial to the image processing a model-basedsegmentation 32 and a data-driven-segmentation 33 are preformed. Thesteps segmentation 32 of the image data is done wherein information is derived from the result of the model-based segmentation and used for or during the data-drivensegmentation 33. An example of such a deriving of information is the calculation of a threshold value for the data-driven segmentation from the data set of image data and the model representing a first approximation of the image object. It is also possible to perform the data-drivensegmentation 33 prior to the model-basedsegmentation 32 wherein - for example - the obtained subset may be used to estimate orientation, position and size of the image object to be segmented and to use this estimations during the model-based segmentation. Further, a sequence comprising a plurality of segmentations may be performed wherein information derived from results of a model-based segmentation are used for a data-driven segmentation and vice versa. - When both, model and subset, are obtained from the segmentation steps 32 and 33, the two approximations are compared by calculating a similarity value and adaptations are applied to model, subset or model and subset in order to optimize said similarity value during an
optimization step 34 after which one adapted approximation, i.e. model or subset, is selected in aselection step 35. After the selection step 35 apost-processing step 36 follows which may include storing of the subset and/or the model, storing of additional information, displaying or outputting the results of the segmentation and other steps of a image processing method. - A schematic block diagram of a second embodiment of an image processing device according to the present invention is shown in
Fig. 5 . This embodiment corresponds to the embodiment shown inFig. 3 but comprises further elements. User input means 8 are provided so that a user can input data in response to a display of (intermediary) results of the segmentation of the image data by display means 5. Asegmentation unit 16 of the image processing device comprises model-based segmentation means 161, data-driven segmentation means 162, optimizing means 163 and selectingmeans 164 similar to thesegmentation unit 6 of the embodiment shown inFig. 3 . Further, thesegmentation unit 16 comprises a correction means 165 for modifying the methods and/or results of the segmentation according to the user input given via the input means 8. The input means 8 may further be utilized for inputting a selecting input according to which the selecting means 164 selects an adapted approximation of the image object as a final segmentation. - A flow process chart in
Fig. 6 illustrates the operation of the embodiment shown inFig. 5 and a second embodiment of an image processing method according to the present invention. The operation and method are similar to the operation and method shown inFig. 4 . After a pre-processing step 31 a model-basedsegmentation 32 and a data-driven-segmentation 33 are performed. The results of thesteps step 34 during which a similarity value is optimized and the results obtained by the model-basedsegmentation 32 and the data-drivensegmentation 33 are adapted. The adapted results are displayed (step 37), so a user can perform a user input 38 as a response to said displaying 36. According to the user input 38 acorrection 39 is performed which may induce a re-do of the process starting at a point prior to thedisplay 36. Further, thedisplay 36 allows a user to select one of the representations of the image object as the final segmentation. Acorresponding user input 40 causes theselection 35 of the final segmentation which is followed by a post-processing 36. - In
Fig. 7 a schematic block diagram of a third embodiment of an image processing device according to the present invention is shown which comprises elements corresponding to the elements of the embodiment shown inFig. 3 . As shown inFig. 3 forsegmentation unit 6, thesegmentation unit 26 of the image processing device comprises means for a model-based segmentation, for a data-driven segmentation, for an optimization and aselection 261 to 264. In addition, thesegmentation unit 26 includes gap detection means 265 and completion means 266. Depending on the used data-driven segmentation method and the optimization there may be gaps in the subset representing an approximation of the image object. These gaps may -for example - result from artifacts and disturbing elements in the image data or from an exclusion of a part of the subset during an optimization. The gap detections means 265 detects these gaps and localizes thus areas which have to be filled in order to obtain a subset of the image data which represents a closed surface. The detected gaps are closed by the completion means 266 which for this purpose may utilize information derived the model, in particular topological information. Even without such topological information a completion is possible, for example by an interpolation of the regions of the subset surrounding gap. - The operation of the embodiment shown in
Fig. 7 and a third embodiment of an image processing method according to the present invention are illustrated by the flow process chart shown inFig. 8 . Similar toFig. 6 Fig 8 shows again steps of flow process chart shown inFig. 4 . A model-basedsegmentation 32 and a data-drivensegmentation 33 are performed after apre-processing step 31 and the results obtained thereby are optimized during an optimizingstep 34 followed by a step ofgap detection 41. The gap detection finds regions of the subset of image data in which the surface or boundary is interrupted. Methods for this gap detection as well as methods for the closing of gaps are well known and will therefore no be further described here in detail. - Information which can be derived from a model-based segmentation may additionally be used for detecting and closing of gaps in the surface of an object represented by the subset of image data. Once the gaps are closed, a final segmentation is selected during a selecting
step 35 from the adapted model and subset. Apost-processing step 36 follows. - In the following, a selection and labeling of iso-surface faces is described being a part of the
optimization step 34. If the results of both, model-based segmentation and data-driven segmentation, i.e. model and subset, are represented by a triangular mesh they can easily be compared and a similarity value can be calculated as follows. -
- The distance measure was
where
was assigned to - Results from an implementation of the invention using seven sample data sets of image data may be summarized as follows:
- With about 15s time the automatic fitting of a model is comparably fast. Taking the iso-surface triangulation as comparison data, the method described above in regard of the distance measure can be applied to also assess the model-based segmentation results
Figs. 9a and 9b shows the results for ε=2mm (Fig. 9a ) and ε=1mm (Fig. 9b ) (always ω=0mm) forimage # 5 ofFig. 10 . It follows that the predominant part of the iso-surface is closer than 2mm to at least one of the model triangle centres in Cartesian coordinates, and a large part is even closer than 1mm. Compared to the other six images, this one turned out to have a rather bad accuracy.Fig. 9b clearly illustrates how an adaptation of the subset by excluding parts which are too far apart from the model surface can produce holes or gaps in the boundary of the subset. - The mean Hounsfield value of the adapted model surface θn differed for the images due to different procedures and timing of contrast medium donation. It ranged between 74 HU and 229 HU (average for all seven images: 134 HU, standard deviation within image was 131 HU on average). The iso-surface triangulation of the whole image volume yielded between 0.7 and 8.9 million faces per image at a sampling grid of 2x2x2 voxels. This procedure takes about 10s on a common 2.6GHz PC.
Fig. 10 shows the resulting surfacesFigs. 11a to 11c shows both variants applied to image #1 for comparison. -
Fig. 11a shows a segmentation of image #1 using θ1 wherein included coronary fractions are coming close to the auricula. When a value of θ1 + 50HU is applied instead (Fig. 11 b) , the surface slightly shrinks and the coronaries are now disconnected from the left atrium.Fig. 11c shows the sagittal cut-plane from previous images with both iso-surfaces (light: θ1, dark: θ1 + 50HU). - Making the distance criterion corresponding to a similarity value more loose will include coronaries or arterial structures (especially the right pulmonary artery close to the RUVT and the descending aorta). Also the blood of left ventricle and the ascending aorta will be partially included then. Making the criterion more strict reduces the extent of distal branches of the vein trunks and of the auricula. Reducing it too much will produce holes (see
Fig. 9b ). Due to the similarity transformations acquired for the model, also these surfaces could be co-registered with each other by applyingFigs. 13a and 13b . A mean iso-surfacei̅ i cannot be calculated similarly to v̅ and v̅*, because there does not exist a direct correspondence between vertices. -
- The variability of the pulmonary drainage patterns is remarkably high. In the classification scheme proposed in E.M. Marom, J.E. Herndon, Y.H. Kim, H.P. McAdams. Variations in pulmonary venous drainage to the left atrium: Implications for radiofrequency ablation. Radiology 230(3), 824-829, 2004 only 86% of all examined patients had a normal left pulmonary vein pattern (two ostia), and 71 % had a normal right pulmonary vein pattern. In our present sample there were three of seven subjects (#5, #6, #7) with a single left ostium that splits up into LUVT and LLVT more than one centimeter distal from the ostium (classified as exceptional type L1b by Marom et al.).
Figs. 12a and 12b give a rendering from inside the atrium for one of them. Rendering of two patients' surfaces from inside the atrium with a co-registered perspective.Fig. 12a shows a common left pulmonary vein trunk that splits up distally into upper and lower trunk whereinFig. 12b shows two distinct ostia (common type). - The deformable surface model was however able to adapt also to these images properly in the leave-one-out test, because there is no topological difference between both variants. Thus, the method succeeded in segmenting and labeling of the four main pulmonary vein trunks in all cases despite the observed anatomical variability. As
Figs. 13 and 13b show, the variability in the present small sample is such that once the surface models are co-registered, all five attached entities (the four main venous trunks as well as the auricula) are spatially separated from each other. - A shape model was built for the left atrium from seven multi slice computed tomography images by semi-automatic fitting of an initial deformable surface. The four main pulmonary vein trunks as well as the auricula were separately labelled in this surface model. This model was applied for automatic fitting to the images and labelling of the named anatomical sub-regions in a leave-one-out test. It reached an accuracy of about 1mm. The mean Hounsfield value of the fitted model surface was then used for an iso-surface triangulation of each image. The results of these two complementary segmentation approaches were then combined with each other by selecting only those iso-surface triangles that are close to the fitted model. The final result was a fine representation of the left atrial surface for each image. A co-registration of the different atrial surfaces was possible by the similarity transformations estimated for their models. Also anatomical labels could be transferred from the model to the iso-surface. Once a surface model like v̅ is available, the procedure works automatically and yields a representation for a given MSCT image like in
Figs. 11a and 11b in about 1 min time on a common 2.6GHz PC excluding hard drive access. - Using a model-based approach to select dedicated anatomical structures from the fine and detailed iso-surface turns out to be a suitable means towards more automated presentation of relevant structures only. This may become a pre-requisite both for more efficient presentation to a physician for a given dedicated purpose, but also for subsequent procedures in cardiac diagnostics and treatment. A focus was put herein on the left atrium that is gaining more and more attention in the context of image guided atrial defibrillation.
- The straight forward iso-surface detection is a powerful means to complement model-based methods with some uncommitted ``ground truth". Especially in case of high resolution images of complicated compound structures like the heart it is expensive to get reliable expert reference segmentations for training and verification of model based methods. As the variability of pulmonary drainage patterns beyond the main four proximal trunks is still too high for such a surface model adaptation approach, the combination with the iso-surface allows a visualisation of these more distal branches, although there was no explicit model for them. Unlike a pure model based approach, anatomical variations and misadaptations become immediately visible in presentations like those in
Fig. 10 . - Some steps of the segmentation according to the present invention are illustrated in
Fig. 14 a to 14d.Fig. 14a shows a rendering of the iso-surface of a cardiac MSCT image including a left atrium which is covered by and confused with other structures like the ribs (top right), spine (left) and lung vessels.Fig. 14b shows a model of the left atrium which is coarsely fitted to the image shownFig. 14a. Fig. 14c shows an overlay of theFigs. 14a and 14b and -
Fig. 14d shows a segmentation of the left atrium according to the present invention wherein the results of the model-based segmentation shown inFig. 14b and the data-driven segmentation shown inFig. 14a are combined into a single segmentation of the image object, i.e. of the atrium. -
Fig. 15 shows the embodiment of an image processing device according to the present invention shown inFig. 3 wherein calculation means 9 for calculating a transformation relation and deformation means 10 for deforming a segmentation by applying said transformation relation to said segmentation are added. The starting model which is fitted to data set of image data is compared with the fitted or adjusted model and a transformation is computed which transforms the adjusted or fitted model to the starting model which has the function of a reference model. A corresponding transformation is applied to the segmentation or the adjusted subset of image data which allows a comparison of the structure, the shape or the topology of the segmented image object to those of other data sets of image data, i.e. to other images. For all of the different images the same template of a deformable model may be used having the same number of triangles and the same topology. Thus, a simple correspondence between different results of the model-based segmentation exists. Therefore, a transformation may be calculated or estimated (rigid, affine or free deformation) and applied to the vertices of a iso-surface triangulation in order to map corresponding anatomic details to each other. - An operation flow process chart similar to the one shown in
Fig. 4 is shown inFig. 16 and illustrates the steps of acalculation 43 of the transformation and of adeformation 44 of a segmentation according to this transformation following the selectingstep 35 and preceding the step ofpost-processing 36. - An image processing device for segmenting a data set of image data and a corresponding image processing method are proposed which give better results in terms of accuracy, reliability and robustness than known devices or methods for segmenting image data. The advantage of a high robustness of model-based segmentations using global information on the image data like general shapes and the advantage of a high accuracy of data-driven segmentations using local information on the image data like pixel- or voxel-values are combined whereas the disadvantages of both segmentations are avoided.
Claims (17)
- Image processing device for processing a multi-dimensional data set of image data, in particular of three-dimensional medical volume image data, including a segmentation unit (6, 16, 26) for a segmentation of an image object in said data set, said segmentation unit (6, 16, 26) comprising:- model-based segmentation means (61, 161, 261) for performing a model-based segmentation (32) of said data set for obtaining a multi-dimensional model representing a first approximation of said image object;- data-driven segmentation means (62, 162, 262) for performing a data-driven segmentation (33) of said multi-dimensional data set for obtaining a multi-dimensional subset of said multi-dimensional data set representing a second approximation of said image object;- optimizing means (63, 163, 263) for adapting said model and/or said subset for optimizing a similarity value between said first and said second approximation; and- selecting means (64, 164, 264) for selecting said adapted model or said adapted subset as a segmentation of said image object.
- Image processing device as claimed in Claim 1,
wherein said model-based segmentation means (61, 161, 261) is adapted for performing a mesh-based segmentation, a functional-approximation-based segmentation and/or combinations thereof. - Image processing device as claimed in Claim 1,
wherein said data-driven segmentation means (62, 162, 262) is adapted for performing a pixel-based segmentation, a boundary-based segmentation, a region-based segmentation, a texture-based segmentation, a watershed-segmentation and/or combinations thereof. - Image processing device as claimed in Claim 1,
wherein said data-driven segmentation means (62, 162, 262) is adapted for using information derived by said model-based segmentation means (61, 161, 261) from said model-based segmentation (32) in said data-driven segmentation (33). - Image processing device as claimed in Claim 4,
wherein said information derived from said model-based segmentation (32) includes starting values for said data-driven segmentation (33), threshold values for said data-driven segmentation (33) and/or a region of interest for selecting a subsection of said data set for which said data-driven segmentation (33) is to be performed. - Image processing device as claimed in Claim 1,
wherein said model-based segmentation means (61, 161, 261) is adapted for using information derived by said data-driven segmentation means (62, 162, 262) from said data-driven segmentation (33) in said model-based segmentation (32). - Image processing device as claimed in Claim 6,
wherein said information derived from said data-driven segmentation (33) includes a region of interest for selecting a subsection of said data set for which said model-based segmentation (32) is to be performed and/or estimations for position, orientation and/or size of said object as starting values for said model-based segmentation (32). - Image processing device as claimed in Claim 4 or Claim 6,
wherein said information is derived locally. - Image processing device as claimed in Claim 1,
wherein said similarity value is calculated from proximities between said approximations and/or differences between orientations of partial faces of said approximations. - Image processing device as claimed in Claim 1,
wherein said optimizing means (63, 163, 263) is adapted for adapting said model and/or said subset by re-modeling of said model and/or of a part of said model, excluding a part of said subset from said subset and/or including a part of said data set into said subset. - Image processing device as claimed in Claim 1,
further comprising- display means (5) for displaying said multi-dimensional model, said multi-dimensional subset, said similarity value, said adapted model, said adapted subset and/or said derived information;- user input means (8) for reading a correction input from a user; and- correction means (165) for correcting and/or altering said model-based segmentation, said data-driven segmentation, said model, said subset, said calculation of said similarity value, said optimization of said similarity value, said adapted model, said adapted subset, said derivation of said information and/or said derived information according to said correction input. - Image processing device as claimed in Claim 1,
further comprising- gap detecting means (265) for detecting gaps in said multi-dimensional subset; and- completion means (266) for closing of detected gaps in said subset. - Image processing device as claimed in Claim 12,
wherein said completion means (266) is adapted for using closing information derived from said multi-dimensional model, in particular topological closing information. - Image processing device as claimed in Claim 1,
further comprising- display means (5) for displaying said multi-dimensional model, said multi-dimensional subset, said similarity value, said adapted model, said adapted subset and/or said derived information and- user input means (8) for reading a selecting input from a user, wherein said selecting means (64, 164, 264) is adapted for selecting said adapted model or said adapted subset according to said selecting input. - Image processing device as claimed in Claim 1,
further comprising- calculation means (9) for calculating a transformation relation between said model and a predetermined reference model; and- deformation means (10) for deforming said adapted subset by applying said transformation relation to said adapted subset. - Image processing method for processing a multi-dimensional data set of image data, in particular of three-dimensional medical volume image data, including a segmentation step for a segmentation of an image object in said data set, said segmentation step comprising:- a model-based segmentation step (32) for performing a model-based segmentation of said data set for obtaining a multi-dimensional model representing a first approximation of said image object;- a data-driven segmentation step (33) for performing a data-driven segmentation of said multi-dimensional data set for obtaining a multi-dimensional subset of said multi-dimensional data set representing a second approximation of said image object;- an optimizing step (34) for adapting said model and/or said subset for optimizing a similarity value between said first and said second approximation; and- a selecting step (35) for selecting said adapted model or said adapted subset as a segmentation of said image object.
- Computer program comprising program code means for causing a computer to perform the steps of the method as claimed in claim 16, when said computer program is executed on a computer.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
EP06710848A EP1851722B8 (en) | 2005-02-11 | 2006-02-08 | Image processing device and method |
Applications Claiming Priority (3)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
EP05101009 | 2005-02-11 | ||
EP06710848A EP1851722B8 (en) | 2005-02-11 | 2006-02-08 | Image processing device and method |
PCT/IB2006/050403 WO2006085268A1 (en) | 2005-02-11 | 2006-02-08 | Image processing device and method |
Publications (3)
Publication Number | Publication Date |
---|---|
EP1851722A1 EP1851722A1 (en) | 2007-11-07 |
EP1851722B1 true EP1851722B1 (en) | 2011-08-10 |
EP1851722B8 EP1851722B8 (en) | 2012-03-14 |
Family
ID=36609259
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
EP06710848A Active EP1851722B8 (en) | 2005-02-11 | 2006-02-08 | Image processing device and method |
Country Status (5)
Country | Link |
---|---|
US (1) | US7957572B2 (en) |
EP (1) | EP1851722B8 (en) |
JP (1) | JP4918048B2 (en) |
AT (1) | ATE520101T1 (en) |
WO (1) | WO2006085268A1 (en) |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2022084074A1 (en) * | 2020-10-20 | 2022-04-28 | Koninklijke Philips N.V. | Detecting anatomical abnormalities by segmentation results with and without shape priors |
Families Citing this family (41)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US8050473B2 (en) * | 2007-02-13 | 2011-11-01 | The Trustees Of The University Of Pennsylvania | Segmentation method using an oriented active shape model |
DE102007015230A1 (en) * | 2007-03-29 | 2008-10-02 | Siemens Ag | Bildverarbeitungssytem for an X-ray system or the like |
DE102007024154A1 (en) * | 2007-05-24 | 2008-11-27 | Siemens Ag | Method for automatic selection of a display mode for an image data record of an organ to be examined |
CN102099833B (en) * | 2008-04-07 | 2016-08-10 | 皇家飞利浦电子股份有限公司 | Mesh collision is avoided |
JP5514195B2 (en) * | 2008-05-09 | 2014-06-04 | コーニンクレッカ フィリップス エヌ ヴェ | Device for generating images of moving objects |
US8786594B2 (en) * | 2008-05-30 | 2014-07-22 | Siemens Aktiengesellschaft | Method and system for rendering a medical image |
CN102187368B (en) * | 2008-10-14 | 2015-07-15 | 皇家飞利浦电子股份有限公司 | One-click correction of tumor segmentation results |
EP2189945A1 (en) * | 2008-11-21 | 2010-05-26 | A&P ip B.V. | Method of and arrangement for linking image coordinates to coordinates of reference model |
US8478012B2 (en) * | 2009-09-14 | 2013-07-02 | General Electric Company | Methods, apparatus and articles of manufacture to process cardiac images to detect heart motion abnormalities |
US8422753B2 (en) * | 2009-10-22 | 2013-04-16 | Siemens Corporation | Method and system for automatic extraction of personalized left atrium models |
GB201002855D0 (en) * | 2010-02-19 | 2010-04-07 | Materialise Dental Nv | Method and system for achiving subject-specific, three-dimensional information about the geometry of part of the body |
US8861891B2 (en) * | 2010-03-05 | 2014-10-14 | Siemens Aktiengesellschaft | Hierarchical atlas-based segmentation |
WO2012035488A1 (en) * | 2010-09-17 | 2012-03-22 | Koninklijke Philips Electronics N.V. | Choosing anatomical variant model for image segmentation |
EP2642923B1 (en) * | 2010-11-23 | 2017-03-22 | ClaroNav Inc. | Method and apparatus for automated registration and pose tracking |
CN103732134B (en) | 2010-12-29 | 2016-08-17 | 迪亚卡帝奥有限公司 | System, device, equipment and method for automatic Assessment of left ventricular function |
US8644576B2 (en) * | 2011-03-09 | 2014-02-04 | Siemens Aktiengesellschaft | Method and system for multi-part left atrium segmentation in C-arm computed tomography volumes using shape constraints |
EP2777034B1 (en) * | 2011-11-08 | 2018-05-30 | Koninklijke Philips N.V. | Interacting with a three-dimensional object dataset |
US8724881B2 (en) * | 2011-11-09 | 2014-05-13 | Siemens Aktiengesellschaft | Method and system for precise segmentation of the left atrium in C-arm computed tomography volumes |
US9439627B2 (en) | 2012-05-22 | 2016-09-13 | Covidien Lp | Planning system and navigation system for an ablation procedure |
US8750568B2 (en) | 2012-05-22 | 2014-06-10 | Covidien Lp | System and method for conformal ablation planning |
US9439622B2 (en) | 2012-05-22 | 2016-09-13 | Covidien Lp | Surgical navigation system |
US9498182B2 (en) | 2012-05-22 | 2016-11-22 | Covidien Lp | Systems and methods for planning and navigation |
US9439623B2 (en) | 2012-05-22 | 2016-09-13 | Covidien Lp | Surgical planning system and navigation system |
WO2014164992A1 (en) * | 2013-03-13 | 2014-10-09 | Miller David G | Coregistered intravascular and angiographic images |
US9761006B2 (en) * | 2013-06-28 | 2017-09-12 | Koninklijke Philips N.V. | Methods of utilizing image noise information |
EP3077989B1 (en) | 2013-12-04 | 2022-11-09 | Koninklijke Philips N.V. | Model-based segmentation of an anatomical structure. |
JP2016539701A (en) * | 2013-12-11 | 2016-12-22 | コーニンクレッカ フィリップス エヌ ヴェKoninklijke Philips N.V. | Volumetric image data processing based on three-dimensional (3D) prescan |
EP3120323B1 (en) * | 2014-03-21 | 2020-05-20 | Koninklijke Philips N.V. | Image processing apparatus and method for segmenting a region of interest |
US11172860B2 (en) | 2014-05-06 | 2021-11-16 | Peacs Investments B.V. | Estimating distribution fluctuation and/or movement of electrical activity through a heart tissue |
US11289207B2 (en) | 2015-07-09 | 2022-03-29 | Peacs Investments B.V. | System for visualizing heart activation |
DE112016005742T5 (en) | 2015-12-15 | 2018-10-04 | Koninklijke Philips N.V. | Data processing method for computed tomography |
CN105719278B (en) * | 2016-01-13 | 2018-11-16 | 西北大学 | A kind of medical image cutting method based on statistics deformation model |
US11458320B2 (en) | 2016-09-06 | 2022-10-04 | Peacs Investments B.V. | Method of cardiac resynchronization therapy |
CN108629800A (en) * | 2017-03-20 | 2018-10-09 | 北京三星通信技术研究有限公司 | Plane determines that method and augmented reality show the display methods of information, related device |
US10685172B2 (en) * | 2018-05-24 | 2020-06-16 | International Business Machines Corporation | Generating a textual description of an image using domain-independent anomaly analysis |
US10937172B2 (en) | 2018-07-10 | 2021-03-02 | International Business Machines Corporation | Template based anatomical segmentation of medical images |
EP3833290A1 (en) | 2018-08-10 | 2021-06-16 | Covidien LP | Systems for ablation visualization |
WO2020091096A1 (en) * | 2018-10-30 | 2020-05-07 | Samsung Electronics Co., Ltd. | Methods for determining a planes, methods for displaying augmented reality display information and corresponding devices |
EP4222716A1 (en) | 2020-09-29 | 2023-08-09 | Koninklijke Philips N.V. | Rendering of b-mode images based on tissue differentiation |
CN114066913B (en) * | 2022-01-12 | 2022-04-22 | 广东工业大学 | Heart image segmentation method and system |
CN114419020B (en) * | 2022-01-26 | 2022-10-18 | 深圳大学 | Medical image segmentation method, medical image segmentation device, computer equipment and storage medium |
Family Cites Families (9)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5891030A (en) * | 1997-01-24 | 1999-04-06 | Mayo Foundation For Medical Education And Research | System for two dimensional and three dimensional imaging of tubular structures in the human body |
JP2000105838A (en) * | 1998-09-29 | 2000-04-11 | Toshiba Corp | Image display method and image processor |
AU2001251539A1 (en) * | 2000-04-11 | 2001-10-23 | Cornell Research Foundation Inc. | System and method for three-dimensional image rendering and analysis |
WO2002080110A1 (en) * | 2001-03-29 | 2002-10-10 | Koninklijke Philips Electronics N.V. | Image processing method for fitness estimation of a 3d mesh model mapped onto a 3d surface of an object |
JP3902765B2 (en) * | 2001-05-17 | 2007-04-11 | シーメンス コーポレイト リサーチ インコーポレイテツド | Various approaches to left ventricular segmentation in MR heart images |
WO2004053792A1 (en) | 2002-12-12 | 2004-06-24 | Philips Intellectual Property & Standards Gmbh | Method of segmenting a three-dimensional data set allowing user corrections |
EP1437685A3 (en) * | 2003-01-13 | 2004-07-21 | Philips Intellectual Property & Standards GmbH | Method for segmenting three-dimensional structures |
EP1588325B1 (en) | 2003-01-15 | 2006-10-11 | Koninklijke Philips Electronics N.V. | Image processing method for automatic adaptation of 3-d deformable model onto a subtantially tubular surface of a 3-d object |
JP2007530088A (en) * | 2003-07-16 | 2007-11-01 | コーニンクレッカ フィリップス エレクトロニクス エヌ ヴィ | Object-specific segmentation |
-
2006
- 2006-02-08 US US11/815,597 patent/US7957572B2/en active Active
- 2006-02-08 AT AT06710848T patent/ATE520101T1/en not_active IP Right Cessation
- 2006-02-08 JP JP2007554712A patent/JP4918048B2/en not_active Expired - Fee Related
- 2006-02-08 WO PCT/IB2006/050403 patent/WO2006085268A1/en active Application Filing
- 2006-02-08 EP EP06710848A patent/EP1851722B8/en active Active
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2022084074A1 (en) * | 2020-10-20 | 2022-04-28 | Koninklijke Philips N.V. | Detecting anatomical abnormalities by segmentation results with and without shape priors |
Also Published As
Publication number | Publication date |
---|---|
US20080205716A1 (en) | 2008-08-28 |
EP1851722B8 (en) | 2012-03-14 |
US7957572B2 (en) | 2011-06-07 |
JP4918048B2 (en) | 2012-04-18 |
ATE520101T1 (en) | 2011-08-15 |
JP2008529644A (en) | 2008-08-07 |
EP1851722A1 (en) | 2007-11-07 |
WO2006085268A1 (en) | 2006-08-17 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
EP1851722B1 (en) | Image processing device and method | |
Ecabert et al. | Automatic model-based segmentation of the heart in CT images | |
Sluimer et al. | Toward automated segmentation of the pathological lung in CT | |
Häme et al. | Semi-automatic liver tumor segmentation with hidden Markov measure field model and non-parametric distribution estimation | |
JP6877868B2 (en) | Image processing equipment, image processing method and image processing program | |
Bauer et al. | Segmentation of interwoven 3d tubular tree structures utilizing shape priors and graph cuts | |
Zheng et al. | Four-chamber heart modeling and automatic segmentation for 3-D cardiac CT volumes using marginal space learning and steerable features | |
Jolly | Automatic segmentation of the left ventricle in cardiac MR and CT images | |
Hernandez et al. | Non-parametric geodesic active regions: method and evaluation for cerebral aneurysms segmentation in 3DRA and CTA | |
US6766043B2 (en) | Pleural nodule detection from CT thoracic images | |
US8582854B2 (en) | Method and system for automatic coronary artery detection | |
Campadelli et al. | Liver segmentation from computed tomography scans: a survey and a new algorithm | |
EP2916738B1 (en) | Lung, lobe, and fissure imaging systems and methods | |
Zagrodsky et al. | Registration-assisted segmentation of real-time 3-D echocardiographic data using deformable models | |
US7359538B2 (en) | Detection and analysis of lesions in contact with a structural boundary | |
Bernier et al. | Graph cut-based method for segmenting the left ventricle from MRI or echocardiographic images | |
Chan et al. | 2D-3D vascular registration between digital subtraction angiographic (DSA) and magnetic resonance angiographic (MRA) images | |
Campadelli et al. | Automatic liver segmentation from abdominal CT scans | |
Jeon et al. | Maximum a posteriori estimation method for aorta localization and coronary seed identification | |
O’Donnell et al. | Comprehensive cardiovascular image analysis using MR and CT at Siemens Corporate Research | |
von Berg et al. | Accurate left atrium segmentation in multislice CT images using a shape model | |
Meyer et al. | Fully automatic segmentation of complex organ systems: example of trachea, esophagus and heart segmentation in CT images | |
Zheng | Model-based 3d cardiac image segmentation with marginal space learning | |
Zhong et al. | Automatic heart isolation in 3D CT images | |
Kovács | Automatic segmentation of the vessel lumen from 3D CTA images of aortic dissection |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PUAI | Public reference made under article 153(3) epc to a published international application that has entered the european phase |
Free format text: ORIGINAL CODE: 0009012 |
|
17P | Request for examination filed |
Effective date: 20070911 |
|
AK | Designated contracting states |
Kind code of ref document: A1 Designated state(s): AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HU IE IS IT LI LT LU LV MC NL PL PT RO SE SI SK TR |
|
DAX | Request for extension of the european patent (deleted) | ||
GRAP | Despatch of communication of intention to grant a patent |
Free format text: ORIGINAL CODE: EPIDOSNIGR1 |
|
RIC1 | Information provided on ipc code assigned before grant |
Ipc: G06T 7/00 20060101AFI20110110BHEP |
|
GRAS | Grant fee paid |
Free format text: ORIGINAL CODE: EPIDOSNIGR3 |
|
GRAA | (expected) grant |
Free format text: ORIGINAL CODE: 0009210 |
|
AK | Designated contracting states |
Kind code of ref document: B1 Designated state(s): AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HU IE IS IT LI LT LU LV MC NL PL PT RO SE SI SK TR |
|
REG | Reference to a national code |
Ref country code: GB Ref legal event code: FG4D |
|
REG | Reference to a national code |
Ref country code: CH Ref legal event code: EP |
|
REG | Reference to a national code |
Ref country code: IE Ref legal event code: FG4D |
|
REG | Reference to a national code |
Ref country code: DE Ref legal event code: R084 Ref document number: 602006023663 Country of ref document: DE |
|
REG | Reference to a national code |
Ref country code: DE Ref legal event code: R096 Ref document number: 602006023663 Country of ref document: DE Effective date: 20111013 |
|
REG | Reference to a national code |
Ref country code: DE Ref legal event code: R084 Ref document number: 602006023663 Country of ref document: DE Effective date: 20110820 |
|
REG | Reference to a national code |
Ref country code: NL Ref legal event code: VDEP Effective date: 20110810 |
|
RAP2 | Party data changed (patent owner data changed or rights of a patent transferred) |
Owner name: PHILIPS INTELLECTUAL PROPERTY & STANDARDS GMBH Owner name: KONINKLIJKE PHILIPS ELECTRONICS N.V. |
|
LTIE | Lt: invalidation of european patent or patent extension |
Effective date: 20110810 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: NL Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20110810 Ref country code: SE Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20110810 Ref country code: FI Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20110810 Ref country code: PT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20111212 Ref country code: LT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20110810 Ref country code: IS Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20111210 |
|
REG | Reference to a national code |
Ref country code: AT Ref legal event code: MK05 Ref document number: 520101 Country of ref document: AT Kind code of ref document: T Effective date: 20110810 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: GR Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20111111 Ref country code: CY Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20110810 Ref country code: PL Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20110810 Ref country code: LV Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20110810 Ref country code: SI Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20110810 Ref country code: AT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20110810 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: BE Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20110810 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: CZ Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20110810 Ref country code: SK Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20110810 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: IT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20110810 Ref country code: RO Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20110810 Ref country code: EE Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20110810 |
|
PLBE | No opposition filed within time limit |
Free format text: ORIGINAL CODE: 0009261 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: DK Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20110810 |
|
26N | No opposition filed |
Effective date: 20120511 |
|
REG | Reference to a national code |
Ref country code: DE Ref legal event code: R119 Ref document number: 602006023663 Country of ref document: DE Ref country code: DE Ref legal event code: R409 Ref document number: 602006023663 Country of ref document: DE |
|
REG | Reference to a national code |
Ref country code: DE Ref legal event code: R097 Ref document number: 602006023663 Country of ref document: DE Effective date: 20120511 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: MC Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20120229 |
|
REG | Reference to a national code |
Ref country code: CH Ref legal event code: PL |
|
REG | Reference to a national code |
Ref country code: DE Ref legal event code: R409 Ref document number: 602006023663 Country of ref document: DE |
|
GBPC | Gb: european patent ceased through non-payment of renewal fee |
Effective date: 20120208 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: LI Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20120229 Ref country code: CH Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20120229 |
|
REG | Reference to a national code |
Ref country code: IE Ref legal event code: MM4A |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: IE Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20120208 Ref country code: GB Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20120208 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: ES Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20111121 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: BG Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20111110 |
|
REG | Reference to a national code |
Ref country code: DE Ref legal event code: R081 Ref document number: 602006023663 Country of ref document: DE Owner name: PHILIPS GMBH, DE Free format text: FORMER OWNER: PHILIPS INTELLECTUAL PROPERTY &, KONINKLIJKE PHILIPS ELECTRONICS, , NL Effective date: 20110824 Ref country code: DE Ref legal event code: R081 Ref document number: 602006023663 Country of ref document: DE Owner name: PHILIPS GMBH, DE Free format text: FORMER OWNER: PHILIPS INTELLECTUAL PROPERTY & STANDARDS GMBH, 20099 HAMBURG, DE Effective date: 20140327 Ref country code: DE Ref legal event code: R081 Ref document number: 602006023663 Country of ref document: DE Owner name: PHILIPS GMBH, DE Free format text: FORMER OWNERS: PHILIPS INTELLECTUAL PROPERTY & STANDARDS GMBH, 20099 HAMBURG, DE; KONINKLIJKE PHILIPS ELECTRONICS N.V., EINDHOVEN, NL Effective date: 20110824 Ref country code: DE Ref legal event code: R081 Ref document number: 602006023663 Country of ref document: DE Owner name: PHILIPS DEUTSCHLAND GMBH, DE Free format text: FORMER OWNER: PHILIPS INTELLECTUAL PROPERTY & STANDARDS GMBH, 20099 HAMBURG, DE Effective date: 20140327 Ref country code: DE Ref legal event code: R081 Ref document number: 602006023663 Country of ref document: DE Owner name: PHILIPS DEUTSCHLAND GMBH, DE Free format text: FORMER OWNER: PHILIPS INTELLECTUAL PROPERTY &, KONINKLIJKE PHILIPS ELECTRONICS, , NL Effective date: 20110824 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: LU Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20120208 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: HU Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20060208 |
|
REG | Reference to a national code |
Ref country code: FR Ref legal event code: CD Owner name: PHILIPS INTELLECTUAL PROPERTY & STANDARDS GMBH, DE Effective date: 20141126 Ref country code: FR Ref legal event code: CD Owner name: KONINKLIJKE PHILIPS ELECTRONICS N.V., NL Effective date: 20141126 Ref country code: FR Ref legal event code: CA Effective date: 20141126 |
|
REG | Reference to a national code |
Ref country code: DE Ref legal event code: R082 Ref document number: 602006023663 Country of ref document: DE Representative=s name: MEISSNER, BOLTE & PARTNER GBR, DE Ref country code: DE Ref legal event code: R081 Ref document number: 602006023663 Country of ref document: DE Owner name: PHILIPS GMBH, DE Free format text: FORMER OWNER: PHILIPS DEUTSCHLAND GMBH, 20099 HAMBURG, DE Ref country code: DE Ref legal event code: R082 Ref document number: 602006023663 Country of ref document: DE Representative=s name: MEISSNER BOLTE PATENTANWAELTE RECHTSANWAELTE P, DE |
|
REG | Reference to a national code |
Ref country code: FR Ref legal event code: PLFP Year of fee payment: 11 |
|
PGFP | Annual fee paid to national office [announced via postgrant information from national office to epo] |
Ref country code: TR Payment date: 20160127 Year of fee payment: 11 |
|
PGFP | Annual fee paid to national office [announced via postgrant information from national office to epo] |
Ref country code: FR Payment date: 20160229 Year of fee payment: 11 |
|
REG | Reference to a national code |
Ref country code: FR Ref legal event code: ST Effective date: 20171031 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: FR Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20170228 |
|
REG | Reference to a national code |
Ref country code: DE Ref legal event code: R082 Ref document number: 602006023663 Country of ref document: DE Representative=s name: MEISSNER BOLTE PATENTANWAELTE RECHTSANWAELTE P, DE Ref country code: DE Ref legal event code: R081 Ref document number: 602006023663 Country of ref document: DE Owner name: PHILIPS GMBH, DE Free format text: FORMER OWNER: PHILIPS GMBH, 20099 HAMBURG, DE |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: TR Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20170208 |
|
PGFP | Annual fee paid to national office [announced via postgrant information from national office to epo] |
Ref country code: DE Payment date: 20220628 Year of fee payment: 18 |