EP1817742A1 - Planar angular visualization of the bronchial tree - Google Patents

Planar angular visualization of the bronchial tree

Info

Publication number
EP1817742A1
EP1817742A1 EP05806939A EP05806939A EP1817742A1 EP 1817742 A1 EP1817742 A1 EP 1817742A1 EP 05806939 A EP05806939 A EP 05806939A EP 05806939 A EP05806939 A EP 05806939A EP 1817742 A1 EP1817742 A1 EP 1817742A1
Authority
EP
European Patent Office
Prior art keywords
tree structure
anatomical
planar
visualization
anatomical tree
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
EP05806939A
Other languages
German (de)
French (fr)
Other versions
EP1817742B1 (en
Inventor
Rafael Wiemker
Thomas Blaffert
Thomas Buelow
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Philips Intellectual Property and Standards GmbH
Koninklijke Philips NV
Original Assignee
Philips Intellectual Property and Standards GmbH
Koninklijke Philips Electronics NV
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Philips Intellectual Property and Standards GmbH, Koninklijke Philips Electronics NV filed Critical Philips Intellectual Property and Standards GmbH
Priority to EP05806939.4A priority Critical patent/EP1817742B1/en
Publication of EP1817742A1 publication Critical patent/EP1817742A1/en
Application granted granted Critical
Publication of EP1817742B1 publication Critical patent/EP1817742B1/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/11—Region-based segmentation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10072—Tomographic images
    • G06T2207/10081—Computed x-ray tomography [CT]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30004—Biomedical image processing
    • G06T2207/30101—Blood vessel; Artery; Vein; Vascular

Definitions

  • This invention pertains in general to the field of medical imaging. More particularly the invention relates to a method and an apparatus for planar angular visualization of an anatomical tree structure, and more particularly to a planar topological mapping of a three-dimensional bronchial tree structure of the lungs.
  • CT computed tomography
  • a patient is advanced linearly through the examination region along a direction that is perpendicular to a gantry rotation plane to effectuate a helical orbiting of the x-ray source about the subject.
  • X-ray absorption data obtained during the helical orbiting is reconstructed using filtered back projection or another reconstruction method to generate a three-dimensional image representation of the subj ect or of a selected portion thereof.
  • High resolution CT data sets from multi-slice scanners allow the inspection of the bronchial down to smaller airways for diagnostic purposes.
  • the resulting images can be used to inspect the airway lumen and wall thickness of the bronchia.
  • the bronchial tree structure contains hundreds of sub-segments. Therefore, a visual inspection of all paths in the three-dimensional bronchial tree is very time consuming. On the other hand, a planar projection of the three-dimensional tree leads to multiple occlusions, as illustrated in Figure 1, and the continuation of the specific airway path is difficult to follow, due to the delicate structure of the tree.
  • an object of the present invention is to provide a system and method for providing an improved representation of a three-dimensional anatomical tree.
  • the present invention preferably seeks to mitigate, alleviate or eliminate one or more of the above-identified deficiencies in the art and disadvantages singly or in any combination and solves at least the above mentioned problems by providing a system, an image acquisition device, a method, a computer-readable medium and an image workstation that provides a planar angular visualization of an anatomical tree according to the appended claims.
  • the general solution according to the invention suggests a planar topological mapping of the three-dimensional bronchial tree structure of the lungs such that no occlusions or intersection of segments occur, that the hierarchy of the sub-segments is derivable, and that conveys a similarity to the true anatomical regions such as the lung lobes.
  • this map allows inspection of the bronchial tree in a view, and with a low degree of freedom, for instance through a Graphical User Interface slider which varies the selected distance to the carina of all bronchial segments.
  • a method for visualizing an anatomical tree structure which comprises the following steps.
  • the anatomical tree structure is segmented from a three-dimensional image set of at least a portion of a body.
  • a planar angular visualization of the anatomical tree structure is then determined.
  • the planar angular visualization of the anatomical tree structure is then displayed.
  • a system for visualizing an anatomical tree structure is disclosed.
  • Segmenting means segments the anatomical tree structure from a three-dimensional image set of at least a portion of a body.
  • Determination means determines a planar visualization of the anatomical tree structure.
  • Displaying means displays the planar angular visualization of the anatomical tree structure.
  • a computer-readable medium having embodied thereon a computer program, for automatic extraction of an anatomical tree structure, for processing by a computer.
  • the computer program comprises a code segment for segmenting the anatomical tree structure from a three-dimensional image set of at least a portion of a body; a code segment for determining a planar angular visualization of the anatomical tree structure; and a code segment for displaying the planar angular visualization of the anatomical tree structure.
  • the present invention has the advantage over the prior art that it provides a two-dimensional representation of an anatomical map in which occlusions or intersection of segments are minimized, i.e. do not occur, thus making the visual inspection of the paths of the anatomical tree much easier, thus facilitating subsequent automated or manual processing of the derived information, e.g. for analysis of the tree for anomalies.
  • Fig. 1 illustrates a coronal projection of a bronchial tree as extracted from a particular CT lung data set
  • Fig. 2 illustrates a CT acquisition device according to one embodiment of the invention
  • Fig. 3 is a flow chart illustrating the method for creating a planar angular visualization of an anatomical tree structure according to one embodiment of the invention
  • Fig. 4 illustrates a planar angular mapping of the bronchial tree extracted from a particular CT lung data set according to one embodiment of the invention
  • Fig. 5 illustrates a planar angular mapping of the areas of the main anatomical regions (lung lobes) according to one embodiment of the invention
  • Fig. 6 illustrates various reformatted images according to one embodiment of the invention
  • Fig. 7 illustrates an alternative display of the planar mapping of the bronchial tree as extracted from a particular CT lung data set according to one embodiment of the invention
  • Fig. 8 schematically illustrates a computer readable medium according to one embodiment of the invention.
  • Fig. 9 illustrates an image workstation according to one embodiment of the invention.
  • Fig. 10 illustrates a bronchial tree wherein the segments of the tree are proportional to their real length according to one embodiment of the invention.
  • CT computed tomography
  • MRI Magnetic Resonance Imaging
  • PET Position Emission Tomography
  • SPECT Single Photon Emission Computed Tomography
  • the CT acquisition device 20 comprises a multi-array CT gantry 22 and a patient table 24 that can be positioned within the gantry 22.
  • the patient table 24 supports a patient during acquisition of raw image data of the patient.
  • the coarse image data is applied to a computer 26, which reconstructs volumetric image data of the raw image data.
  • the computer 26 is programmed in such a manner that in conformity with the invention the computer 26 calculates a segmented tree of the tracheobronchial tree.
  • the computer 26 determines a planar angular visualization of the anatomical tree structure, the tree and visualization being displayed on a display unit of the computer.
  • the reconstructed volumetric image data can be transferred to an image processing system 28 for processing the data according to the method of the invention.
  • This image processing system 28 may be a suitably programmed computer of a workstation 30.
  • the workstation 30, which will be described in more detail below with reference to Figure 9, is connected to a screen 32.
  • the system further comprises a microprocessor, a general purpose memory like random access memory (RAM) and a further memory that are being communicatively connected to each other through a software bus.
  • the memory comprises computer readable software code designed to perform the method according to the invention.
  • the memory is a display buffer that is designed to comprise the segmented tracheobronchial tree, the planar angular visualization and the original image set as previously described.
  • the workstation 30 comprises a suitable storage reading device, like a CD-drive, that can read the software from the storage device. This CD-drive is then operatively connected to the software bus too.
  • a person is x-rayed using a multi-slice CT and a 3-D data set representing the person's bronchial tree is created using a known manner in step 300.
  • the bronchial tree is segmented from a three-dimensional image in step 302, as for instance based on the algorithm described in Schlat ⁇ lter et al. (Simultaneous Segmentation and Tree Reconstruction of the Airways for Virtual Bronchoscopy, SPIE Conference on Medical Imaging, Proceedings of SPIE Vol. 4684, pp. 103-113 (2002)).
  • the unsupervised bronchial tree segmentation starts from the trachea and is based on a so-called front propagation approach.
  • a region growing aggregates all voxels below a certain Hounsfield threshold. Whenever the growth front splits into different segments, a further region growing is started in each of the segments. "Leakages" into the parenchymal tissue is detected and stopped by monitoring a possible sudden increase of the growth front. After the region growing process is complete, then the centerlines (skeleton lines) of trachea, bronchi and smaller airways are extracted and a linked graph structure is built which represents the branching points of the tree structure.
  • a planar angular visualization of the bronchial tree structure is then determined in step 304.
  • the planar visualization of the bronchial tree structure is then displayed in step 306.
  • the invention suggests a two-dimensional planar mapping of the bronchial tree, as illustrated in Figure 4, such that no occlusions or intersection of segments occur, and that the hierarchy of the segments is obvious.
  • the true geometric relations are not preserved, but the topological relations are true as for example in a typical subway map.
  • the main segments point in the approximate direction of their true geometric location in a coronal projection wherein upper lung lobes (42, 41) are directed upwards and lower lung lobes (44, 43) are directed downwards as illustrated in Figures 4-5.
  • Figure 5 also illustrates a middle 45 lung lobe.
  • the suggested planar mapping gives an overview of the bronchial tree in a single two-dimensional view, which is intuitively similar to the anatomical relations, as illustrated in Figure 5.
  • the main bifurcation (carina) in the trachea 50 into left 52 and right 54 main bronchus is plotted in the center of the planar diagram as illustrated in Figure 4. All other points in the diagram are plotted such that their radial distance to the diagram center is proportional to the centerline distance through the airways to the carina, measured in millimetres independent of CT data set resolution.
  • the trachea 50 descends towards the center of the diagram, representing the carina, and then left and right main bronchi bifurcate at 90° angles. It will be understood by those skilled in the art that the angle could also be fixed at another value, for example, as illustrated in Figure 7 which will be described in more detail below.
  • the plotted centerline branches with an equal angular step width to the left and right of the current angular direction.
  • the angular step width decreases by a factor of 1 A after each bifurcation.
  • the bifurcations are characterized as the locations where in the above described region growing scheme the growth front has disintegrated (split). In this way, it can be guaranteed that the subtree below a given point will not enter the diagram area of another subtree.
  • Each point in the mapping is given by angle and distance to the center of the diagram.
  • the distance to the diagram center is proportional to the bronchial-path-distance to the carina, and the angle is determined by the number of previous bifurcations. So the length of a segment in the diagram is just given by the connection between the point positions determined as previously explained.
  • the length of a segment in the diagram is not a true representation of its real length. This is not possible as the segments have to be spread out far enough to avoid overlapping. Rather, the radial distance is a true representation of the length of the airway back to the carina. In other words, the tangential distances in the diagram (on the circles) are just used to spread out the segments, but the radial distances are true.
  • the direction of the branching in the diagram is controlled such that the one of the two subtrees which has the higher centroid (in foot-to-head direction) is chosen as the branch in the diagram which bends towards the top, whereas the other branch bends towards the bottom of the diagram.
  • the colors or grey scale values of the displayed diagram may be chosen such as to reflect the anatomically differentiated lobes and main segments as illustrated in Fig. 10. It will be understood by those skilled in the art that the actual displayed colors or grey scale values can be selected by the user to provide a visual image which is distinctive to the user.
  • the diagram may be interactive in a way that a mouse click on a given point in the diagram will set another display device such as an orthoviewer directly to the place in the three-dimensional CT data set which is represented by the click-on point in the planar mapping.
  • the diagram may be interactive in a way that a mouse click on a given point in the diagram selects the airway centreline path from this point to the trachea and displays further quantitative information along this path, such as lumen diameter, bronchial wall thickness, etc.
  • the diagram may be interactive in a way that a mouse movement over a certain diagram region magnifies this region such that also higher orders of tree generations become discernable.
  • a iurther feature allows the user to move a slider in a graphical user interface such that for all of the points in the tree which have the specified distance from the carina, a perpendicular reformatted image is shown in the corresponding position in the diagram and moving the slider allows changing the specified carina distance, as illustrated by the images in Figure 6.
  • An alternative display of the planar mapping indicating the airway lumen diameter in relation to the carina distance is illustrated in Fig. 7.
  • the diagram can be drawn such that the line thickness at each point of the diagram is proportional to the measured airway lumen diameter at the corresponding point in the bronchial tree.
  • the different line thickness is illustrated in Fig. 7.
  • the diagram can be drawn such that smooth angular interpolation is used to give a more organic appearance as illustrated in Fig. 7.
  • the bifurcations are at the same positions as in Figs. 4-5, wherein position is defined by the number of earlier bifurcations and length to carina, but between the bifurcation points the centreline points are linearly interpolated in their angle and radial distance (p and ⁇ coordinate).
  • the diagram can be drawn such that the bronchial wall thickness at each point of the diagram is proportional to the measured wall thickness at the corresponding point in the bronchial tree.
  • Anomalies in the bronchial lumen diameter and wall thickness can be automatically detected and pointed out by markers or color coding at the respective positions in the diagram.
  • another image processing module can find at each point of the bronchial tree the accompanying artery. The diameter of the accompanying artery can be automatically measured. Then the ratio between the bronchial diameter and the arterial diameter (which is an important clinical parameter) can be color-coded into each point of the planar mapping.
  • planar mapping is to map the segmented pulmonary arterial vessel tree, and to render at each point of the graph the vessel diameter or any possibly detected anomalies. For example, at each point of the graph, the minimum projection of the CT-Hounsfield values at this vessel portion can be rendered, so that filling defects (caused by clots, and possibly indicating pulmonary embolisms) can be detected at a single glance.
  • a computer readable medium is illustrated schematically.
  • a computer-readable medium 100 has embodied thereon a computer program 110 for automatic extraction of a planar angular visualization of an anatomical tree from a 3D medical image 112, for processing by a computer 113.
  • the computer program comprises a code segment 114 for segmenting the anatomical tree structure from a three-dimensional image set of at least a portion of a body, a code segment 115 for determining a planar angular visualization of the anatomical tree structure, a code segment 116 for displaying the planar angular visualization of the anatomical tree structure.
  • Fig. 9 illustrates an exemplary medical image workstation 119 according to a further embodiment of the present invention.
  • the medical workstation is arranged for implementing the method of the invention, and configured to receive and process a 3D medical image.
  • the workstation is arranged to run the above described program code segments in order to perform the method according to the invention.
  • the medical workstation 119 is configured for automatic extraction of a planar angular visualization of an anatomical tree from a 3D medical image, and comprises segmenting means 120 for segmenting the anatomical tree structure from a three-dimensional image set of at least a portion of a body, determination means 121 for determining a planar visualization of the anatomical tree structure, displaying means 122 for displaying the planar angular visualization of the anatomical tree structure.
  • Said means 120-122 are preferably electronic components operatively connected to each other in a suitable way.
  • Other components of the medical workstation as e.g. display(s) or pointer actuating devices are not illustrated or discussed in detail.
  • the invention can be implemented in any suitable form including hardware, software, firmware or any combination of these. However, preferably, the invention is implemented as computer software running on one or more data processors and/or digital signal processors.
  • the elements and components of an embodiment of the invention may be physically, functionally and logically implemented in any suitable way. Indeed, the functionality may be implemented in a single unit, in a plurality of units or as part of other functional units. As such, the invention may be implemented in a single unit, or may be physically and functionally distributed between different units and processors.

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Apparatus For Radiation Diagnosis (AREA)

Abstract

A system (20) an image aquisition device, an image workstation, computer- readable medium and method for visualizing an anatomical tree structure is disclosed. The anatomical tree structure is segmented from a three-dimensional image set of at least a portion of a body. A planar angular visualization of the anatomical tree structure is then determined. The planar angular visualization of the anatomical tree structure is then displayed. Since the displayed anatomical tree structure has no occlusions or intersections of segments, the anatomical tree is much easier to read which facilitates subsequent automated or manual diagnosis of the tree for anomalies.

Description

Planar angular visualization of the bronchial tree
This invention pertains in general to the field of medical imaging. More particularly the invention relates to a method and an apparatus for planar angular visualization of an anatomical tree structure, and more particularly to a planar topological mapping of a three-dimensional bronchial tree structure of the lungs. In helical computed tomography (CT) imaging, a patient is advanced linearly through the examination region along a direction that is perpendicular to a gantry rotation plane to effectuate a helical orbiting of the x-ray source about the subject. X-ray absorption data obtained during the helical orbiting is reconstructed using filtered back projection or another reconstruction method to generate a three-dimensional image representation of the subj ect or of a selected portion thereof.
High resolution CT data sets from multi-slice scanners allow the inspection of the bronchial down to smaller airways for diagnostic purposes. The resulting images can be used to inspect the airway lumen and wall thickness of the bronchia.
The bronchial tree structure contains hundreds of sub-segments. Therefore, a visual inspection of all paths in the three-dimensional bronchial tree is very time consuming. On the other hand, a planar projection of the three-dimensional tree leads to multiple occlusions, as illustrated in Figure 1, and the continuation of the specific airway path is difficult to follow, due to the delicate structure of the tree.
Hence, an object of the present invention is to provide a system and method for providing an improved representation of a three-dimensional anatomical tree.
Accordingly, the present invention preferably seeks to mitigate, alleviate or eliminate one or more of the above-identified deficiencies in the art and disadvantages singly or in any combination and solves at least the above mentioned problems by providing a system, an image acquisition device, a method, a computer-readable medium and an image workstation that provides a planar angular visualization of an anatomical tree according to the appended claims.
The general solution according to the invention suggests a planar topological mapping of the three-dimensional bronchial tree structure of the lungs such that no occlusions or intersection of segments occur, that the hierarchy of the sub-segments is derivable, and that conveys a similarity to the true anatomical regions such as the lung lobes. In contrast to conventional geometric 3D-to-2D projections, this map allows inspection of the bronchial tree in a view, and with a low degree of freedom, for instance through a Graphical User Interface slider which varies the selected distance to the carina of all bronchial segments.
According to one aspect of the invention, a method is provided for visualizing an anatomical tree structure which comprises the following steps. The anatomical tree structure is segmented from a three-dimensional image set of at least a portion of a body. A planar angular visualization of the anatomical tree structure is then determined. The planar angular visualization of the anatomical tree structure is then displayed.
According to another aspect of the invention, a system for visualizing an anatomical tree structure is disclosed. Segmenting means segments the anatomical tree structure from a three-dimensional image set of at least a portion of a body. Determination means determines a planar visualization of the anatomical tree structure. Displaying means displays the planar angular visualization of the anatomical tree structure.
According to a further aspect of the invention, a computer-readable medium having embodied thereon a computer program, for automatic extraction of an anatomical tree structure, for processing by a computer is provided. The computer program comprises a code segment for segmenting the anatomical tree structure from a three-dimensional image set of at least a portion of a body; a code segment for determining a planar angular visualization of the anatomical tree structure; and a code segment for displaying the planar angular visualization of the anatomical tree structure.
The present invention has the advantage over the prior art that it provides a two-dimensional representation of an anatomical map in which occlusions or intersection of segments are minimized, i.e. do not occur, thus making the visual inspection of the paths of the anatomical tree much easier, thus facilitating subsequent automated or manual processing of the derived information, e.g. for analysis of the tree for anomalies.
These and other aspects, features and advantages of which the invention is capable of will be apparent and elucidated from the following description of embodiments of the present invention, reference being made to the accompanying drawings, in which
Fig. 1 illustrates a coronal projection of a bronchial tree as extracted from a particular CT lung data set; Fig. 2 illustrates a CT acquisition device according to one embodiment of the invention;
Fig. 3 is a flow chart illustrating the method for creating a planar angular visualization of an anatomical tree structure according to one embodiment of the invention; Fig. 4 illustrates a planar angular mapping of the bronchial tree extracted from a particular CT lung data set according to one embodiment of the invention;
Fig. 5 illustrates a planar angular mapping of the areas of the main anatomical regions (lung lobes) according to one embodiment of the invention;
Fig. 6 illustrates various reformatted images according to one embodiment of the invention;
Fig. 7 illustrates an alternative display of the planar mapping of the bronchial tree as extracted from a particular CT lung data set according to one embodiment of the invention;
Fig. 8 schematically illustrates a computer readable medium according to one embodiment of the invention;
Fig. 9 illustrates an image workstation according to one embodiment of the invention; and
Fig. 10 illustrates a bronchial tree wherein the segments of the tree are proportional to their real length according to one embodiment of the invention.
The following description focuses on an embodiment of the present invention applicable to a computed tomography (CT) system and in particular to multi- slice CT data. However, it will be appreciated that the invention is not limited to this application but may be applied to many other imaging systems, including for example Magnetic Resonance Imaging (MRI) systems, a three-dimensional Rotational Angiography (3D-RA) scanner, Position Emission Tomography (PET) scanner, single Photon Emission Computed Tomography (SPECT) scanner, etc.
Below, an automated method for the extraction of an anatomical tree from multi- slice CT data is described. While much of the discussion of the invention will relate to a bronchial tree, it will be understood by those skilled in the art that the invention relates to any type of anatomical tree such as a vessel tree and the invention is not limited thereto.
An example of a CT acquisition device 20 according to one embodiment of the invention is illustrated in Figure 2. The CT acquisition device 20 comprises a multi-array CT gantry 22 and a patient table 24 that can be positioned within the gantry 22. The patient table 24 supports a patient during acquisition of raw image data of the patient. The coarse image data is applied to a computer 26, which reconstructs volumetric image data of the raw image data. The computer 26 is programmed in such a manner that in conformity with the invention the computer 26 calculates a segmented tree of the tracheobronchial tree. The computer 26 then determines a planar angular visualization of the anatomical tree structure, the tree and visualization being displayed on a display unit of the computer. Alternatively, the reconstructed volumetric image data can be transferred to an image processing system 28 for processing the data according to the method of the invention. This image processing system 28 may be a suitably programmed computer of a workstation 30. The workstation 30, which will be described in more detail below with reference to Figure 9, is connected to a screen 32. The system further comprises a microprocessor, a general purpose memory like random access memory (RAM) and a further memory that are being communicatively connected to each other through a software bus. The memory comprises computer readable software code designed to perform the method according to the invention. The memory is a display buffer that is designed to comprise the segmented tracheobronchial tree, the planar angular visualization and the original image set as previously described. The contents of this buffer are displayed at the display device 32. It is further possible to download the computer readable software from a storage device like a compact disk (CD), digital versatile disk (DVD) etc., or to download the computer readable software as such, e.g. from the Internet, into the memory of the workstation. Therefore, the workstation 30 comprises a suitable storage reading device, like a CD-drive, that can read the software from the storage device. This CD-drive is then operatively connected to the software bus too.
In a first embodiment of the method of the invention, with reference to Fig. 3, the following procedure is performed.
Firstly, a person is x-rayed using a multi-slice CT and a 3-D data set representing the person's bronchial tree is created using a known manner in step 300. Next, the bronchial tree is segmented from a three-dimensional image in step 302, as for instance based on the algorithm described in Schlatδlter et al. (Simultaneous Segmentation and Tree Reconstruction of the Airways for Virtual Bronchoscopy, SPIE Conference on Medical Imaging, Proceedings of SPIE Vol. 4684, pp. 103-113 (2002)).
The unsupervised bronchial tree segmentation starts from the trachea and is based on a so-called front propagation approach. A region growing aggregates all voxels below a certain Hounsfield threshold. Whenever the growth front splits into different segments, a further region growing is started in each of the segments. "Leakages" into the parenchymal tissue is detected and stopped by monitoring a possible sudden increase of the growth front. After the region growing process is complete, then the centerlines (skeleton lines) of trachea, bronchi and smaller airways are extracted and a linked graph structure is built which represents the branching points of the tree structure.
Referring back to Fig. 3, a planar angular visualization of the bronchial tree structure is then determined in step 304. Finally, the planar visualization of the bronchial tree structure is then displayed in step 306.
The invention suggests a two-dimensional planar mapping of the bronchial tree, as illustrated in Figure 4, such that no occlusions or intersection of segments occur, and that the hierarchy of the segments is obvious. The true geometric relations are not preserved, but the topological relations are true as for example in a typical subway map. Moreover, the main segments point in the approximate direction of their true geometric location in a coronal projection wherein upper lung lobes (42, 41) are directed upwards and lower lung lobes (44, 43) are directed downwards as illustrated in Figures 4-5. Figure 5 also illustrates a middle 45 lung lobe. The suggested planar mapping gives an overview of the bronchial tree in a single two-dimensional view, which is intuitively similar to the anatomical relations, as illustrated in Figure 5.
Features of the planar mapping process will now be described in more detail. The main bifurcation (carina) in the trachea 50 into left 52 and right 54 main bronchus is plotted in the center of the planar diagram as illustrated in Figure 4. All other points in the diagram are plotted such that their radial distance to the diagram center is proportional to the centerline distance through the airways to the carina, measured in millimetres independent of CT data set resolution. In this embodiment, the trachea 50 descends towards the center of the diagram, representing the carina, and then left and right main bronchi bifurcate at 90° angles. It will be understood by those skilled in the art that the angle could also be fixed at another value, for example, as illustrated in Figure 7 which will be described in more detail below. At each bifurcation of the airway, the plotted centerline branches with an equal angular step width to the left and right of the current angular direction. The angular step width decreases by a factor of 1A after each bifurcation. The bifurcations are characterized as the locations where in the above described region growing scheme the growth front has disintegrated (split). In this way, it can be guaranteed that the subtree below a given point will not enter the diagram area of another subtree.
Each point in the mapping is given by angle and distance to the center of the diagram. The distance to the diagram center is proportional to the bronchial-path-distance to the carina, and the angle is determined by the number of previous bifurcations. So the length of a segment in the diagram is just given by the connection between the point positions determined as previously explained.
In this embodiment, the length of a segment in the diagram is not a true representation of its real length. This is not possible as the segments have to be spread out far enough to avoid overlapping. Rather, the radial distance is a true representation of the length of the airway back to the carina. In other words, the tangential distances in the diagram (on the circles) are just used to spread out the segments, but the radial distances are true.
In another embodiment of the planar mapping as illustrated in Fig. 10, it is also possible to make the length of a segment in the diagram proportional to its real length. In this alternative mode, the radial distance of a point in the diagram to the center of the diagram is not proportional to the carina-distance. Rather, a path is drawn for each segment (starting from the carina) of length proportional to its real length. Then, at each bifurcation of generation i, the mapping branches from the current direction into two new directions by an angle ± θi , and the branching angle decreases with each generation θi+1 = 1A θi , so that occlusion- free mapping is guaranteed.
In order to improve the similarity of the diagram to the true anatomical positions, the direction of the branching in the diagram is controlled such that the one of the two subtrees which has the higher centroid (in foot-to-head direction) is chosen as the branch in the diagram which bends towards the top, whereas the other branch bends towards the bottom of the diagram.
The colors or grey scale values of the displayed diagram may be chosen such as to reflect the anatomically differentiated lobes and main segments as illustrated in Fig. 10. It will be understood by those skilled in the art that the actual displayed colors or grey scale values can be selected by the user to provide a visual image which is distinctive to the user. The diagram may be interactive in a way that a mouse click on a given point in the diagram will set another display device such as an orthoviewer directly to the place in the three-dimensional CT data set which is represented by the click-on point in the planar mapping. The diagram may be interactive in a way that a mouse click on a given point in the diagram selects the airway centreline path from this point to the trachea and displays further quantitative information along this path, such as lumen diameter, bronchial wall thickness, etc. The diagram may be interactive in a way that a mouse movement over a certain diagram region magnifies this region such that also higher orders of tree generations become discernable. A iurther feature allows the user to move a slider in a graphical user interface such that for all of the points in the tree which have the specified distance from the carina, a perpendicular reformatted image is shown in the corresponding position in the diagram and moving the slider allows changing the specified carina distance, as illustrated by the images in Figure 6. An alternative display of the planar mapping indicating the airway lumen diameter in relation to the carina distance is illustrated in Fig. 7.
The diagram can be drawn such that the line thickness at each point of the diagram is proportional to the measured airway lumen diameter at the corresponding point in the bronchial tree. The different line thickness is illustrated in Fig. 7.
The diagram can be drawn such that smooth angular interpolation is used to give a more organic appearance as illustrated in Fig. 7. The bifurcations are at the same positions as in Figs. 4-5, wherein position is defined by the number of earlier bifurcations and length to carina, but between the bifurcation points the centreline points are linearly interpolated in their angle and radial distance (p and φ coordinate).
The diagram can be drawn such that the bronchial wall thickness at each point of the diagram is proportional to the measured wall thickness at the corresponding point in the bronchial tree.
Anomalies in the bronchial lumen diameter and wall thickness can be automatically detected and pointed out by markers or color coding at the respective positions in the diagram. Also, another image processing module can find at each point of the bronchial tree the accompanying artery. The diameter of the accompanying artery can be automatically measured. Then the ratio between the bronchial diameter and the arterial diameter (which is an important clinical parameter) can be color-coded into each point of the planar mapping.
Another application of the planar mapping is to map the segmented pulmonary arterial vessel tree, and to render at each point of the graph the vessel diameter or any possibly detected anomalies. For example, at each point of the graph, the minimum projection of the CT-Hounsfield values at this vessel portion can be rendered, so that filling defects (caused by clots, and possibly indicating pulmonary embolisms) can be detected at a single glance. In another embodiment of the invention according to Fig. 8, a computer readable medium is illustrated schematically. A computer-readable medium 100 has embodied thereon a computer program 110 for automatic extraction of a planar angular visualization of an anatomical tree from a 3D medical image 112, for processing by a computer 113. The computer program comprises a code segment 114 for segmenting the anatomical tree structure from a three-dimensional image set of at least a portion of a body, a code segment 115 for determining a planar angular visualization of the anatomical tree structure, a code segment 116 for displaying the planar angular visualization of the anatomical tree structure. Fig. 9 illustrates an exemplary medical image workstation 119 according to a further embodiment of the present invention. The medical workstation is arranged for implementing the method of the invention, and configured to receive and process a 3D medical image. Preferably the workstation is arranged to run the above described program code segments in order to perform the method according to the invention. According to the embodiment, the medical workstation 119 is configured for automatic extraction of a planar angular visualization of an anatomical tree from a 3D medical image, and comprises segmenting means 120 for segmenting the anatomical tree structure from a three-dimensional image set of at least a portion of a body, determination means 121 for determining a planar visualization of the anatomical tree structure, displaying means 122 for displaying the planar angular visualization of the anatomical tree structure. Said means 120-122 are preferably electronic components operatively connected to each other in a suitable way. Other components of the medical workstation as e.g. display(s) or pointer actuating devices are not illustrated or discussed in detail.
The invention can be implemented in any suitable form including hardware, software, firmware or any combination of these. However, preferably, the invention is implemented as computer software running on one or more data processors and/or digital signal processors. The elements and components of an embodiment of the invention may be physically, functionally and logically implemented in any suitable way. Indeed, the functionality may be implemented in a single unit, in a plurality of units or as part of other functional units. As such, the invention may be implemented in a single unit, or may be physically and functionally distributed between different units and processors.
Although the present invention has been described above with reference to (a) specific embodiment(s), it is not intended to be limited to the specific form set forth herein. Rather, the invention is limited only by the accompanying claims and, other embodiments than the specific above are equally possible within the scope of these appended claims, e.g. different anatomical tree structures than those described above.
In the claims, the term "comprises/comprising" does not exclude the presence of other elements or steps. Furthermore, although individually listed, a plurality of means, elements or method steps may be implemented by e.g. a single unit or processor.
Additionally, although individual features may be included in different claims, these may possibly advantageously be combined, and the inclusion in different claims does not imply that a combination of features is not feasible and/or advantageous. In addition, singular references do not exclude a plurality. The terms "a", "an", "first", "second" etc do not preclude a plurality. Reference signs in the claims are provided merely as a clarifying example and shall not be construed as limiting the scope of the claims in any way.

Claims

CLAIMS:
1. A system for visualizing an anatomical tree structure comprising: segmenting means (26, 28, 120) for segmenting the anatomical tree structure from a three- dimensional image set of at least a portion of a body; determination means (26, 28, 121) for determining a planar visualization of the anatomical tree structure.
2. The system according to claim 1, further comprising: displaying means (32, 122) for displaying the planar angular visualization of the anatomical tree structure.
3. The system according to claim 2, wherein a displayed length of each segment of the anatomical tree structure is proportional to the real length of each segment.
4. The system according to claim 3, wherein at each bifurcation of generation i, the planar visualization of the tree structure branches from a current direction into at least two new directions by a branching angle ± θi, and the branching angle decreases with each generation θi+1 = 1A θi.
5. The system according to claim 2, wherein said displayed planar visualization of the tree structure is interactive.
6. The system according to claim 2, wherein a displayed width at each point of the displayed planar visualization of the tree structure is proportional to a measured airway lumen diameter at a corresponding point in the anatomical tree structure.
7. The system according to claim 1, wherein the anatomical tree structure is a bronchial tree.
8. The system according to claim 7, further comprising: determination means (26, 28) for determining anomalies in bronchial diameter and wall thickness.
9. The system according to claim 1, wherein the anatomical tree structure is a pulmonary arterial vessel tree.
10. Image acquisition device (20) comprising the system according to any of the claims 1 to 9.
11. Image workstation (30) comprising the system according to any of the claims
I to 9.
12. A method of visualizing an anatomical tree structure, comprising the steps of: segmenting the anatomical tree structure from a three-dimensional image set of at least a portion of a body; determining a planar angular visualization of the anatomical tree structure.
13. The method according to claim 12, further comprising the step of: displaying the planar angular visualization of the anatomical tree structure.
14. The method according to claim 12, further comprising the step of determining anomalies in a diameter and wall thickness of the anatomical tree structure wherein the anamalies are color coded on the displayed planar visualization of the tree structure.
15. The method according to claim 12, wherin the anatomical tree structure is a bronchial tree, further comprising the steps of: determining a diameter of an accompanying artery for any point on the bronchial tree; determining a ratio between the bronchial diameter and the arterial diameter.
16. A computer-readable medium (100) having embodied thereon a computer program (110) for automatic extraction of an anatomical tree structure for processing by a computer (113), the computer program (110) comprising: a code segment for segmenting the anatomical tree structure from a three- dimensional image set of at least a portion of a body; a code segment for determining a planar angular visualization of the anatomical tree structure.
17. The computer-readable medium (100) according to claim 16, further comprising: a code segment for displaying the planar angular visualization of the anatomical tree structure.
EP05806939.4A 2004-11-23 2005-11-21 Planar angular visualization of the bronchial tree Active EP1817742B1 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
EP05806939.4A EP1817742B1 (en) 2004-11-23 2005-11-21 Planar angular visualization of the bronchial tree

Applications Claiming Priority (4)

Application Number Priority Date Filing Date Title
EP04106013 2004-11-23
EP05102793 2005-04-08
PCT/IB2005/053829 WO2006056923A1 (en) 2004-11-23 2005-11-21 Planar angular visualization of the bronchial tree
EP05806939.4A EP1817742B1 (en) 2004-11-23 2005-11-21 Planar angular visualization of the bronchial tree

Publications (2)

Publication Number Publication Date
EP1817742A1 true EP1817742A1 (en) 2007-08-15
EP1817742B1 EP1817742B1 (en) 2018-01-10

Family

ID=35814189

Family Applications (1)

Application Number Title Priority Date Filing Date
EP05806939.4A Active EP1817742B1 (en) 2004-11-23 2005-11-21 Planar angular visualization of the bronchial tree

Country Status (5)

Country Link
US (1) US8045769B2 (en)
EP (1) EP1817742B1 (en)
JP (1) JP4871292B2 (en)
CN (1) CN101065771B (en)
WO (1) WO2006056923A1 (en)

Families Citing this family (18)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP4950071B2 (en) * 2005-02-11 2012-06-13 コーニンクレッカ フィリップス エレクトロニクス エヌ ヴィ Method for automatic extraction of pulmonary artery tree from 3D medical images
WO2009128020A1 (en) 2008-04-18 2009-10-22 Koninklijke Philips Electronics N.V. Segmenting pulmonary arteries
AU2011210257B2 (en) * 2010-02-01 2013-12-19 Covidien Lp Region-growing algorithm
JP5584006B2 (en) * 2010-03-31 2014-09-03 富士フイルム株式会社 Projection image generation apparatus, projection image generation program, and projection image generation method
DE102012204063B4 (en) * 2012-03-15 2021-02-18 Siemens Healthcare Gmbh Generation of visualization command data
JP5833994B2 (en) * 2012-09-20 2015-12-16 富士フイルム株式会社 Image processing apparatus and method, and program
CN104036484B (en) 2013-03-06 2017-08-08 东芝医疗系统株式会社 Image segmenting device, image partition method and medical image equipment
JP6080267B2 (en) * 2014-09-12 2017-02-15 富士フイルム株式会社 Three-dimensional object division output device and its application
US9901406B2 (en) 2014-10-02 2018-02-27 Inneroptic Technology, Inc. Affected region display associated with a medical device
US10188467B2 (en) 2014-12-12 2019-01-29 Inneroptic Technology, Inc. Surgical guidance intersection display
US9949700B2 (en) * 2015-07-22 2018-04-24 Inneroptic Technology, Inc. Medical device approaches
CN105224759B (en) * 2015-10-13 2018-03-20 张帆 A kind of human anatomic structure model, implant quick molding method
AU2017213639B2 (en) * 2016-02-05 2021-03-11 Pulmonx Corporation Methods, systems, and devices for analyzing lung imaging data
CN110246126A (en) * 2019-06-14 2019-09-17 吉林大学第一医院 A method of extracting terminal bronchi tree from lung CT image
US11875459B2 (en) 2020-04-07 2024-01-16 Vida Diagnostics, Inc. Subject specific coordinatization and virtual navigation systems and methods
CN111861988A (en) * 2020-06-09 2020-10-30 深圳市旭东数字医学影像技术有限公司 Method and system for automatic and semi-automatic lung lobular segmentation based on bronchus
US12076158B2 (en) * 2021-02-05 2024-09-03 Siemens Healthineers Ag Intuitive display for rotator cuff tear diagnostics
CN114820919B (en) * 2022-03-09 2024-10-18 深圳湾实验室 Human body full respiratory tract model construction method and system conforming to real anatomical structure

Family Cites Families (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPS575109A (en) * 1980-06-10 1982-01-11 Fanuc Ltd Curved surface forming method
JP3030108B2 (en) * 1991-03-18 2000-04-10 日本板硝子株式会社 Multi-branch optical circuit
JPH0696223A (en) * 1992-09-16 1994-04-08 Matsushita Electric Ind Co Ltd Tree structure drawing plotter
FR2708166A1 (en) * 1993-07-22 1995-01-27 Philips Laboratoire Electroniq A method of processing digitized images for the automatic detection of stenoses.
US6246784B1 (en) * 1997-08-19 2001-06-12 The United States Of America As Represented By The Department Of Health And Human Services Method for segmenting medical images and detecting surface anomalies in anatomical structures
US6690816B2 (en) * 2000-04-07 2004-02-10 The University Of North Carolina At Chapel Hill Systems and methods for tubular object processing
US7191119B2 (en) * 2002-05-07 2007-03-13 International Business Machines Corporation Integrated development tool for building a natural language understanding application
JP4342784B2 (en) 2002-09-27 2009-10-14 ザイオソフト株式会社 Branched CPR image display processing method, branched CPR image display processing apparatus and program
GB2395880B (en) 2002-11-27 2005-02-02 Voxar Ltd Curved multi-planar reformatting of three-dimensional volume data sets
US7469073B2 (en) * 2004-05-24 2008-12-23 Siemens Medical Solutions Usa, Inc. Image-based method for detection and removal of small fragments in segmented three-dimensional volumes

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
See references of WO2006056923A1 *

Also Published As

Publication number Publication date
US20090148008A1 (en) 2009-06-11
JP2008520319A (en) 2008-06-19
CN101065771B (en) 2011-05-18
CN101065771A (en) 2007-10-31
EP1817742B1 (en) 2018-01-10
WO2006056923A1 (en) 2006-06-01
US8045769B2 (en) 2011-10-25
JP4871292B2 (en) 2012-02-08

Similar Documents

Publication Publication Date Title
US8045769B2 (en) Planar angular visualization of the bronchial tree
US7599535B2 (en) System and method for tree-model visualization for pulmonary embolism detection
Sonka et al. Rule-based detection of intrathoracic airway trees
US20230346236A1 (en) System for vascular assessment
US10039510B2 (en) Visualizing different types of airway wall abnormalities
CN103002808B (en) The cardiac roadmapping originating from 3D is drawn
EP2104453B1 (en) Visualizing a vascular structure
US8754888B2 (en) Systems and methods for segmenting three dimensional image volumes
JP2006075599A (en) Visualization method of anatomical tubular structure
JP2008503303A (en) Method and apparatus for displaying a tracheobronchial tree
JP2010528750A (en) Inspection of tubular structures
Bullitt et al. Volume rendering of segmented image objects
CN111932665A (en) Hepatic vessel three-dimensional reconstruction and visualization method based on vessel tubular model
Ropinski et al. Multimodal vessel visualization of mouse aorta PET/CT scans
EP3828836B1 (en) Method and data processing system for providing a two-dimensional unfolded image of at least one tubular structure
EP1735752A1 (en) Method and imaging diagnostic apparatus for finding a stenosis
Andronikou et al. Technical developments in postprocessing of paediatric airway imaging
WO2010001327A1 (en) Processing anatomy and associated quantitative analysis data of tissue
Habert et al. A novel method for an automatic 3D reconstruction of coronary arteries from angiographic images
Muhammad Abdullah Adli Investigation of in vivo cardiac chambers composite rendering/Muhammad Abdullah Adli Ahmad Kamal
Kamal Investigation of in Vivo Cardiac Chambers Composite Rendering
WO2021207289A1 (en) Subject specific coordinatization and virtual navigation systems and methods
Douek et al. Image processing in contrast-enhanced MR angiography
Gruden et al. MDCT pulmonary angiography: Image processing tools
Chen et al. Bolus tracking by cone-beam reconstruction and reprojection

Legal Events

Date Code Title Description
PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

17P Request for examination filed

Effective date: 20070625

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HU IE IS IT LI LT LU LV MC NL PL PT RO SE SI SK TR

DAX Request for extension of the european patent (deleted)
RAP1 Party data changed (applicant data changed or rights of an application transferred)

Owner name: PHILIPS INTELLECTUAL PROPERTY & STANDARDS GMBH

Owner name: KONINKLIJKE PHILIPS N.V.

17Q First examination report despatched

Effective date: 20140901

GRAP Despatch of communication of intention to grant a patent

Free format text: ORIGINAL CODE: EPIDOSNIGR1

INTG Intention to grant announced

Effective date: 20170714

GRAS Grant fee paid

Free format text: ORIGINAL CODE: EPIDOSNIGR3

GRAA (expected) grant

Free format text: ORIGINAL CODE: 0009210

REG Reference to a national code

Ref country code: DE

Ref legal event code: R081

Ref document number: 602005053362

Country of ref document: DE

Owner name: KONINKLIJKE PHILIPS N.V., NL

Free format text: FORMER OWNERS: PHILIPS INTELLECTUAL PROPERTY & STANDARDS GMBH, 20099 HAMBURG, DE; KONINKLIJKE PHILIPS ELECTRONICS N.V., EINDHOVEN, NL

Ref country code: DE

Ref legal event code: R081

Ref document number: 602005053362

Country of ref document: DE

Owner name: PHILIPS GMBH, DE

Free format text: FORMER OWNERS: PHILIPS INTELLECTUAL PROPERTY & STANDARDS GMBH, 20099 HAMBURG, DE; KONINKLIJKE PHILIPS ELECTRONICS N.V., EINDHOVEN, NL

AK Designated contracting states

Kind code of ref document: B1

Designated state(s): AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HU IE IS IT LI LT LU LV MC NL PL PT RO SE SI SK TR

REG Reference to a national code

Ref country code: GB

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: CH

Ref legal event code: EP

Ref country code: AT

Ref legal event code: REF

Ref document number: 963168

Country of ref document: AT

Kind code of ref document: T

Effective date: 20180115

REG Reference to a national code

Ref country code: IE

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: DE

Ref legal event code: R096

Ref document number: 602005053362

Country of ref document: DE

REG Reference to a national code

Ref country code: DE

Ref legal event code: R081

Ref document number: 602005053362

Country of ref document: DE

Owner name: PHILIPS GMBH, DE

Free format text: FORMER OWNERS: KONINKLIJKE PHILIPS N.V., EINDHOVEN, NL; PHILIPS INTELLECTUAL PROPERTY & STANDARDS GMBH, 20099 HAMBURG, DE

Ref country code: DE

Ref legal event code: R081

Ref document number: 602005053362

Country of ref document: DE

Owner name: KONINKLIJKE PHILIPS N.V., NL

Free format text: FORMER OWNERS: KONINKLIJKE PHILIPS N.V., EINDHOVEN, NL; PHILIPS INTELLECTUAL PROPERTY & STANDARDS GMBH, 20099 HAMBURG, DE

REG Reference to a national code

Ref country code: DE

Ref legal event code: R084

Ref document number: 602005053362

Country of ref document: DE

REG Reference to a national code

Ref country code: GB

Ref legal event code: 746

Effective date: 20180322

REG Reference to a national code

Ref country code: NL

Ref legal event code: MP

Effective date: 20180110

REG Reference to a national code

Ref country code: AT

Ref legal event code: MK05

Ref document number: 963168

Country of ref document: AT

Kind code of ref document: T

Effective date: 20180110

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: NL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180110

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: LT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180110

Ref country code: CY

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180110

Ref country code: FI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180110

Ref country code: ES

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180110

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: AT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180110

Ref country code: PL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180110

Ref country code: BG

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180410

Ref country code: GR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180411

Ref country code: SE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180110

Ref country code: LV

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180110

Ref country code: IS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180510

REG Reference to a national code

Ref country code: DE

Ref legal event code: R097

Ref document number: 602005053362

Country of ref document: DE

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: EE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180110

Ref country code: IT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180110

Ref country code: RO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180110

PLBE No opposition filed within time limit

Free format text: ORIGINAL CODE: 0009261

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180110

Ref country code: CZ

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180110

Ref country code: DK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180110

26N No opposition filed

Effective date: 20181011

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180110

REG Reference to a national code

Ref country code: CH

Ref legal event code: PL

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: LU

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20181121

Ref country code: MC

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180110

REG Reference to a national code

Ref country code: BE

Ref legal event code: MM

Effective date: 20181130

REG Reference to a national code

Ref country code: IE

Ref legal event code: MM4A

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: LI

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20181130

Ref country code: CH

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20181130

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20181121

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: BE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20181130

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: TR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180110

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: PT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180110

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: HU

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT; INVALID AB INITIO

Effective date: 20051121

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: FR

Payment date: 20211126

Year of fee payment: 17

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: GB

Payment date: 20221122

Year of fee payment: 18

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: FR

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20221130

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: DE

Payment date: 20231127

Year of fee payment: 19

GBPC Gb: european patent ceased through non-payment of renewal fee

Effective date: 20231121

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: GB

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20231121