EP2831841B1 - Multikamera-tracking - Google Patents
Multikamera-tracking Download PDFInfo
- Publication number
- EP2831841B1 EP2831841B1 EP13737409.6A EP13737409A EP2831841B1 EP 2831841 B1 EP2831841 B1 EP 2831841B1 EP 13737409 A EP13737409 A EP 13737409A EP 2831841 B1 EP2831841 B1 EP 2831841B1
- Authority
- EP
- European Patent Office
- Prior art keywords
- image data
- interventional device
- model
- needle
- bending
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000005452 bending Methods 0.000 claims description 34
- 238000012545 processing Methods 0.000 claims description 28
- 238000000034 method Methods 0.000 claims description 20
- 230000003287 optical effect Effects 0.000 claims description 19
- 238000004590 computer program Methods 0.000 claims description 16
- 238000002059 diagnostic imaging Methods 0.000 claims description 12
- 239000012634 fragment Substances 0.000 description 11
- 238000003384 imaging method Methods 0.000 description 5
- 238000003780 insertion Methods 0.000 description 5
- 230000037431 insertion Effects 0.000 description 5
- 238000002604 ultrasonography Methods 0.000 description 5
- FGUUSXIOTUKUDN-IBGZPJMESA-N C1(=CC=CC=C1)N1C2=C(NC([C@H](C1)NC=1OC(=NN=1)C1=CC=CC=C1)=O)C=CC=C2 Chemical compound C1(=CC=CC=C1)N1C2=C(NC([C@H](C1)NC=1OC(=NN=1)C1=CC=CC=C1)=O)C=CC=C2 FGUUSXIOTUKUDN-IBGZPJMESA-N 0.000 description 3
- 210000003484 anatomy Anatomy 0.000 description 3
- 238000013459 approach Methods 0.000 description 3
- 238000001574 biopsy Methods 0.000 description 3
- 230000001419 dependent effect Effects 0.000 description 3
- 238000012285 ultrasound imaging Methods 0.000 description 3
- 230000001755 vocal effect Effects 0.000 description 3
- 229910003460 diamond Inorganic materials 0.000 description 2
- 239000010432 diamond Substances 0.000 description 2
- 230000003936 working memory Effects 0.000 description 2
- 238000004891 communication Methods 0.000 description 1
- 238000002594 fluoroscopy Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 238000005259 measurement Methods 0.000 description 1
- 238000012634 optical imaging Methods 0.000 description 1
- 238000002360 preparation method Methods 0.000 description 1
- 239000000523 sample Substances 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 230000002195 synergetic effect Effects 0.000 description 1
- 238000012795 verification Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T19/00—Manipulating 3D models or images for computer graphics
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B34/00—Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
- A61B34/20—Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T19/00—Manipulating 3D models or images for computer graphics
- G06T19/20—Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/30—Determination of transform parameters for the alignment of images, i.e. image registration
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B34/00—Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
- A61B34/10—Computer-aided planning, simulation or modelling of surgical operations
- A61B2034/101—Computer-aided simulation of surgical operations
- A61B2034/102—Modelling of surgical devices, implants or prosthesis
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B34/00—Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
- A61B34/20—Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
- A61B2034/2046—Tracking techniques
- A61B2034/2055—Optical tracking systems
- A61B2034/2057—Details of tracking cameras
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B90/00—Instruments, implements or accessories specially adapted for surgery or diagnosis and not covered by any of the groups A61B1/00 - A61B50/00, e.g. for luxation treatment or for protecting wound edges
- A61B90/36—Image-producing devices or illumination devices not otherwise provided for
- A61B90/37—Surgical systems with images on a monitor during operation
- A61B2090/371—Surgical systems with images on a monitor during operation with simultaneous use of two cameras
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B90/00—Instruments, implements or accessories specially adapted for surgery or diagnosis and not covered by any of the groups A61B1/00 - A61B50/00, e.g. for luxation treatment or for protecting wound edges
- A61B90/36—Image-producing devices or illumination devices not otherwise provided for
- A61B90/37—Surgical systems with images on a monitor during operation
- A61B2090/376—Surgical systems with images on a monitor during operation using X-rays, e.g. fluoroscopy
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B90/00—Instruments, implements or accessories specially adapted for surgery or diagnosis and not covered by any of the groups A61B1/00 - A61B50/00, e.g. for luxation treatment or for protecting wound edges
- A61B90/36—Image-producing devices or illumination devices not otherwise provided for
- A61B90/37—Surgical systems with images on a monitor during operation
- A61B2090/378—Surgical systems with images on a monitor during operation using ultrasound
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10116—X-ray image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2210/00—Indexing scheme for image generation or computer graphics
- G06T2210/41—Medical
Definitions
- the present invention relates to a medical imaging system for tracking an interventional device, and a method for tracking an interventional device inserted partially inside an object, as well as to a computer program element and a computer readable medium.
- interventional devices are used that can be inserted at least partially into an object, e.g. they extend inside the object.
- the inside of the object is invisible to a human eye, for example when examining a patient, the location, and thus also the range of the inserted interventional device cannot be followed without imaging systems that provide respective image data.
- X-ray images are acquired to provide a better understanding of the present situation.
- An examination with an interventional device is, for example, a biopsy procedure.
- US 2011/0125011 A1 it is described to acquire three-dimensional X-ray images as a roadmap. These are then combined with real-time 2D X-ray images that are angularly offset from each other.
- the acquisition of 3D images for a roadmap means additional preparation steps.
- a needle tracking device for ultrasound guided percutaneous procedures (2005-11-01) by CHAN C ET AL discloses a "tracking devicetinct for measuring the position and orientation of a needle with respect to an ultrasound probe. This device is intended to guide an operator during a percutaneous needle insertion so that the needle trajectory can be visually aligned with the target before insertion.
- the tracking device uses a pair of cameras to track the needle location so that a standard needle can be used without attaching a separate sensor to the needle”.
- a medical imaging system for tracking an interventional device comprising an interface unit, and a processing unit.
- the interface unit is configured to provide first image data of a first part of an interventional device, which first part is arranged outside an object.
- the first image data comprises 3D image data.
- the interface unit is configured to provide second image data of a second part of the interventional device, which second part is a continuation of the first part, and which second part is arranged at least partially inside the object.
- the second image data comprises 2D image data.
- the processing unit is configured to compute a first 3D model portion of the interventional device based on the first image data, and to compute a second 3D model portion of the interventional device based on the second image data and the first model portion.
- the interface display unit is configured to provide data of a graphical representation of the interventional device based on the first and second 3D model portions.
- the processing unit may be configured to register the first 3D image data and the second 2D image data.
- a display unit configured to display the graphical representation of the interventional device based on the first and second 3D model portions.
- the medical imaging system further comprises a first image acquisition device, and a second image acquisition device.
- the first image acquisition device is configured to acquire the first image data.
- the second image acquisition device is configured to acquire the second image data.
- the first portion is also referred to as an outside portion and the second portion is referred to as an inserted portion.
- the interventional device may be an elongated device with a longitudinal axis, which is aligned with an insertion direction.
- the first image data is provided by at least two optical cameras.
- the second image data is provided by an X-ray imaging arrangement.
- an ultrasound imaging arrangement may be provided.
- the first image acquisition device is an optical imaging system providing optical, i.e. visible images.
- the second image acquisition device is an X-ray imaging system providing X-ray image data or an ultrasound imaging device providing ultrasound image data.
- the first and second image data may be are acquired live, i.e. in real-time.
- the first image acquisition device may comprise an optical image acquisition device with at least two cameras.
- the first image data may be provided by laser scanners, depth cameras, e.g. time-of-flight cameras, structured light cameras, ultrasound through air and the like. Another possibility is radar.
- the second image acquisition device comprises an X-ray image acquisition device or an ultrasound image acquisition device, according to an example.
- the X-ray image acquisition device may be an X-ray image acquisition device with an X-ray source and an X-ray detector.
- the processing unit is configured to register the first 3D image data and the second 2D image data.
- the display is configured to display the first 3D image data and the second 2D image data in a combined image together with a model representation of the interventional device based on the first and second model portions.
- the processing unit is configured to combine the first model portion and the second model portion to form a 3D model of the interventional device.
- the processing unit is configured to determine a bounding space, in which the interventional device is located'.
- the processing unit is further configured to determine the bounding space based on the first model portion and the second model portion, and/or on the model of the interventional device.
- the processing unit is configured to determine a bounding box, in which a predetermined part of the interventional device is located.
- the interventional device is a flexible interventional device configured for a bending in simple curve geometries.
- the processing unit is configured to determine two possible bending poses of the needle.
- the display is configured to present the bending poses to the user.
- the processing unit is configured to determine a bending of the interventional device for the first image data.
- the processing unit is configured to predict a bending of the interventional device inside the object based on the determined bending in the first image data.
- the processing unit is configured to register the first 3D image data and the second 2D image data.
- the display unit is configured to display the registered images in a combined image.
- a method for tracking an interventional device inserted partially inside an object comprising the steps:
- a step e) is provided, in which the first model portion and the second model portion are combined to form a 3D model of the interventional device.
- a step f) is provided, in which a bounding space is determined, in which the interventional device is located.
- the determination of the bounding space is based on the first model portion and the second model portion, and/or on the model of the interventional device.
- bounding space refers to the space that defines the possible boundaries of the area in which the device may be positioned, i.e. the determined location considering certain deviations that are possible and reasonable.
- a bounding box is determined, in which a predetermined part of the interventional device is located.
- a confidence volume is determined for the range of location of the part of the device arranged inside the object. Based on the length of the device, for example a needle, it is possible to determine the length of the portion arranged outside the body on behalf of the 3D image data. Further, the length of the portion inside the object can be determined. Together with the direction, i.e. orientation of the outside part, the insertion point is the starting point for the confidence volume. Depending on the rigidity, i.e. flexibility of the device, the inside part could be bended inside the object, i.e. deviate from the straight or linear direction as given by the orientation. The possible space in which the device could thus be located can be visualized to the user as the confidence volume.
- the interventional device is rigid, wherein a location of the predetermined part is determined.
- the interventional device is flexible, wherein a location of the predetermined part is determined.
- the bounding spatial region i.e. the bounding space
- the bounding spatial region is presented to the user in relation with image data of the object.
- a planned needle path is shown overlay to the determined model.
- an interventional device tracking e.g. needle tracking
- the interventional device profile in three dimensions, corresponding to the outside of the object, which is obtained from, for example, a video setup is combined with the two-dimensional device footprint corresponding to the object's inside, which 2D needle or device footprint is obtained from needle tracking, for example, in live X-ray images.
- an a-priori known length of the device i.e. a length known before using the device, can be used in these combination steps.
- the present approach allows the modelling of the full needle device extent, i.e.
- the outside and inside parts and the determining where the object's front end is located. This determination may be achieved with various degrees of accuracy, depending on assumptions and needle types. For example, rigid needles can be located with high perfection.
- the provision of three-dimensional image data of the outside portion of the device can be arranged, for example, with optical cameras, thus providing an enhanced and facilitated way of providing the 3D outside data. Further, the provision of image data of the inside of the object is having a reduced effort, since only two-dimensional image data is provided.
- the 3D outside data is used for providing the missing third dimension of the device inside, i.e. 3D image data inside the object is provided by the combination of the 3D outside data and the 2D inside data.
- the interventional device is a needle.
- the needle path in the X-ray image does not reveal the needle pose perpendicular to the X-ray image plane. However, in order to assure that the planned target is hit and that critical anatomical structures are avoided, this information is needed in particular.
- the present invention allows determining the needle pose perpendicular to the X-ray image, or at least a limited region in which the needle can be found, without the necessity to move the X-ray gantry or other additional measurements. By tracking the needle in the optical camera images, and combining the information from multiple cameras, it is possible to accurately determine the actual 3D path of the needle for part that is outside the patient's body.
- the needle is tracked in the 2D live X-ray image and the full needle extent (outside plus inside) is modelled, based on the combination of the 3D video needle profile (outside part) and on the 2D X-ray needle footprint (inside part). Further, the a-priori known needle's length is used in this modelling.
- Fig. 1 shows a medical imaging system 10 for tracking an interventional device in a first example, comprising an interface unit 12 and a processing unit 14.
- a display unit 16 may be provided.
- the interface 12 unit is configured to provide first image data, indicated with a first arrow 18, of a first part of an interventional device, which first part is arranged outside an object.
- the first image data comprises 3D image data.
- the interface unit 12 is further configured to provide second image data, indicated with a second arrow 20, of a second part of the interventional device, which second part is a continuation of the first part, and which second part is arranged at least partially inside the object.
- the second image data comprises 2D image data.
- the processing unit 14 is configured to compute a first 3D model portion of the interventional device based on the first image data, and to compute a second 3D model portion of the interventional device based on the second image data of the first model portion.
- the interface unit is configured to provide data of a graphical representation 22 of the interventional device based on the first and second 3D model portions.
- the display unit 16 is configured to display the graphical representation 22 (not further shown in detail in Fig. 1 ) of the interventional device.
- Fig. 2 shows a further example of a medical imaging system 100 for tracking an interventional device 110, wherein the medical imaging system 100 comprises a first image acquisition device 112, for example shown as a first camera 112a and a second camera 112b. Further, a second image acquisition device 114 is provided, together with a processing unit 116.
- the first image acquisition device 112 is configured to acquire the first image data comprising 3D image information, or 3D image data, of a first part of an interventional device, which first part is arranged outside an object, for example a patient 118.
- the first part of the interventional device 110 is indicated with reference numeral 120.
- the second image acquisition device 114 is configured to acquire the second image data comprising 2D image information, or 2D image data, of a second part of the interventional device, which second part is a continuation of the first part, and which second part is arranged, e.g. inserted, at least partially inside the object.
- the second part is indicated with reference numeral 122.
- the processing unit 116 being in data communication or data connection 124 with the other components, is configured to compute a first 3D model portion of the interventional device based on the first image data, and to compute a second 3D model portion of the interventional device based on the second image data and the first model portion. It should be noted that the interface unit as mentioned above is not further shown.
- the first image data is provided by at least two optical cameras 112a and 112b.
- the respective image acquisition is indicated with a camera viewing beam 124a and 124b, respectively.
- a stereo camera may be provided, for example with distance determination means.
- the second image data may be provided by an X-ray imaging arrangement 126, not further shown.
- a dotted beam structure 128 indicates a respective X-ray beam. It must be noted that further details, for example a detector, are not further shown in Fig. 2 .
- the second image data is provided by an ultrasound imaging arrangement.
- the processing unit 116 is configured to register the first 3D image data and the second 2D image data.
- a display 130 is provided, which is configured to display the first 3D image data and the second 2D image data in a combined image together with a model representation of the interventional device based on the first and second model portions.
- the display 130 although shown in relation with the other features of Fig. 2 , does not represent a feature that is absolutely necessary for the embodiment of the other features. Thus, the display 130, shown in Fig. 2 , must be seen as an option.
- Fig. 3 shows a further example of a medical imaging system 100, comprising a C-arm X-ray imaging arrangement 132, with an X-ray source 134 and an X-ray detector 136 provided on opposite ends of a C-arm structure 138.
- a movable support 140 is shown, together with a ceiling mounting system 142.
- an object 144 is shown supported on a movable or adaptable support, for example a patient table 146.
- two cameras 148 are shown for the first image acquisition device.
- the processing unit 116 is shown in the foreground with monitors 150 and user interfaces 152, for example a mouse, a keyboard, a touchpad or other control panels.
- lighting equipment 154 is shown, together with a larger display arrangement 156.
- a C-arm also other types of X-ray image acquisition can be provided.
- Fig. 4 shows a method 200 for tracking an interventional device inserted partially inside an object, comprising the following steps.
- first image data 212 of a first part of an interventional device is acquired, which first part is arranged outside an object, wherein the first image data comprises 3D image data.
- second image data 216 is acquired of a second part of the interventional device, which second part is a continuation of the first part, and which second part is arranged inside the object, wherein the second image data comprises 2D image data.
- a first 3D model portion 220 of the interventional device is computed based on the first image data.
- a second 3D model portion 224 of the interventional device is computed based on the second image data and the first model portion.
- the first step 210 is also referred to as step a), the second step 214 as step b), the third step 218 as step c), and the fourth step 222 as step d).
- the interventional device position perpendicular to the image plane of the second image data may be determined based on the 3D information provided by the first image data. For the position, a limited region, in which the device can be found, may be determined.
- the first part is also referred to as first fragment and the second part is also referred to as second fragment.
- the interventional device For computing the first model portion, the interventional device is determined in the first image data, and for computing the second model portion, the interventional device is determined in the second image data.
- the interventional device For computing the first model portion, the interventional device may be tracked in the first image data, and for computing the second model portion, the interventional device is tracked in the second image data.
- a first end of the interventional device may be tracked and an entry point, where the interventional device enters the object's surface, is determined.
- Fig. 5 shows a further example of the method, wherein a fifth step 226 is provided, which is also referred to as step e).
- step e the first model portion and the second model portion are combined to form a 3D model 228 of the interventional device.
- step e For example, for computing of the second model portion in step e), this is also based on a predetermined length of the interventional device.
- the interventional device may be a biopsy needle.
- the fifth step 226 is also referred to as step e).
- a sixth step 230 may be provided, in which a determination 234 of a bounding space takes place, in which bounding space the interventional device is located.
- the determination of the bounding space is based on the first model portion and the second model portion; and/or on the model of the interventional device.
- the bounding space may also be referred to as bounding spatial region.
- the sixth step 230 is also referred to as step f).
- step f) can be provided in combination with step e), following step e), or in combination with steps a) to d), following step d).
- a bounding box may be determined, in which a predetermined part of the interventional device is located.
- the bounding spatial region may be presented to the user in relation with image data of the object.
- a device track for the interventional device is computed based on at least the first model portion.
- the interventional device may be a biopsy needle and the device track is a needle path.
- a bending of the interventional device is determined for the first image data; and a bending of the interventional device inside the object is predicted, based on the determined bending in the first image data.
- a volumetric region is determined indicating the expected position of the interventional device.
- a bending of the interventional device is determined for the first part of the interventional device; and a bending of the interventional device is determined for the second part of the interventional device.
- a cone-shaped region is determined in which the second part is located.
- the first 3D image data and the second 2D image data are registered and displayed in a combined image.
- Fig. 7 shows an example of a displayed image, in which optical images from two cameras are fused with a planned needle path and a three-dimensional X-ray reconstruction.
- a user's hand 310 is visible, holding a needle 312.
- a small circle 314 indicates an insertion point of the needle, i.e. a point where the needle enters the inside of the patient.
- the part of the needle inside the object, i.e. the patient for example is also indicated with reference numeral 316. Similar is also shown in right half of Fig. 7 .
- a planned needle path 318 is shown in the upper left half. Further, a live fluoroscopy image with fused overlay is shown in the upper row in the middle portion and the right portion, as well as in the lower row for the left portion and the middle portion.
- the bounding region in which the needle, and in particular the needle tip, can possibly be present, can be determined, and visualized to a physician.
- the physician can decide to move the gantry to another angle to verify the needle location, in case the bounding region seems to be critical.
- the physician can decide to make a 3D reconstruction, for example using Philips XperCT, to verify the needle location.
- the two possible poses can be indicated to the physician, which shall also be described in the following.
- the bending outside the body detected by the optical cameras for example, the bending inside the body can be predicted. If a needle is rigid, the exact needle tip location can be determined in three-dimension, i.e. the bounding box is shrunk to a point.
- FIG. 9a an X-ray source 410 is shown, and a needle 412 arranged between the X-ray source 410 and an X-ray detector 414. On the detector 414, a projection 412' of the needle is indicated with a dotted line.
- FIG. 10 showing a first line 510 indicating the needle, where a diamond shape indicates a head 512, and a ball shape 514 indicates a skin entry point. Further, the optical images show a handle portion 516. A further line 518 indicates a possible position inside the object.
- Fig. 9b shows the needle 412 viewed under two different viewing angles, namely from a first vocal spot 410a and a second vocal spot 410b.
- the first focal spot is projecting towards a first detector 414a
- the second vocal spot 410b is projecting towards a detector 414b.
- respective needle projections 412'a and 412'b are provided.
- the tracked needle fragment from the optical images can be reprojected into 3D space.
- the intersection of the reprojections delivers the actual needle fragment location in 3D space.
- Fig. 9b relates to the image data provided by the optical cameras, for example.
- the lines and projections indicating the needle 412 are also shown with a diamond shape on one end, representing the needle's head, and a ball shape at the other end, indicating the needle's entry portion.
- Fig. 9c refers to a first embodiment for stiff or straight needles, where the combination of the 3D position of the needle fragment outside the patient's body, as taken from the optical camera images (see Fig. 9b ), and the X-ray image, as described in Fig. 9a , is combined to deliver the exact 3D needle location.
- a dotted line 420 indicates the portion arranged inside the body
- a straight line 422 indicates the portion of the needle arranged outside the body.
- the a-priori knowledge of the needle length is combined with the 3D position of the needle fragment from the optical cameras, such that the exact location within the patient's anatomy is known for stiff, straight needles.
- both described approaches using the X-ray image and using a-priori knowledge of the needle length
- a volumetric region 424 can be indicated, regarding the expected position of the needle. It can be taken into account that increasing bending angles are increasingly unlikely. The confidence region can then be visualized in the fused images.
- this method can be further defined by estimating the bending in the hidden needle fragment by extrapolating any bending found in the optically visible needle fragment.
- Fig. 9e relates to a further exemplary embodiment, where the tracked needle fragment in 3D from the optical cameras is combined with the a-priori knowledge of the needle length, the X-ray image, and a simple model for the bending of the needle, it is possible to find or several possible solutions that fit all information.
- Fig. 9e indicates the portion of the needle arranged outside the body, indicated with straight line 426, as described above, and in combination a predicted needle, indicated with thinner straight line 428. On the X-ray detector 414, the respective projection 430 is shown.
- a second degree polynomial or spline is used, together with the 3D optically visible fragment 426 and the X-ray image, i.e. the projection 430, can be used to predict the actual 3D bended needle part 428.
- Fig. 11 shows a graphical representation of the line drawing of Fig. 7
- Fig. 12 shows a graphical representation of Fig. 8
- Fig. 13 shows a graphical representation of Fig. 10 .
- a computer program or a computer program element is provided that is characterized by being adapted to execute the method steps of the method according to one of the preceding embodiments, on an appropriate system.
- the computer program element might therefore be stored on a computer unit, which might also be part of an embodiment of the present invention.
- This computing unit may be adapted to perform or induce a performing of the steps of the method described above. Moreover, it may be adapted to operate the components of the above described apparatus.
- the computing unit can be adapted to operate automatically and/or to execute the orders of a user.
- a computer program may be loaded into a working memory of a data processor.
- the data processor may thus be equipped to carry out the method of the invention.
- This exemplary embodiment of the invention covers both, a computer program that right from the beginning uses the invention and a computer program that by means of an up-date turns an existing program into a program that uses the invention.
- the computer program element might be able to provide all necessary steps to fulfil the procedure of an exemplary embodiment of the method as described above.
- a computer readable medium such as a CD-ROM
- the computer readable medium has a computer program element stored on it which computer program element is described by the preceding section.
- a computer program may be stored and/or distributed on a suitable medium, such as an optical storage medium or a solid state medium supplied together with or as part of other hardware, but may also be distributed in other forms, such as via the internet or other wired or wireless telecommunication systems.
- a suitable medium such as an optical storage medium or a solid state medium supplied together with or as part of other hardware, but may also be distributed in other forms, such as via the internet or other wired or wireless telecommunication systems.
- the computer program may also be presented over a network like the World Wide Web and can be downloaded into the working memory of a data processor from such a network.
- a medium for making a computer program element available for downloading is provided, which computer program element is arranged to perform a method according to one of the previously described embodiments of the invention.
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Health & Medical Sciences (AREA)
- Computer Hardware Design (AREA)
- Software Systems (AREA)
- General Engineering & Computer Science (AREA)
- Computer Graphics (AREA)
- Surgery (AREA)
- Life Sciences & Earth Sciences (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Architecture (AREA)
- Robotics (AREA)
- Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
- Biomedical Technology (AREA)
- Heart & Thoracic Surgery (AREA)
- Medical Informatics (AREA)
- Molecular Biology (AREA)
- Animal Behavior & Ethology (AREA)
- General Health & Medical Sciences (AREA)
- Public Health (AREA)
- Veterinary Medicine (AREA)
- Apparatus For Radiation Diagnosis (AREA)
- Measuring And Recording Apparatus For Diagnosis (AREA)
Claims (12)
- Medizinisches Bildgebungssystem (10; 100) zum Verfolgen einer interventionellen Vorrichtung, umfassend:- eine Schnittstelleneinheit (12); und- eine Verarbeitungseinheit (14; 116);wobei die Schnittstelleneinheit konfiguriert ist, um erste Bilddaten (18) eines ersten Teils einer interventionellen Vorrichtung bereitzustellen, wobei der erste Teil außerhalb eines Objekts angeordnet ist; wobei die ersten Bilddaten 3D-Bilddaten umfassen;
wobei die Schnittstelleneinheit konfiguriert ist, um zweite Bilddaten (20) eines zweiten Teils der interventionellen Vorrichtung bereitzustellen, wobei der zweite Teil eine Fortsetzung des ersten Teils ist und wobei der zweite Teil zumindest teilweise innerhalb des Objekts angeordnet ist; wobei die zweiten Bilddaten 2D-Bilddaten umfassen;
wobei die Verarbeitungseinheit konfiguriert ist, um einen ersten 3D-Modellabschnitt der interventionellen Vorrichtung basierend auf den ersten Bilddaten zu berechnen und um einen zweiten 3D-Modellabschnitt der interventionellen Vorrichtung basierend auf den zweiten Bilddaten und dem ersten Modellabschnitt zu berechnen, indem eine Biegung der interventionellen Vorrichtung innerhalb des Objekts geschätzt wird, indem eine ermittelte Biegung der interventionellen Vorrichtung in den ersten Bilddaten unter Verwendung eines Biegungsmodells und der a-priori-Kenntnis der Nadellänge extrapoliert wird; und
wobei die Schnittstelleneinheit konfiguriert ist, um Daten einer graphischen Darstellung (22) der interventionellen Vorrichtung basierend auf dem ersten und dem zweiten 3D-Modellabschnitt bereitzustellen; und
wobei die interventionelle Vorrichtung eine sich biegende Nadel ist;
wobei die ersten Bilddaten durch mindestens zwei optische Kameras bereitgestellt werden; und
wobei die zweiten Bilddaten ein Röntgenbild sind. - System nach Anspruch 1, umfassend eine Anzeigeeinheit (16; 130), die konfiguriert ist, um eine graphische Darstellung (22) einer interventionellen Vorrichtung basierend auf dem ersten und dem zweiten 3D-Modellabschnitt anzuzeigen.
- System nach Anspruch 1 oder 2, das weiterhin Folgendes umfasst:- eine erste Bilderfassungsvorrichtung (112); und- eine zweite Bilderfassungsvorrichtung (114);wobei die erste Bilderfassungsvorrichtung konfiguriert ist, um die ersten Bilddaten zu erfassen;
wobei die zweite Bilderfassungsvorrichtung konfiguriert ist, um die zweiten Bilddaten zu erfassen. - System nach Anspruch 2 oder 3, wobei die Verarbeitungseinheit konfiguriert ist, um die ersten 3D-Bilddaten mit den zweiten 2D-Bilddaten zu registrieren; und
wobei die Anzeige konfiguriert ist, um die ersten 3D-Bilddaten und die zweiten 2D-Bilddaten in einem kombinierten Bild zusammen mit einer Modelldarstellung der interventionellen Vorrichtung basierend auf dem ersten und dem zweiten Modellabschnitt anzuzeigen. - System nach einem der vorhergehenden Ansprüche, wobei die Verarbeitungseinheit konfiguriert ist, um den ersten Modellabschnitt und den zweiten Modellabschnitt zu kombinieren, um ein 3D-Modell der interventionellen Vorrichtung zu bilden.
- System nach einem der vorhergehenden Ansprüche, wobei die Verarbeitungseinheit konfiguriert ist, um einen Begrenzungsraum zu ermitteln, in dem sich die interventionelle Vorrichtung befindet; und
wobei die Verarbeitungseinheit konfiguriert ist, um den Begrenzungsraum basierend auf dem ersten Modellabschnitt und dem zweiten Modellabschnitt und/oder auf dem Modell der interventionellen Vorrichtung zu ermitteln. - System nach einem der vorhergehenden Ansprüche, wobei die Verarbeitungseinheit konfiguriert ist, um einen Begrenzungsrahmen zu ermitteln, in dem sich ein vorgegebener Teil der interventionellen Vorrichtung befindet.
- System nach einem der vorhergehenden Ansprüche, wobei die interventionelle Vorrichtung eine biegsame interventionelle Vorrichtung ist, die zum Biegen in einfachen Kurvengeometrien konfiguriert ist;
und wobei die Verarbeitungseinheit konfiguriert ist, um zwei mögliche Biegungsposen der Nadel zu ermitteln; und wobei die Anzeige konfiguriert ist, um dem Benutzer die Biegungsposen darzustellen. - System nach einem der vorhergehenden Ansprüche, wobei die Verarbeitungseinheit konfiguriert ist, um die ersten 3D-Bilddaten mit den zweiten 2D-Bilddaten zu registrieren; und
wobei die Anzeigeeinheit konfiguriert ist, um die registrierten Bilder in einem kombinierten Bild anzuzeigen. - Verfahren (200) zum Verfolgen einer interventionellen Vorrichtung, die teilweise in ein Objekt eingeführt ist, wobei das Verfahren die folgenden Schritte umfasst:a) Erfassen (210) von ersten Bilddaten (212) eines ersten Teils einer interventionellen Vorrichtung bereitzustellen, wobei der erste Teil außerhalb eines Objekts angeordnet ist; wobei die ersten Bilddaten 3D-Bilddaten umfassen;b) Erfassen (214) von zweiten Bilddaten (216) eines zweiten Teils der interventionellen Vorrichtung, wobei der zweite Teil eine Fortsetzung des ersten Teils ist und wobei der zweite Teil innerhalb des Objekts angeordnet ist; wobei die zweiten Bilddaten 2D-Bilddaten umfassen;c) Berechnen (218) eines ersten 3D-Modellabschnitts (220) der interventionellen Vorrichtung basierend auf den ersten Bilddaten; undd) Berechnen (222) eines zweiten 3D-Modellabschnitts (224) der interventionellen Vorrichtung basierend auf den zweiten Bilddaten und dem ersten Modellabschnitt, indem eine Biegung der interventionellen Vorrichtung innerhalb des Objekts geschätzt wird, indem eine ermittelte Biegung der interventionellen Vorrichtung in den ersten Bilddaten unter Verwendung eines Biegungsmodells und der a-priori-Kenntnis der Nadellänge extrapoliert wird;e) Bereitstellen von Daten einer graphischen Darstellung (22) der interventionellen Vorrichtung basierend auf dem ersten und dem zweiten 3D-Modellabschnitt;wobei die interventionelle Vorrichtung eine sich biegende Nadel ist;
wobei die ersten Bilddaten durch mindestens zwei optische Kameras bereitgestellt werden; und
wobei die zweiten Bilddaten ein Röntgenbild sind. - Computerprogrammelement zum Steuern eines Geräts nach einem der Ansprüche 1 bis 9, das, wenn es durch eine Verarbeitungseinheit ausgeführt wird, dafür ausgelegt ist, das Verfahren nach Anspruch 10 durchzuführen.
- Computerlesbares Medium, auf dem das Programmelement nach Anspruch 11 gespeichert ist.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
EP13737409.6A EP2831841B1 (de) | 2012-06-20 | 2013-05-29 | Multikamera-tracking |
Applications Claiming Priority (3)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
EP12305700 | 2012-06-20 | ||
PCT/IB2013/054424 WO2013190409A2 (en) | 2012-06-20 | 2013-05-29 | Multicamera device tracking |
EP13737409.6A EP2831841B1 (de) | 2012-06-20 | 2013-05-29 | Multikamera-tracking |
Publications (2)
Publication Number | Publication Date |
---|---|
EP2831841A2 EP2831841A2 (de) | 2015-02-04 |
EP2831841B1 true EP2831841B1 (de) | 2016-05-18 |
Family
ID=48793334
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
EP13737409.6A Active EP2831841B1 (de) | 2012-06-20 | 2013-05-29 | Multikamera-tracking |
Country Status (6)
Country | Link |
---|---|
US (1) | US10255721B2 (de) |
EP (1) | EP2831841B1 (de) |
JP (1) | JP5837261B2 (de) |
CN (1) | CN104272349B (de) |
RU (1) | RU2015101519A (de) |
WO (1) | WO2013190409A2 (de) |
Families Citing this family (15)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US10842461B2 (en) * | 2012-06-21 | 2020-11-24 | Globus Medical, Inc. | Systems and methods of checking registrations for surgical systems |
EP2884900B1 (de) | 2012-08-17 | 2018-03-07 | Koninklijke Philips N.V. | Kamerabasierte visuelle einstellung eines beweglichen röntgenbildgebungssystems |
GB201304798D0 (en) | 2013-03-15 | 2013-05-01 | Univ Dundee | Medical apparatus visualisation |
JP6157919B2 (ja) * | 2013-05-09 | 2017-07-05 | 東芝メディカルシステムズ株式会社 | X線診断装置 |
WO2015193150A1 (en) * | 2014-06-17 | 2015-12-23 | Koninklijke Philips N.V. | Guidance device for a tee probe. |
US20160354049A1 (en) * | 2015-06-04 | 2016-12-08 | Biosense Webster (Israel) Ltd. | Registration of coronary sinus catheter image |
JP6878435B2 (ja) * | 2015-12-18 | 2021-05-26 | コーニンクレッカ フィリップス エヌ ヴェKoninklijke Philips N.V. | 医療機器の追跡 |
US10806468B2 (en) * | 2016-03-16 | 2020-10-20 | Koninklijke Philips N.V. | Optical camera selection in multi-modal X-ray imaging |
EP3459044B1 (de) * | 2016-05-19 | 2021-03-10 | Koninklijke Philips N.V. | Bewegungskompensierung in hybriden röntgenstrahl/kamera-interventionen |
CN107133028B (zh) * | 2017-03-30 | 2021-07-16 | 联想(北京)有限公司 | 一种信息处理方法及电子设备 |
US20190110685A1 (en) * | 2017-10-13 | 2019-04-18 | General Electric Company | Systems and methods for guided interventional procedures |
CN112261965A (zh) * | 2018-06-11 | 2021-01-22 | 泰尔茂株式会社 | 诊断方法、诊断系统及诊断系统的控制方法 |
EP3703011A1 (de) | 2019-02-26 | 2020-09-02 | Koninklijke Philips N.V. | Interventionelle vorrichtungsverfolgung |
EP3824839A1 (de) | 2019-11-19 | 2021-05-26 | Koninklijke Philips N.V. | Roboterpositionierung einer vorrichtung |
IL292783A (en) * | 2022-05-04 | 2023-12-01 | Vidisco Ltd | A system and method for the three-dimensional detection of an object |
Citations (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20060258938A1 (en) * | 2005-05-16 | 2006-11-16 | Intuitive Surgical Inc. | Methods and system for performing 3-D tool tracking by fusion of sensor and/or camera derived data during minimally invasive robotic surgery |
Family Cites Families (28)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US4400890A (en) * | 1981-10-26 | 1983-08-30 | Sony Corporation | Length measuring device |
DE69424733T2 (de) | 1993-04-20 | 2001-02-01 | General Electric Co., Schenectady | Graphisches digitalverarbeitungssystem und echtzeitvideosystem zur verbesserung der darstellung von körperstrukturen während eines chirugischen eingriffs. |
US5765561A (en) | 1994-10-07 | 1998-06-16 | Medical Media Systems | Video-based surgical targeting system |
US6122541A (en) * | 1995-05-04 | 2000-09-19 | Radionics, Inc. | Head band for frameless stereotactic registration |
RU2234244C2 (ru) | 1997-05-23 | 2004-08-20 | Дзе Каролинас Харт Инститьют | Электромагнитные отображающие и лечебные (эмол) системы |
US5930329A (en) * | 1997-09-22 | 1999-07-27 | Siemens Corporate Research, Inc. | Apparatus and method for detection and localization of a biopsy needle or similar surgical tool in a radiographic image |
WO2000063658A2 (en) * | 1999-04-15 | 2000-10-26 | Ultraguide Ltd. | Apparatus and method for detecting the bending of medical invasive tools in medical interventions |
JP2001061861A (ja) * | 1999-06-28 | 2001-03-13 | Siemens Ag | 画像撮影手段を備えたシステムおよび医用ワークステーション |
US6782287B2 (en) | 2000-06-27 | 2004-08-24 | The Board Of Trustees Of The Leland Stanford Junior University | Method and apparatus for tracking a medical instrument based on image registration |
JP4828802B2 (ja) * | 2004-05-12 | 2011-11-30 | 株式会社東芝 | 穿刺治療のための超音波診断装置 |
US8126224B2 (en) * | 2004-02-03 | 2012-02-28 | Ge Medical Systems Global Technology Company, Llc | Method and apparatus for instrument tracking on a scrolling series of 2D fluoroscopic images |
US20080020362A1 (en) * | 2004-08-10 | 2008-01-24 | Cotin Stephane M | Methods and Apparatus for Simulaton of Endovascular and Endoluminal Procedures |
US7641638B2 (en) * | 2004-12-15 | 2010-01-05 | Wilson-Cook Medical Inc. | Flexible elongate surgical needle device having a tissue engaging section being of greater flexibility than an intermediate section, and methods of using the device |
BRPI0709421A2 (pt) * | 2006-03-30 | 2011-07-12 | Activiews Ltd | sistema de mediação de posição de uma ferramenta manual, método para guiar a extremidade distl de uma ferramenta e atadura |
US8165360B2 (en) * | 2006-12-06 | 2012-04-24 | Siemens Medical Solutions Usa, Inc. | X-ray identification of interventional tools |
EP2082686B1 (de) * | 2008-01-22 | 2015-01-21 | Brainlab AG | Orientierte Wiedergabe von Aufnahmen |
US8172753B2 (en) * | 2008-07-11 | 2012-05-08 | General Electric Company | Systems and methods for visualization of an ultrasound probe relative to an object |
WO2010067281A1 (en) | 2008-12-11 | 2010-06-17 | Koninklijke Philips Electronics N.V. | System and method for generating images of a patient's interior and exterior |
US8690776B2 (en) * | 2009-02-17 | 2014-04-08 | Inneroptic Technology, Inc. | Systems, methods, apparatuses, and computer-readable media for image guided surgery |
US9955927B2 (en) | 2009-05-13 | 2018-05-01 | Koninklijke Philips N.V. | System for detecting global patient movement during imaging procedures |
US20100292565A1 (en) | 2009-05-18 | 2010-11-18 | Andreas Meyer | Medical imaging medical device navigation from at least two 2d projections from different angles |
WO2011030260A2 (en) | 2009-09-08 | 2011-03-17 | Koninklijke Philips Electronics N.V. | Sterilization coat |
US9014780B2 (en) | 2009-11-20 | 2015-04-21 | Koninklijke Philips N.V. | Image-based biopsy guidance method |
WO2011127379A2 (en) * | 2010-04-09 | 2011-10-13 | University Of Florida Research Foundation Inc. | Interactive mixed reality system and uses thereof |
FR2960332B1 (fr) * | 2010-05-21 | 2013-07-05 | Gen Electric | Procede de traitement d'images radiologiques pour determiner une position 3d d'une aiguille. |
JP5486449B2 (ja) * | 2010-09-28 | 2014-05-07 | 富士フイルム株式会社 | 超音波画像生成装置及び超音波画像生成装置の作動方法 |
US9572539B2 (en) * | 2011-04-08 | 2017-02-21 | Imactis | Device and method for determining the position of an instrument in relation to medical images |
US20130211244A1 (en) * | 2012-01-25 | 2013-08-15 | Surgix Ltd. | Methods, Devices, Systems, Circuits and Associated Computer Executable Code for Detecting and Predicting the Position, Orientation and Trajectory of Surgical Tools |
-
2013
- 2013-05-29 JP JP2015517880A patent/JP5837261B2/ja active Active
- 2013-05-29 CN CN201380022752.1A patent/CN104272349B/zh active Active
- 2013-05-29 RU RU2015101519A patent/RU2015101519A/ru not_active Application Discontinuation
- 2013-05-29 EP EP13737409.6A patent/EP2831841B1/de active Active
- 2013-05-29 US US14/400,936 patent/US10255721B2/en active Active
- 2013-05-29 WO PCT/IB2013/054424 patent/WO2013190409A2/en active Application Filing
Patent Citations (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20060258938A1 (en) * | 2005-05-16 | 2006-11-16 | Intuitive Surgical Inc. | Methods and system for performing 3-D tool tracking by fusion of sensor and/or camera derived data during minimally invasive robotic surgery |
Also Published As
Publication number | Publication date |
---|---|
JP5837261B2 (ja) | 2015-12-24 |
WO2013190409A2 (en) | 2013-12-27 |
CN104272349B (zh) | 2018-03-02 |
CN104272349A (zh) | 2015-01-07 |
US10255721B2 (en) | 2019-04-09 |
EP2831841A2 (de) | 2015-02-04 |
US20150164607A1 (en) | 2015-06-18 |
JP2015519985A (ja) | 2015-07-16 |
WO2013190409A3 (en) | 2014-03-20 |
RU2015101519A (ru) | 2016-08-10 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
EP2831841B1 (de) | Multikamera-tracking | |
EP3073926B1 (de) | Interventionelles röntgensystem mit automatischer isozentrierung | |
EP2874556B1 (de) | Bildgebungssystem mit erweiterter realität für chirurgische instrumentenführung | |
EP2861149B1 (de) | Computertomografiesystem | |
EP2099378B1 (de) | Vorrichtung zur bestimmung einer position eines ersten objekts innerhalb eines zweiten objekts | |
US8165660B2 (en) | System and method for selecting a guidance mode for performing a percutaneous procedure | |
CN106163408B (zh) | 使用同时的x平面成像的图像配准和引导 | |
EP2680755B1 (de) | Visualisierung für eine navigationsführung | |
JP2019500185A (ja) | 放射線照射を低減された手術中の3次元視覚化 | |
US10478143B2 (en) | System and method of generating and updatng a three dimensional model of a luminal network | |
US20080234575A1 (en) | Method and device for making correction information available | |
US20070189457A1 (en) | Method for displaying a devise in a 3-D image of a volumetric data set | |
EP2916740B1 (de) | Erweiterung von ultraschallbildern | |
WO2015193150A1 (en) | Guidance device for a tee probe. | |
JP2003079616A (ja) | 身体部位に挿入された検査器具の3次元位置の検出方法 | |
US20220096165A1 (en) | Interventional device tracking | |
CN110313991B (zh) | 静态虚拟相机定位 | |
CN114533267A (zh) | 一种2d图像手术定位导航系统及方法 | |
Kahrs et al. | Measurement of distances between anatomical structures using a translating stage with mounted endoscope | |
CN110322506A (zh) | 定位体腔的开口 |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PUAI | Public reference made under article 153(3) epc to a published international application that has entered the european phase |
Free format text: ORIGINAL CODE: 0009012 |
|
17P | Request for examination filed |
Effective date: 20141027 |
|
AK | Designated contracting states |
Kind code of ref document: A2 Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR |
|
AX | Request for extension of the european patent |
Extension state: BA ME |
|
17Q | First examination report despatched |
Effective date: 20150320 |
|
DAX | Request for extension of the european patent (deleted) | ||
GRAP | Despatch of communication of intention to grant a patent |
Free format text: ORIGINAL CODE: EPIDOSNIGR1 |
|
INTG | Intention to grant announced |
Effective date: 20160216 |
|
GRAS | Grant fee paid |
Free format text: ORIGINAL CODE: EPIDOSNIGR3 |
|
GRAA | (expected) grant |
Free format text: ORIGINAL CODE: 0009210 |
|
AK | Designated contracting states |
Kind code of ref document: B1 Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR |
|
REG | Reference to a national code |
Ref country code: GB Ref legal event code: FG4D |
|
REG | Reference to a national code |
Ref country code: CH Ref legal event code: EP |
|
REG | Reference to a national code |
Ref country code: IE Ref legal event code: FG4D Ref country code: AT Ref legal event code: REF Ref document number: 801054 Country of ref document: AT Kind code of ref document: T Effective date: 20160615 |
|
REG | Reference to a national code |
Ref country code: DE Ref legal event code: R096 Ref document number: 602013007715 Country of ref document: DE |
|
REG | Reference to a national code |
Ref country code: NL Ref legal event code: MP Effective date: 20160518 |
|
REG | Reference to a national code |
Ref country code: LT Ref legal event code: MG4D |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: NL Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20160518 Ref country code: FI Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20160518 Ref country code: LT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20160518 Ref country code: NO Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20160818 |
|
REG | Reference to a national code |
Ref country code: AT Ref legal event code: MK05 Ref document number: 801054 Country of ref document: AT Kind code of ref document: T Effective date: 20160518 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: PT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20160919 Ref country code: SE Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20160518 Ref country code: ES Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20160518 Ref country code: HR Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20160518 Ref country code: LV Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20160518 Ref country code: RS Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20160518 Ref country code: GR Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20160819 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: IT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20160518 Ref country code: BE Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20160531 |
|
REG | Reference to a national code |
Ref country code: CH Ref legal event code: PL |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: DK Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20160518 Ref country code: CZ Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20160518 Ref country code: EE Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20160518 Ref country code: SK Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20160518 Ref country code: LI Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20160531 Ref country code: RO Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20160518 Ref country code: CH Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20160531 |
|
REG | Reference to a national code |
Ref country code: DE Ref legal event code: R097 Ref document number: 602013007715 Country of ref document: DE |
|
REG | Reference to a national code |
Ref country code: IE Ref legal event code: MM4A |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: BE Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20160518 Ref country code: SM Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20160518 Ref country code: PL Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20160518 Ref country code: AT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20160518 |
|
PLBE | No opposition filed within time limit |
Free format text: ORIGINAL CODE: 0009261 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: MC Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20160518 |
|
REG | Reference to a national code |
Ref country code: FR Ref legal event code: ST Effective date: 20170313 |
|
26N | No opposition filed |
Effective date: 20170221 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: FR Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20160718 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: IE Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20160529 Ref country code: SI Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20160518 |
|
GBPC | Gb: european patent ceased through non-payment of renewal fee |
Effective date: 20170529 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: GB Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20170529 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: HU Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT; INVALID AB INITIO Effective date: 20130529 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: LU Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20160529 Ref country code: CY Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20160518 Ref country code: MT Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20160531 Ref country code: MK Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20160518 Ref country code: IS Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20160518 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: BG Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20160518 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: AL Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20160518 Ref country code: TR Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20160518 |
|
PGFP | Annual fee paid to national office [announced via postgrant information from national office to epo] |
Ref country code: DE Payment date: 20240529 Year of fee payment: 12 |