WO2015094726A1 - Marker-based tool tracking - Google Patents
Marker-based tool tracking Download PDFInfo
- Publication number
- WO2015094726A1 WO2015094726A1 PCT/US2014/068899 US2014068899W WO2015094726A1 WO 2015094726 A1 WO2015094726 A1 WO 2015094726A1 US 2014068899 W US2014068899 W US 2014068899W WO 2015094726 A1 WO2015094726 A1 WO 2015094726A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- marker
- surgical tool
- ophthalmic surgical
- image
- eye
- Prior art date
Links
- 239000003550 marker Substances 0.000 title claims abstract description 103
- 238000003384 imaging method Methods 0.000 claims abstract description 37
- 238000000034 method Methods 0.000 claims abstract description 31
- 239000000284 extract Substances 0.000 claims abstract description 10
- 230000003595 spectral effect Effects 0.000 claims abstract description 8
- 230000008569 process Effects 0.000 claims abstract description 7
- 238000012545 processing Methods 0.000 claims description 6
- 238000001514 detection method Methods 0.000 claims description 5
- 238000001356 surgical procedure Methods 0.000 abstract description 18
- 238000012014 optical coherence tomography Methods 0.000 description 9
- 238000005286 illumination Methods 0.000 description 7
- 239000000523 sample Substances 0.000 description 5
- 210000000695 crystalline len Anatomy 0.000 description 4
- 238000002329 infrared spectrum Methods 0.000 description 4
- 239000003086 colorant Substances 0.000 description 3
- 210000004087 cornea Anatomy 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 210000001525 retina Anatomy 0.000 description 3
- 238000001228 spectrum Methods 0.000 description 3
- 210000002159 anterior chamber Anatomy 0.000 description 2
- 230000008859 change Effects 0.000 description 2
- 238000013534 fluorescein angiography Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 239000003973 paint Substances 0.000 description 2
- 239000007787 solid Substances 0.000 description 2
- 229920002994 synthetic fiber Polymers 0.000 description 2
- 238000010521 absorption reaction Methods 0.000 description 1
- 239000000853 adhesive Substances 0.000 description 1
- 230000001070 adhesive effect Effects 0.000 description 1
- 230000004075 alteration Effects 0.000 description 1
- 238000007796 conventional method Methods 0.000 description 1
- 238000003745 diagnosis Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 230000007613 environmental effect Effects 0.000 description 1
- 238000010191 image analysis Methods 0.000 description 1
- 230000002262 irrigation Effects 0.000 description 1
- 238000003973 irrigation Methods 0.000 description 1
- 239000007788 liquid Substances 0.000 description 1
- 239000012528 membrane Substances 0.000 description 1
- 230000001737 promoting effect Effects 0.000 description 1
- 238000012552 review Methods 0.000 description 1
- 210000003786 sclera Anatomy 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000011477 surgical intervention Methods 0.000 description 1
- 210000004127 vitreous body Anatomy 0.000 description 1
Classifications
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/06—Devices, other than using radiation, for detecting or locating foreign bodies ; determining position of probes within or on the body of the patient
- A61B5/065—Determining position of the probe employing exclusively positioning means located on or in the probe, e.g. using position sensors arranged on the probe
- A61B5/066—Superposing sensor position on an image of the patient, e.g. obtained by ultrasound or x-ray imaging
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B3/00—Apparatus for testing the eyes; Instruments for examining the eyes
- A61B3/10—Objective types, i.e. instruments for examining the eyes independent of the patients' perceptions or reactions
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B3/00—Apparatus for testing the eyes; Instruments for examining the eyes
- A61B3/10—Objective types, i.e. instruments for examining the eyes independent of the patients' perceptions or reactions
- A61B3/102—Objective types, i.e. instruments for examining the eyes independent of the patients' perceptions or reactions for optical coherence tomography [OCT]
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B3/00—Apparatus for testing the eyes; Instruments for examining the eyes
- A61B3/10—Objective types, i.e. instruments for examining the eyes independent of the patients' perceptions or reactions
- A61B3/13—Ophthalmic microscopes
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B34/00—Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
- A61B34/20—Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B34/00—Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
- A61B34/25—User interfaces for surgical systems
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/0059—Measuring for diagnostic purposes; Identification of persons using light, e.g. diagnosis by transillumination, diascopy, fluorescence
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/06—Devices, other than using radiation, for detecting or locating foreign bodies ; determining position of probes within or on the body of the patient
- A61B5/061—Determining position of a probe within the body employing means separate from the probe, e.g. sensing internal probe position employing impedance electrodes on the surface of the body
- A61B5/064—Determining position of a probe within the body employing means separate from the probe, e.g. sensing internal probe position employing impedance electrodes on the surface of the body using markers
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/68—Arrangements of detecting, measuring or recording means, e.g. sensors, in relation to patient
- A61B5/6801—Arrangements of detecting, measuring or recording means, e.g. sensors, in relation to patient specially adapted to be attached to or worn on the body surface
- A61B5/6813—Specially adapted to be attached to a specific body part
- A61B5/6814—Head
- A61B5/6821—Eye
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B90/00—Instruments, implements or accessories specially adapted for surgery or diagnosis and not covered by any of the groups A61B1/00 - A61B50/00, e.g. for luxation treatment or for protecting wound edges
- A61B90/36—Image-producing devices or illumination devices not otherwise provided for
- A61B90/37—Surgical systems with images on a monitor during operation
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61F—FILTERS IMPLANTABLE INTO BLOOD VESSELS; PROSTHESES; DEVICES PROVIDING PATENCY TO, OR PREVENTING COLLAPSING OF, TUBULAR STRUCTURES OF THE BODY, e.g. STENTS; ORTHOPAEDIC, NURSING OR CONTRACEPTIVE DEVICES; FOMENTATION; TREATMENT OR PROTECTION OF EYES OR EARS; BANDAGES, DRESSINGS OR ABSORBENT PADS; FIRST-AID KITS
- A61F9/00—Methods or devices for treatment of the eyes; Devices for putting-in contact lenses; Devices to correct squinting; Apparatus to guide the blind; Protective devices for the eyes, carried on the body or in the hand
- A61F9/007—Methods or devices for eye surgery
- A61F9/00736—Instruments for removal of intra-ocular material or intra-ocular injection, e.g. cataract instruments
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B34/00—Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
- A61B34/20—Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
- A61B2034/2046—Tracking techniques
- A61B2034/2065—Tracking using image or pattern recognition
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B90/00—Instruments, implements or accessories specially adapted for surgery or diagnosis and not covered by any of the groups A61B1/00 - A61B50/00, e.g. for luxation treatment or for protecting wound edges
- A61B90/36—Image-producing devices or illumination devices not otherwise provided for
- A61B2090/364—Correlation of different images or relation of image positions in respect to the body
- A61B2090/365—Correlation of different images or relation of image positions in respect to the body augmented reality, i.e. correlating a live optical image with another image
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B90/00—Instruments, implements or accessories specially adapted for surgery or diagnosis and not covered by any of the groups A61B1/00 - A61B50/00, e.g. for luxation treatment or for protecting wound edges
- A61B90/36—Image-producing devices or illumination devices not otherwise provided for
- A61B2090/364—Correlation of different images or relation of image positions in respect to the body
- A61B2090/368—Correlation of different images or relation of image positions in respect to the body changing the image on a display according to the operator's position
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B90/00—Instruments, implements or accessories specially adapted for surgery or diagnosis and not covered by any of the groups A61B1/00 - A61B50/00, e.g. for luxation treatment or for protecting wound edges
- A61B90/36—Image-producing devices or illumination devices not otherwise provided for
- A61B90/37—Surgical systems with images on a monitor during operation
- A61B2090/373—Surgical systems with images on a monitor during operation using light, e.g. by using optical scanners
- A61B2090/3735—Optical coherence tomography [OCT]
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B90/00—Instruments, implements or accessories specially adapted for surgery or diagnosis and not covered by any of the groups A61B1/00 - A61B50/00, e.g. for luxation treatment or for protecting wound edges
- A61B90/39—Markers, e.g. radio-opaque or breast lesions markers
- A61B2090/3937—Visible markers
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B90/00—Instruments, implements or accessories specially adapted for surgery or diagnosis and not covered by any of the groups A61B1/00 - A61B50/00, e.g. for luxation treatment or for protecting wound edges
- A61B90/30—Devices for illuminating a surgical field, the devices having an interrelation with other surgical devices or with a surgical procedure
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B21/00—Microscopes
- G02B21/0004—Microscopes specially adapted for specific applications
- G02B21/0012—Surgical microscopes
Definitions
- the devices, systems, and methods disclosed herein relate generally to marker-based tool tracking, and more particularly, to devices, systems, and methods that are configured to perform marker-based tool tracking in ophthalmic surgeries.
- Surgical tools such as surgical imaging probes, surgical forceps, surgical scissors, surgical vitrectomy probes, and the like, may be inserted into an eye during an ophthalmic surgery to perform various surgeries in the eye.
- a distal portion of a surgical tool is inserted into the eye during the ophthalmic surgery.
- the area of the eye surrounding the distal tip of the surgical tool is a region of interest to a surgeon.
- guided surgical interventions such as intra-operative Optical Coherence Tomography (OCT) for Internal Limited Membrane (ILM) peeling
- OCT Optical Coherence Tomography
- ILM Internal Limited Membrane
- useful surgical data may be overlaid to the surgeon's current area of interest.
- the area of interest may shift accordingly.
- automatic tool tracking may be used to locate the area of interest to adjust the surgical data overlay so the surgeon may visualize it without looking away from the current area of interest.
- Motion-based object tracking may be used for automated surveillance.
- Motion-based object tracking may use image processing algorithms, such as background subtraction, frame difference, and optical flow, to track an object.
- image processing algorithms such as background subtraction, frame difference, and optical flow.
- motion-based object tracking algorithm requires a quasi- stationary background and may not be suitable for tool tracking in an ophthalmic surgery in which background may vary constantly.
- Region-based object tracking may be used for tracking simple objects.
- an object template is preselected offline or during a first frame. For the subsequent frames, the template is searched across the whole field of view and the location with the greatest similarity to the template is identified as the object.
- region-based object tracking is sensitive to object pose variations and local illumination changes and may not be suitable for tool tracking in an ophthalmic surgery, in which illumination and orientation of the tool vary greatly.
- the third technique for general object tracking is feature-based object tracking.
- Feature-based object tracking may extract and search for unique features of an object, such as contour, edge, shape, color, corner/interest point and the like, across the entire field of view for object detection. Nevertheless, in the feature-based tracking algorithm, a high contrast feature, which is not sensitive to environmental and object pose changes and is unique to the object, is required. Since most of the surgical tools do not possess high contrast intrinsic features, feature based object tracking may not provide suitable results.
- illumination conditions may be challenging for tool tracking.
- An endo illuminator may be inserted into the eye for illumination. Because the endo illuminator may move during a surgery, the illumination condition may vary greatly from image frame to image frame and the images of the fundus area being illuminated may change greatly over time. Motion-based and region-based object tracking techniques may be difficult to implement under inconsistent illumination conditions. Further, with a single illuminator illuminating from one side, shadow artifacts and specular reflection from the surgical tool may increase complexity for tool tracking.
- a beam path of an imaging light from the eye may pass through multiple optical elements and media, such as eye vitreous body, an aged crystalline lens, eye cornea, and Binocular Indirect Ophthalmomicroscope (BIOM) lenses.
- These optical elements in the beam path of imaging light may further degrade the image quality and reduce contrast.
- the present disclosure is directed to devices, systems, and methods that address one or more of the disadvantages of the prior art.
- the present disclosure is directed to an ophthalmic surgical tool tracking system.
- the ophthalmic surgical tool tracking system may include an ophthalmic surgical tool with a distal portion configured to be inserted into an eye, a marker positioned at the distal portion of the ophthalmic surgical tool, and a tool position detection system configured to determine a position of the distal portion of the ophthalmic surgical tool in the eye by detecting the marker.
- the tool position detection system may include a light source configured to introduce light into the eye; an imaging device configured to capture an image of the eye along with the distal portion of the ophthalmic surgical tool inserted into the eye, a processor, and a display.
- the processor is configured to process the captured image to identify the marker, generate indicators indicating surgical data, such as a position of a surgical tool, an orientation of a surgical tool, an image, a surgical setting parameter, overlay the indicators on the captured image or a processed image.
- the display is configured to display the captured image or a processed image with the overlaid indicators.
- the present disclosure is directed to a method for tracking an ophthalmic surgical tool inserted into an eye.
- the method may include introducing an imaging light into an eye, capturing an image of the eye along with a distal portion of an ophthalmic surgical tool inserted into the eye, and determining a position of the distal portion of the ophthalmic surgical tool in the eye by detecting a marker positioned at the distal portion of the ophthalmic surgical tool.
- determining a position of the distal portion may include processing the captured image to identify the marker, generating indicators indicating surgical data, such as a position of a surgical tool, an orientation of a surgical tool, an image, a surgical setting parameter, overlaying the indicators on the captured images or a processed image, and displaying the captured image or a processed image with the overlaid indicators on a display.
- Fig. 1 illustrates a schematic diagram of an exemplary ophthalmic surgical tool tracking system according to an aspect consistent with the principles of the present disclosure.
- Fig. 2 illustrates a distal portion of an exemplary surgical tool according to an aspect consistent with the principles of the present disclosure.
- Fig. 3 illustrates a perspective view of a marker and various types of markers according to an aspect consistent with the principles of the present disclosure.
- Fig. 4 is a flow chart illustrating a method for tool-tracking according to an aspect consistent with the principles of the present disclosure.
- Fig. 5 illustrates an image of a fundus during an ophthalmic surgery according to an aspect consistent with the principles of the present disclosure.
- the devices, systems, and methods described herein provide an ophthalmic surgical tool tracking system that includes an ophthalmic surgical tool with a marker positioned at a distal portion of the ophthalmic surgical tool.
- the system tracks the location of the marker when the distal portion of the ophthalmic surgical tool, along with the marker, is inserted into an eye to perform surgeries in the eye.
- a light source e.g., an endo illuminator, may introduce light into a fundus of the eye. The light may be reflected off the fundus of the eye and the distal portion of the ophthalmic surgical tool.
- An imaging device of the system may receive the reflected light to capture images of the fundus of the eye and the distal portion of the ophthalmic surgical tool.
- the system also may include an image processor which may process the captured images to identify and extract the marker from the captured images.
- the marker may have a high-contrast feature in a visible light or infrared light range or other spectral ranges.
- the image processor may identify and extract the marker from the captured images.
- the location of the marker may be used to generate indicators indicating surgical data, such as a position of a surgical tool, an orientation of a surgical tool, an image, a surgical setting parameter. The indicators are then overlaid with the captured images of the fundus or processed images and displayed to a user.
- the system may indicate surgical data, such as a presence, position, and orientation of the surgical tool, an image, a surgical setting parameter in real time to assist a surgeon during an ophthalmic surgical operation in the eye.
- Fig. 1 illustrates an exemplary ophthalmic surgical tool tracking system, generally designated 100 disposed relative to an eye 101 under treatment.
- the eye 101 includes sclera 102, a cornea 104, an anterior chamber 106, and a posterior chamber 108.
- a capsular bag 110 is illustrated in the posterior chamber 108.
- the eye 101 further includes a retina 112.
- An ophthalmic surgical tool 130 may be used to perform surgery in the eye 101.
- the ophthalmic surgical tool 130 may be sized and shaped to be handled by a surgeon and to protrude into the eye 101 of the patient.
- the ophthalmic surgical tool 130 may include a proximal portion 148 and a distal portion 144.
- the proximal portion 148 may be sized and shaped for handheld grasping by a user.
- the proximal portion 148 may define a handle which is sized and shaped for grasping by a single hand of the user.
- the user may control the position of the distal portion 144 by maneuvering the proximal portion 148.
- the distal portion 144 of the ophthalmic surgical tool 130 may include a marker 114.
- the marker may have a high contrast feature in the visible light or infrared spectrum or other spectral ranges detectable by an imaging device 124 of the ophthalmic surgical tool tracking system 100.
- the ophthalmic surgical tool tracking system 100 also may include a light source 122, e.g., an endo illuminator.
- the light source 122 may have a distal portion that is configured to be inserted into the eye 101.
- a distal tip of the light source 122 may emit an imaging light that may illuminate a fundus of the eye 101.
- the fundus is an interior surface of the eye 101 and may include the retina 112.
- the imaging light from the light source 122 may be reflected from the fundus and the distal portion 144 of the ophthalmic surgical tool 130.
- the reflected imaging light may pass through the capsular bag 110, the anterior chamber 106, the cornea 104, and be received by the imaging device 124, which is configured to capture fundus images of the eye 101.
- Lenses 132 and 134 may be provided between the eye 101 and the imaging device 124 to receive the reflected imaging light from the fundus and direct the imaging light to the imaging device 124.
- the imaging device 124 may include one or more video cameras configured to capture images of the fundus.
- the video camera may capture images in visible light spectrum, infrared spectrum or other spectral ranges.
- imaging device 124 may include either or both a video camera that captures images of the fundus in visible light spectrum and a video camera that captures infrared images of an infrared marker 114 near the fundus in the infrared spectrum.
- the ophthalmic surgical tool tracking system 100 also may include an image processor 126.
- the image processor 126 may receive image frames captured by the imaging device 124 and perform various image processing on the image frames. In particular, the image processor 126 may perform image analysis on the image frames to identify and extract the image of the marker 114 from the image frames. Further, the image processor 126 may generate indicators and overlay the indicators on the image of the fundus or a processed image.
- the indicators may include surgical data, such as the position and orientation of the marker 114 in the image of the fundus, the position and orientation of a distal tip 146 of the ophthalmic surgical tool 130, an image, a surgical setting parameter. The overlaid image may then be displayed by a display 128 to the user.
- the imaging device 124, the image processor 126, and the display 128 may be implemented in separate housings communicatively coupled to one another or within a common console or housing.
- a user interface 136 may be associated with the display 128 and/or the image processor 126. It may include, for example, a keyboard, a mouse, a joystick, a touchscreen, an eye tracking device, a speech recognition device, a gesture control module, dials, and/or buttons, among other input devices.
- a user may enter desired instructions or parameters at the user interface 136 to control the imaging device 124 for taking images of the eye 101.
- a surgeon may review the images of the fundus and/or the overlaid indicators on the display 128 to visualize the operation and the relative position of the distal tip 146 of the ophthalmic surgical tool 130 within various portions of the fundus.
- Fig. 2 is an enlarged view of the distal portion 144 of the ophthalmic surgical tool 130.
- One or more markers 114 may be positioned at the distal portion 144.
- the markers 114 may wrap around the distal portion of the ophthalmic surgical tool 130, such that markers 114 may be visible in all directions even when the ophthalmic surgical tool 130 is rotated.
- the markers 114 may be labels or printed decals bonded to the ophthalmic surgical tool 130.
- the markers 114 may be formed with synthetic material, such as plastic. Thus, the markers 114 may not deteriorate in a biological tissue. Further, the markers 114 may be bio-compatible and do not interfere or react with biological tissues.
- the markers 114 may be inscribed on an exterior surface of the ophthalmic surgical tool 130.
- the markers 114 may be a layer of paint inscribed on the exterior surface of the ophthalmic surgical tool 130.
- the markers 114 may be embedded into the wall of the ophthalmic surgical tool 130.
- the markers 114 may have a high-contrast feature in visible, infrared or other spectrum.
- a high- contrast feature may be a color or a pattern that is distinguishable from colors or patterns in the fundus.
- a high-contrast color may be a green color, which typically does not appear in a fundus of an eye. It is noteworthy that adding markers 114 should not increase the size of the surgical tools.
- Fig. 3 illustrates various examples of the markers 114.
- the marker 114 may have a ring, ribbon shape configured to wrap around the distal portion 144 of the ophthalmic surgical tool 130.
- the marker 114 may have an inner surface 116 and an outer surface 118.
- the inner surface 116 may have adhesives and be configured to adhere or bond to an exterior surface of the ophthalmic surgical tool 130.
- the exterior surface of the distal portion 144 of the ophthalmic surgical tool 130 may have a circumferential groove configured to accommodate the ring, ribbon shape marker 114. Thus, the marker 114 may fit securely in the circumferential groove.
- the outer surface 118 of the marker 114 may have colors or patterns configured to distinguish the marker 114 from other elements in the fundus image.
- One or more markers 114 may be used for the ophthalmic surgical tool 130.
- the marker 114 may be formed of bio-compatible and/or synthetic materials, such as sterile plastic.
- the marker 114 may be a layer of paint inscribed on an exterior surface of the distal portion 144 of the ophthalmic surgical tool 130.
- the markers 114 may overlap each other or be separated from each other.
- the markers 114 may have one or more high contrast colors.
- the markers 114 may have a green color, which does not appear in a typical fundus image.
- the green markers 114 may be distinguished from other elements in the fundus image.
- the markers 114 may have various color, texture, or spectral contrast.
- the markers 114 may include patterns that may identify an orientation and angle of the ophthalmic surgical tool 130. For example, as shown in Fig.
- marker 114a may have a solid high-contrast color. When the ring, ribbon shape marker 114a is cut open, the marker 114a may be a ribbon in solid color. In another example, marker 114b may have a texture pattern that may distinguish the marker 114b from the background fundus image. Exemplary marker 114c may include an infrared color configured to reflect or emit infrared light. Markers 114 with various spectral absorption/emission also may be used.
- the markers 114 may include letters, numbers, bar codes, pattern, symbols, or pictures.
- Exemplary marker 114d may include letters. As shown in Fig. 3, assuming that the marker 114d wraps 360 degrees around the distal portion 144 of the ophthalmic surgical tool 130, a letter “A” may be positioned near the zero degree position and the letter “E” may be positioned near the 360 degree position. Letters “B,” “C,” and “D” may be positioned in between “A” and "E” at respective positions. Thus, based on the orientation of the letters, the rotational position of the marker 114d and indirectly the rotational position of the ophthalmic surgical tool 130 may be determined.
- Exemplary marker 114e may include numbers "1" to "5.”
- the numbers may indicate a rotational position of the ophthalmic surgical tool 130.
- the orientation of the letters or number also may indicate a tilting angle of the ophthalmic surgical tool 130.
- the numbers or letters may be orientated relatively to the distal tip 146 of the ophthalmic surgical tool 130 such that the bottoms of the numbers or letter face toward the distal tip 146.
- the tilting angle of the distal tip 146 may be determined.
- Exemplary marker 114f may include barcodes or stripes.
- the direction of the stripes may indicate a tilting angle of the ophthalmic surgical tool 130.
- the number of stripes may vary to indicate a rotational position of the marker 114f and indirectly, the rotational position of the ophthalmic surgical tool 130.
- Marker 114g has various dot patterns.
- the number of dots may indicate the rotational position of the marker 114f and the alignment of the dots may indicate a tilting angle of the marker 114f.
- Other symbols also may be used on the markers 114.
- various symbols, such as shapes or non-character symbols may be used at different rotational positions of the markers 114h and 114i to indicate rotational positions.
- a picture may be used to indicate rotational and tilt positions of the marker 114j.
- Other patterns or symbols that may indicate an orientation and position of the ophthalmic surgical tool 130 also may be used on the markers 114.
- Fig. 4 is a flow chart illustrating a method 400 for tracking an ophthalmic surgical tool 130 inserted in an eye 101.
- a light source 122 may introduce imaging light into the fundus of the eye 101.
- the reflected imaging light from the fundus may be guided by lenses 132 and 134 and received by imaging device 124.
- the imaging device 124 may capture images of the fundus.
- the imaging device 124 may capture frames of images to form a video. Each image frame may be forwarded to the image processor 126 to be processed and analyzed.
- the image processor 126 may perform contrast and feature enhancement processing on the image frame. For example, the image processor 126 may receive the image frame in Red-Green-Blue (RGB) format. At 404, the image processor 126 may convert the RGB format image frame into a Hue-Saturation- Value (HSV) space. At 406, after the image frame has been enhanced to bring out the contrast and feature, the image processor 126 may determine a first-order estimation mask of the marker 114. For example, based on a predetermined color of the marker 114, the image processor 126 may apply criteria to the hue and saturation channels of the HSV image frame that may separate the marker 114 from the background in order to bring out and estimate the image of the marker 114.
- RGB Red-Green-Blue
- HSV Hue-Saturation- Value
- the image processor 126 may extract the image of the marker 114 from the image frame.
- the image processor 126 may implement a blob detection process to detect a boundary of the marker 114 in the image frame.
- a blob may be a region of the image frame where some properties, such as color and brightness, are approximately constant.
- the image processor 126 may search for regions of approximately constant properties in the image frame to detect blobs.
- the image processor 126 may find the boundary of the marker 114 and extract the marker 114 from the image frame.
- the image processor 126 may analyze the shape and orientation of the marker 114 extracted from the image frame. Based on a predetermined pattern and color, the image processor 126 may determine the orientation of the marker 114 in the image frame. For example, if the marker 114 has stripes, the image processor 126 may determine the orientation of the marker 114 based on the orientation and direction of the stripes.
- the image processor 126 may determine the position and orientation of the distal tip 146 of the ophthalmic surgical tool 130. In particular, based on the position and orientation of the marker 114, the image processor 126 may determine the position and orientation of the distal tip 146 of the ophthalmic surgical tool 130.
- the marker 114 may be positioned from the distal tip 146 of the ophthalmic surgical tool 130 by a predetermined distance and may have a pattern that indicates a pointing direction of the ophthalmic surgical tool 130, e.g. , a strip or an arrow.
- the image processor 126 may determine the position of the distal tip 146 of the ophthalmic surgical tool and the pointing direction or orientation of the ophthalmic surgical tool.
- the image processor 126 may display and overlay indicators to indicate the distal tip 146 of the ophthalmic surgical tool 130 or other surgical data for surgical guidance.
- the image processor 126 may generate an indicator, such as a box, a circle, a star, or an arrow, and overlay the indicator into the image frame at the position of the distal tip 146 of the ophthalmic surgical tool 130.
- the indicator may indicate an orientation, e.g. , a pointing angle, of the ophthalmic surgical tool 130.
- an arrow may be used as the indicator to indicate the pointing direction of the ophthalmic surgical tool 130.
- the indicator may also include an image, such as an OCT image of a region of the retina 112, or a surgical setting parameter, such as a cutting speed of a vitrectomy probe.
- the display 128 may display the image frame overlaid with the indicators.
- Fig. 5 illustrates a fundus image 500 during an ophthalmic surgery.
- the fundus image 500 may be displayed or processed and displayed on the display 128.
- the distal portion 144 of the ophthalmic surgical tool 130 is inserted into the eye 101.
- the distal portion 144 may have a marker 114.
- the image processor 126 extracts the marker image and determines the position and orientation of the marker 114, the image processor 126 may generate and overlay indicators 502, 504, and 506 into the fundus image.
- Indicator 502 may be a dot positioned over the marker 114 or at a position of the surgical tool 130 in the fundus image 500. Thus, the indicator 502 may indicate the position of the marker 114 or the position of the surgical tool 130 in real time. Indicator 504 may be an arrow positioned on or adjacent to the marker 114. The arrow of indicator 504 may indicate an orientation or pointing position of the ophthalmic surgical tool 130. Indicator 506 may be text or image positioned at or adjacent to the marker 114. The indicator may include name or description of the ophthalmic surgical tool. The indicator also may include diagnosis information, surgical status, warnings, or other information. For example, the indicator may include the brand name, identification, and/or functional description. In the example shown in Fig.
- indicator 506 has the text: "Alcon” to indicate the name brand of the ophthalmic surgical tool.
- operating parameters such as temperature, flow rate, revolutions per minute (RPM) speed, pressure, and the like, may be included in the indicator 506 to provide additional information to the user.
- images of the tissue under examination from other imaging modalities such as optical coherence tomography (OCT), fluorescein angiography (FA), and the like, may be included in the indicator 506 to provide additional guidance to the user.
- OCT optical coherence tomography
- FA fluorescein angiography
- the image processor 126 may perform the method 400 for each image frame as the image frame is displayed to continuously track the position and orientation of the distal tip 146 of the ophthalmic surgical tool 130 in real time.
- the display 128 may display the indicators 502, 504, and 506 in a real-time video of the fundus to track the position and movement of the distal tip 146 of the ophthalmic surgical tool 130. Accordingly, a surgeon may view the display 128 to visualize the movement of the ophthalmic surgical tool 130 in the eye 101 during a surgery.
- the above systems and methods may be applied to track an infrared marker 114 using an imaging device 124 configured to capture images in infrared spectrum. Further, an OCT enabled surgical tool 130 may use the above systems and methods to track the area of the eye 101 being imaged by the OCT surgical probe 130 around the area of the distal tip 146 of the OCT surgical probe 130.
- the indicator may include identification or parameters of the ophthalmic surgical tool 130.
- the indicator may identify that the ophthalmic surgical tool 130 is an irrigation tool with a certain liquid flow rate, pressure, and temperature. By displaying these parameters, a surgeon may keep track of the operation of the ophthalmic surgical tool 130 without looking away from the display 128.
Abstract
Description
Claims
Priority Applications (6)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201480069020.2A CN105828703B (en) | 2013-12-19 | 2014-12-05 | Tool tracking based on label |
AU2014366911A AU2014366911B2 (en) | 2013-12-19 | 2014-12-05 | Marker-based tool tracking |
EP14872305.9A EP3082569B1 (en) | 2013-12-19 | 2014-12-05 | Marker-based tool tracking |
ES14872305T ES2726895T3 (en) | 2013-12-19 | 2014-12-05 | Instrument tracking based on a marker |
CA2932895A CA2932895C (en) | 2013-12-19 | 2014-12-05 | Marker-based tool tracking |
JP2016541315A JP6302070B2 (en) | 2013-12-19 | 2014-12-05 | Instrument tracking using markers |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US14/134,237 | 2013-12-19 | ||
US14/134,237 US9597009B2 (en) | 2013-12-19 | 2013-12-19 | Marker-based tool tracking |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2015094726A1 true WO2015094726A1 (en) | 2015-06-25 |
Family
ID=53398779
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/US2014/068899 WO2015094726A1 (en) | 2013-12-19 | 2014-12-05 | Marker-based tool tracking |
Country Status (8)
Country | Link |
---|---|
US (1) | US9597009B2 (en) |
EP (1) | EP3082569B1 (en) |
JP (1) | JP6302070B2 (en) |
CN (1) | CN105828703B (en) |
AU (1) | AU2014366911B2 (en) |
CA (1) | CA2932895C (en) |
ES (1) | ES2726895T3 (en) |
WO (1) | WO2015094726A1 (en) |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US11806092B2 (en) | 2018-04-25 | 2023-11-07 | Carl Zeiss Meditec Ag | Microscopy system and method for operating the microscopy system |
Families Citing this family (28)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US9597009B2 (en) | 2013-12-19 | 2017-03-21 | Novartis Ag | Marker-based tool tracking |
US9999350B2 (en) | 2014-09-25 | 2018-06-19 | Novartis Ag | Reduced glare surgical microscope and associated devices, systems, and methods |
US9645379B2 (en) * | 2014-12-29 | 2017-05-09 | Novartis Ag | Magnification in ophthalmic procedures and associated devices, systems, and methods |
US9639917B2 (en) * | 2015-05-19 | 2017-05-02 | Novartis Ag | OCT image modification |
US20170035287A1 (en) * | 2015-08-04 | 2017-02-09 | Novartis Ag | Dynamic surgical data overlay |
US9826900B2 (en) * | 2015-08-17 | 2017-11-28 | Novartis Ag | Surgical microscope with integrated optical coherence tomography and display systems |
US20170172667A1 (en) * | 2015-12-16 | 2017-06-22 | Novartis Ag | Cannula with optical sensing |
US11484363B2 (en) | 2015-12-28 | 2022-11-01 | Elbit Systems Ltd. | System and method for determining the position and orientation of a tool tip relative to eye tissue of interest |
IL243384A (en) * | 2015-12-28 | 2017-05-29 | Schneider Ron | System and method for determining the position and orientation of a tool tip relative to eye tissue of interest |
WO2017169823A1 (en) * | 2016-03-30 | 2017-10-05 | ソニー株式会社 | Image processing device and method, surgery system, and surgical member |
US11071449B2 (en) * | 2016-03-31 | 2021-07-27 | Alcon Inc. | Visualization system for ophthalmic surgery |
IL245560B1 (en) | 2016-05-09 | 2024-01-01 | Elbit Systems Ltd | Localized optical coherence tomography images for ophthalmological surgical procedures |
US10460457B2 (en) * | 2016-07-12 | 2019-10-29 | Novartis Ag | Adaptive adjustment of overlay image parameters |
US10409051B2 (en) | 2016-08-02 | 2019-09-10 | Novartis Ag | Extraction of microscope zoom level using object tracking |
EP3285107B2 (en) * | 2016-08-16 | 2024-02-28 | Leica Instruments (Singapore) Pte. Ltd. | Surgical microscope with gesture control and method for a gesture control of a surgical microscope |
US10973585B2 (en) | 2016-09-21 | 2021-04-13 | Alcon Inc. | Systems and methods for tracking the orientation of surgical tools |
MX2019007116A (en) | 2016-12-15 | 2019-09-16 | Novartis Ag | Adaptive image registration for ophthalmic surgery. |
US10918445B2 (en) * | 2016-12-19 | 2021-02-16 | Ethicon Llc | Surgical system with augmented reality display |
WO2018193932A1 (en) * | 2017-04-21 | 2018-10-25 | ソニー株式会社 | Information processing device, surgical tool, information processing method, and program |
JP2018175790A (en) * | 2017-04-21 | 2018-11-15 | ソニー株式会社 | Information processing device, information processing method and program |
JP2020535890A (en) * | 2017-10-06 | 2020-12-10 | アルコン インコーポレイティド | Tracking eye movements within tracking range |
US11517474B2 (en) | 2017-12-19 | 2022-12-06 | Alcon Inc. | Methods and systems for eye illumination |
US11259960B2 (en) | 2018-02-22 | 2022-03-01 | Alcon Inc. | Surgical instrument using detected light |
EP3696593A1 (en) * | 2019-02-12 | 2020-08-19 | Leica Instruments (Singapore) Pte. Ltd. | A controller for a microscope, a corresponding method and a microscope system |
KR102390116B1 (en) * | 2019-11-19 | 2022-04-25 | 주식회사 마이크로트 | Implant device for lowering intraocular pressure with easy and safe method |
JP7421787B2 (en) | 2019-12-11 | 2024-01-25 | リバーフィールド株式会社 | Vitreous surgery lighting device |
EP3881793A1 (en) * | 2020-03-17 | 2021-09-22 | CHU de NICE | Surgical instrument and computer-implemented method for determining the position and orientation of such surgical instrument |
DE102021202384B3 (en) | 2021-03-11 | 2022-07-14 | Carl Zeiss Meditec Ag | Microscope system, medical instrument and calibration method |
Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20030169603A1 (en) * | 2002-03-05 | 2003-09-11 | Luloh K. Peter | Apparatus and method for illuminating a field of view within an eye |
WO2005107845A1 (en) | 2004-04-29 | 2005-11-17 | Iscience Interventional Corporation | Apparatus and method for ocular treatment |
US20100208202A1 (en) * | 2009-02-16 | 2010-08-19 | Canon Kabushiki Kaisha | Fundus camera |
US20110282331A1 (en) * | 2010-05-13 | 2011-11-17 | Oprobe, Llc | Optical coherence tomography with multiple imaging instruments |
US20130038836A1 (en) * | 2011-08-12 | 2013-02-14 | Ronald T. Smith | Portable pattern-generating ophthalmic probe |
Family Cites Families (18)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JPH09149876A (en) * | 1995-11-30 | 1997-06-10 | Olympus Optical Co Ltd | Endoscope device |
JPH10118076A (en) * | 1996-10-24 | 1998-05-12 | Olympus Optical Co Ltd | Device for surgical operation under endoscope |
US7867186B2 (en) * | 2002-04-08 | 2011-01-11 | Glaukos Corporation | Devices and methods for treatment of ocular disorders |
US8403828B2 (en) * | 2003-07-21 | 2013-03-26 | Vanderbilt University | Ophthalmic orbital surgery apparatus and method and image-guide navigation system |
JP4095044B2 (en) * | 2004-04-19 | 2008-06-04 | 国弘 武蔵 | Endoscopic endoscope |
DE102004049258B4 (en) | 2004-10-04 | 2007-04-26 | Universität Tübingen | Device, method for controlling operation-supporting medical information systems and digital storage medium |
US9867669B2 (en) * | 2008-12-31 | 2018-01-16 | Intuitive Surgical Operations, Inc. | Configuration marker design and detection for instrument tracking |
US8506558B2 (en) * | 2008-01-11 | 2013-08-13 | Oraya Therapeutics, Inc. | System and method for performing an ocular irradiation procedure |
JP5477800B2 (en) * | 2008-02-27 | 2014-04-23 | 株式会社日立製作所 | Method of operating rotation state detection device and rotation state detection device |
ATE509568T1 (en) * | 2008-10-22 | 2011-06-15 | Sensomotoric Instr Ges Fuer Innovative Sensorik Mbh | METHOD AND DEVICE FOR IMAGE PROCESSING FOR COMPUTER-ASSISTED EYE OPERATIONS |
US8903476B2 (en) * | 2009-03-08 | 2014-12-02 | Oprobe, Llc | Multi-function optical probe system for medical and veterinary applications |
US8996173B2 (en) | 2010-09-21 | 2015-03-31 | Intuitive Surgical Operations, Inc. | Method and apparatus for hand gesture control in a minimally invasive surgical system |
US9542001B2 (en) | 2010-01-14 | 2017-01-10 | Brainlab Ag | Controlling a surgical navigation system |
US8414564B2 (en) * | 2010-02-18 | 2013-04-09 | Alcon Lensx, Inc. | Optical coherence tomographic system for ophthalmic surgery |
US10085633B2 (en) * | 2012-04-19 | 2018-10-02 | Novartis Ag | Direct visualization system for glaucoma treatment |
US9492065B2 (en) | 2012-06-27 | 2016-11-15 | Camplex, Inc. | Surgical retractor with video cameras |
EP2986419B1 (en) | 2013-04-16 | 2017-06-07 | Atlas Copco Industrial Technique AB | Power tool |
US9597009B2 (en) | 2013-12-19 | 2017-03-21 | Novartis Ag | Marker-based tool tracking |
-
2013
- 2013-12-19 US US14/134,237 patent/US9597009B2/en active Active
-
2014
- 2014-12-05 ES ES14872305T patent/ES2726895T3/en active Active
- 2014-12-05 CA CA2932895A patent/CA2932895C/en not_active Expired - Fee Related
- 2014-12-05 CN CN201480069020.2A patent/CN105828703B/en not_active Expired - Fee Related
- 2014-12-05 EP EP14872305.9A patent/EP3082569B1/en active Active
- 2014-12-05 AU AU2014366911A patent/AU2014366911B2/en not_active Ceased
- 2014-12-05 JP JP2016541315A patent/JP6302070B2/en active Active
- 2014-12-05 WO PCT/US2014/068899 patent/WO2015094726A1/en active Application Filing
Patent Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20030169603A1 (en) * | 2002-03-05 | 2003-09-11 | Luloh K. Peter | Apparatus and method for illuminating a field of view within an eye |
WO2005107845A1 (en) | 2004-04-29 | 2005-11-17 | Iscience Interventional Corporation | Apparatus and method for ocular treatment |
US20100208202A1 (en) * | 2009-02-16 | 2010-08-19 | Canon Kabushiki Kaisha | Fundus camera |
US20110282331A1 (en) * | 2010-05-13 | 2011-11-17 | Oprobe, Llc | Optical coherence tomography with multiple imaging instruments |
US20130038836A1 (en) * | 2011-08-12 | 2013-02-14 | Ronald T. Smith | Portable pattern-generating ophthalmic probe |
Non-Patent Citations (1)
Title |
---|
See also references of EP3082569A4 |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US11806092B2 (en) | 2018-04-25 | 2023-11-07 | Carl Zeiss Meditec Ag | Microscopy system and method for operating the microscopy system |
Also Published As
Publication number | Publication date |
---|---|
EP3082569A4 (en) | 2017-11-01 |
CN105828703B (en) | 2019-01-01 |
EP3082569B1 (en) | 2019-02-27 |
CN105828703A (en) | 2016-08-03 |
JP2017501802A (en) | 2017-01-19 |
US9597009B2 (en) | 2017-03-21 |
US20150173644A1 (en) | 2015-06-25 |
CA2932895C (en) | 2019-03-19 |
EP3082569A1 (en) | 2016-10-26 |
CA2932895A1 (en) | 2015-06-25 |
AU2014366911B2 (en) | 2017-03-30 |
JP6302070B2 (en) | 2018-03-28 |
ES2726895T3 (en) | 2019-10-10 |
AU2014366911A1 (en) | 2016-07-07 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CA2932895C (en) | Marker-based tool tracking | |
US11071449B2 (en) | Visualization system for ophthalmic surgery | |
US10973585B2 (en) | Systems and methods for tracking the orientation of surgical tools | |
US9788906B2 (en) | Context aware surgical systems for intraoperatively configuring imaging devices | |
US20180125333A1 (en) | Efficient and interactive bleeding detection in a surgical system | |
Richa et al. | Visual tracking of surgical tools for proximity detection in retinal surgery | |
Richa et al. | Vision-based proximity detection in retinal surgery | |
EP2931161A1 (en) | Markerless tracking of robotic surgical tools | |
US20170035287A1 (en) | Dynamic surgical data overlay | |
EP3554383B1 (en) | System for providing images for guiding surgery | |
Speidel et al. | Automatic classification of minimally invasive instruments based on endoscopic image sequences | |
JP2011508618A (en) | Method for detecting and / or tracking the location of characteristic eye components | |
CN111491549B (en) | Method and system for eye illumination | |
WO2022206436A1 (en) | Dynamic position identification and prompt system and method | |
WO2018109227A1 (en) | System providing images guiding surgery | |
EP3881793A1 (en) | Surgical instrument and computer-implemented method for determining the position and orientation of such surgical instrument | |
JP2019058494A (en) | Laser treatment device |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 14872305 Country of ref document: EP Kind code of ref document: A1 |
|
ENP | Entry into the national phase |
Ref document number: 2932895 Country of ref document: CA |
|
ENP | Entry into the national phase |
Ref document number: 2016541315 Country of ref document: JP Kind code of ref document: A |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
ENP | Entry into the national phase |
Ref document number: 2014366911 Country of ref document: AU Date of ref document: 20141205 Kind code of ref document: A |
|
REEP | Request for entry into the european phase |
Ref document number: 2014872305 Country of ref document: EP |
|
WWE | Wipo information: entry into national phase |
Ref document number: 2014872305 Country of ref document: EP |