WO2016179582A1 - Instrument tracking in oct-assisted surgery - Google Patents

Instrument tracking in oct-assisted surgery Download PDF

Info

Publication number
WO2016179582A1
WO2016179582A1 PCT/US2016/031452 US2016031452W WO2016179582A1 WO 2016179582 A1 WO2016179582 A1 WO 2016179582A1 US 2016031452 W US2016031452 W US 2016031452W WO 2016179582 A1 WO2016179582 A1 WO 2016179582A1
Authority
WO
WIPO (PCT)
Prior art keywords
markers
surgical instrument
oct
dimensional
camera assemblies
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/US2016/031452
Other languages
French (fr)
Inventor
Mohamed T. EL-HADDAD
Yuankai K. TAO
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Cleveland Clinic Foundation
Original Assignee
Cleveland Clinic Foundation
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Cleveland Clinic Foundation filed Critical Cleveland Clinic Foundation
Publication of WO2016179582A1 publication Critical patent/WO2016179582A1/en
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Classifications

    • AHUMAN NECESSITIES
    • A61MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61BDIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B90/00Instruments, implements or accessories specially adapted for surgery or diagnosis and not covered by any of the groups A61B1/00 - A61B50/00, e.g. for luxation treatment or for protecting wound edges
    • A61B90/36Image-producing devices or illumination devices not otherwise provided for
    • A61B90/37Surgical systems with images on a monitor during operation
    • AHUMAN NECESSITIES
    • A61MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61BDIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B90/00Instruments, implements or accessories specially adapted for surgery or diagnosis and not covered by any of the groups A61B1/00 - A61B50/00, e.g. for luxation treatment or for protecting wound edges
    • A61B90/20Surgical microscopes characterised by non-optical aspects
    • AHUMAN NECESSITIES
    • A61MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61BDIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B3/00Apparatus for testing the eyes; Instruments for examining the eyes
    • A61B3/0016Operational features thereof
    • A61B3/0025Operational features thereof characterised by electronic signal processing, e.g. eye models
    • AHUMAN NECESSITIES
    • A61MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61BDIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B3/00Apparatus for testing the eyes; Instruments for examining the eyes
    • A61B3/10Objective types, i.e. instruments for examining the eyes independent of the patients' perceptions or reactions
    • AHUMAN NECESSITIES
    • A61MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61BDIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B3/00Apparatus for testing the eyes; Instruments for examining the eyes
    • A61B3/10Objective types, i.e. instruments for examining the eyes independent of the patients' perceptions or reactions
    • A61B3/102Objective types, i.e. instruments for examining the eyes independent of the patients' perceptions or reactions for optical coherence tomography [OCT]
    • AHUMAN NECESSITIES
    • A61MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61BDIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B3/00Apparatus for testing the eyes; Instruments for examining the eyes
    • A61B3/10Objective types, i.e. instruments for examining the eyes independent of the patients' perceptions or reactions
    • A61B3/13Ophthalmic microscopes
    • AHUMAN NECESSITIES
    • A61MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61BDIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B34/00Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
    • A61B34/20Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
    • GPHYSICS
    • G02OPTICS
    • G02BOPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B21/00Microscopes
    • G02B21/0004Microscopes specially adapted for specific applications
    • G02B21/0012Surgical microscopes
    • GPHYSICS
    • G02OPTICS
    • G02BOPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B26/00Optical devices or arrangements for the control of light using movable or deformable optical elements
    • G02B26/08Optical devices or arrangements for the control of light using movable or deformable optical elements for controlling the direction of light
    • G02B26/10Scanning systems
    • G02B26/105Scanning systems with one or more pivoting mirrors or galvano-mirrors
    • GPHYSICS
    • G06COMPUTING OR CALCULATING; COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00Image analysis
    • G06T7/0002Inspection of images, e.g. flaw detection
    • G06T7/0012Biomedical image inspection
    • GPHYSICS
    • G06COMPUTING OR CALCULATING; COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00Image analysis
    • G06T7/70Determining position or orientation of objects or cameras
    • G06T7/73Determining position or orientation of objects or cameras using feature-based methods
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/20Image signal generators
    • H04N13/204Image signal generators using stereoscopic image cameras
    • H04N13/239Image signal generators using stereoscopic image cameras using two two-dimensional [2D] image sensors having a relative position equal to or related to the interocular distance
    • AHUMAN NECESSITIES
    • A61MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61BDIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B34/00Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
    • A61B34/20Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
    • A61B2034/2046Tracking techniques
    • A61B2034/2055Optical tracking systems
    • AHUMAN NECESSITIES
    • A61MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61BDIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B90/00Instruments, implements or accessories specially adapted for surgery or diagnosis and not covered by any of the groups A61B1/00 - A61B50/00, e.g. for luxation treatment or for protecting wound edges
    • A61B90/36Image-producing devices or illumination devices not otherwise provided for
    • A61B90/37Surgical systems with images on a monitor during operation
    • A61B2090/373Surgical systems with images on a monitor during operation using light, e.g. by using optical scanners
    • A61B2090/3735Optical coherence tomography [OCT]
    • GPHYSICS
    • G01MEASURING; TESTING
    • G01BMEASURING LENGTH, THICKNESS OR SIMILAR LINEAR DIMENSIONS; MEASURING ANGLES; MEASURING AREAS; MEASURING IRREGULARITIES OF SURFACES OR CONTOURS
    • G01B11/00Measuring arrangements characterised by the use of optical techniques
    • G01B11/002Measuring arrangements characterised by the use of optical techniques for measuring two or more coordinates
    • GPHYSICS
    • G01MEASURING; TESTING
    • G01BMEASURING LENGTH, THICKNESS OR SIMILAR LINEAR DIMENSIONS; MEASURING ANGLES; MEASURING AREAS; MEASURING IRREGULARITIES OF SURFACES OR CONTOURS
    • G01B9/00Measuring instruments characterised by the use of optical techniques
    • G01B9/02Interferometers
    • G01B9/0209Low-coherence interferometers
    • G01B9/02091Tomographic interferometers, e.g. based on optical coherence
    • GPHYSICS
    • G06COMPUTING OR CALCULATING; COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00Indexing scheme for image analysis or image enhancement
    • G06T2207/10Image acquisition modality
    • G06T2207/10028Range image; Depth image; 3D point clouds
    • GPHYSICS
    • G06COMPUTING OR CALCULATING; COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00Indexing scheme for image analysis or image enhancement
    • G06T2207/10Image acquisition modality
    • G06T2207/10072Tomographic images
    • G06T2207/10101Optical tomography; Optical coherence tomography [OCT]
    • GPHYSICS
    • G06COMPUTING OR CALCULATING; COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00Indexing scheme for image analysis or image enhancement
    • G06T2207/30Subject of image; Context of image processing
    • G06T2207/30204Marker

Definitions

  • This invention relates to medical systems, and more particularly, to instrument tracking in OCT-assisted surgery.
  • OCT Retinal Optical Coherence Tomography
  • perioperative imaging requires interruption of surgery and precludes real-time surgical guidance. This limitation was overcome by the recent development of microscope-integrated intraoperative OCT systems (iOCT), which allows live cross- sectional imaging concurrent with surgery.
  • iOCT microscope-integrated intraoperative OCT systems
  • a major limitation to the clinical utility of intraoperative imaging is real-time visualization of instrument-tissue interactions.
  • visualization of surgical maneuvers may be performed by spatial compounding under the guidance of a Heads ⁇ Up Display (HUD) unit. Volumetric scans around the instrument tip are sparsely sampled and spatially compounded across the
  • a microscope- integrated intraoperative optical coherence tomography (OCT) scanner system having automated tracking of an instrument tip is provided.
  • At least one scanning mirror is configured such that a field of view of the OCT scanner system is
  • a drive system is configured to control the at least one of the position and the orientation of the at least one scanning mirror.
  • a plurality of camera assemblies are configured to determine respective two-dimensional projections of the positions of a plurality of markers attached to a surgical instrument.
  • a stereo vision system is configured to determine a three-dimensional location of each of the plurality of markers from the determined two-dimensional positions at the plurality of camera assemblies.
  • An instrument tracking component is configured to determine a position of a working tip of the surgical instrument according to the determined three- dimensional locations of the plurality of markers.
  • a drive control is configured to instruct the drive system to adjust the at least one of the position and the orientation of the at least one scanner mirror such that the field of view of the OCT scanner system is determined from a position of the working tip of the surgical instrument.
  • a method for adjusting a field of view of microscope-integrated intraoperative optical coherence tomography (OCT) scanner system to track the tip of a surgical instrument.
  • OCT optical coherence tomography
  • Respective two-dimensional projections of the positions of a plurality of markers attached to a surgical instrument is determined.
  • a three-dimensional location of each of the plurality of markers is determined from the determined two- dimensional positions at the plurality of camera assemblies and a known relationship among the plurality of camera assemblies.
  • a position of a working tip of the surgical instrument is determined according to the determined three-dimensional locations of the plurality of markers.
  • At least one of the position and the orientation of at least one scanner mirror associated with the OCT scanner system are controlled such that the field of view of the OCT scanner system is dependent on the position of the working tip of the surgical instrument.
  • OCT optical coherence tomography
  • a plurality of images of the model are captured at each of the plurality of camera assemblies, such that each camera generates a plural set of two- dimensional locations for each marker.
  • a relative pose between the plurality of camera assemblies is determined from the plural set of two-dimensional locations for each marker at each of the plurality of cameras and the known arrangement of the plurality of markers.
  • Respective two-dimensional projections of the positions of a plurality of markers attached to a surgical instrument are determined.
  • a three- dimensional location of each of the plurality of markers is determined from the determined two-dimensional positions at the plurality of camera assemblies and the relative pose.
  • a position of a working tip of the surgical instrument is determined according to the determined three-dimensional locations of the plurality of markers.
  • At least one of the position and the orientation of at least one scanner mirror associated with the OCT scanner system is controlled such that the field of view of the OCT scanner system is determined by the position of the working tip of the surgical instrument.
  • FIG. 1 illustrates a microscope-integrated intraoperative optical coherence tomography scanner system having automated tracking of an instrument tip during a surgical procedure
  • FIG. 2 illustrates one example of a surgical instrument having a working tip and a plurality of markers and illustrated in two opposing views;
  • FIG. 3 illustrates one example of a method for adjusting a field of view of microscope-integrated intraoperative optical coherence tomography (OCT) scanner system to track the tip of a surgical instrument
  • FIG. 4 illustrates a method for calibrating a stereo vision system for instrument tracking in a microscope-integrated intraoperative optical coherence tomography (OCT) scanner system
  • FIG. 5 is a schematic block diagram illustrating an exemplary system of hardware components capable of implementing examples of the systems and methods disclosed in FIGS. 1 -4.
  • systems and methods are provided for tracking an instrument tip and adjusting a field of view of an optical coherence tomography (OCT) system to follow the instrument tip.
  • OCT optical coherence tomography
  • a stereo-vision system comprising two or more cameras tracking a set of active or passive markers, are used to determine multiple positions on or near a free tip of the instrument and, from this information, determine a position of the working tip of the instrument.
  • the position and/or orientations of a set of scanning mirrors associated with the OCT scanner can be altered to maintain the working tip of the instrument within the field of view of the OCT scanner.
  • FIG. 1 illustrates a microscope-integrated intraoperative optical coherence tomography (OCT) scanner system 10 having automated tracking of an instrument tip during a surgical procedure.
  • the scanner system 10 includes a surgical microscope 12, and an OCT scanning unit 20 configured to image a target location in conjunction with at least one optical component associated with the surgical microscope 12, such as an objective lens.
  • the OCT scanning unit 20 includes at least one scanning mirror 22 configured such that a field of view of the OCT scanner system is determined by at least one of a position and an orientation of the at least one scanning mirror.
  • a drive system 24 configured to control the at least one of the position and the orientation of the at least one scanning mirror.
  • Current generation tabletop and microscope- integrated OCT systems utilize actuating mirrors for two-dimensional lateral scanning of their imaging beam. In combination with the axial sectioning capabilities of OCT, this yields three-dimensional tomograms of the tissues of interest.
  • These actuating mirrors are generally paired galvanometer scanners, but may consist of resonant scanners and MEMS scanners or a combination thereof.
  • conventional OCT systems position paired scanners in close proximity, but do not optically image the angular scan fields between the mirror faces to the back aperture of their imaging objective or pupil plane.
  • a plurality of camera assemblies 32 and 33 configured to determine respective two-dimensional projections of the positions of a plurality of markers attached to a surgical instrument.
  • the camera assemblies 32 and 33 can include appropriate filters to limit the response of the camera assemblies to a band of wavelengths within one of the infrared, ultraviolet, or visible range, depending on the wavelength of emission or reflection associated with the plurality of markers. It will be appreciated that, in the illustrated implementation 1 0, two camera assemblies 32 and 33 are shown, but in practice, additional cameras may be employed to improve the accuracy of the location of the working tip of the instrument. It will be
  • the markers can be active, that is, light emitting, or passive, that is, light reflecting. Where passive markers are used, the system 10 can include an appropriate light source (not shown).
  • Depth resolution in stereo vision, Az is directly proportional to the square of the imaging distance, z, and inversely proportional to both the focal length of and separation distance between the cameras, conventionally defined as the baseline.
  • the baseline can be maximized in order to
  • the upper limit of the baseline was set to one hundred millimeters as not exceed the body width of the intraoperative OCT system.
  • the cameras were pointed inwards in a converging stereo setup with a tilt that maximized the stereo vision FOY overlap at a desired imaging distance of approximately nineteen centimeters. This imaging distance was set by the axial distance between the bottom of the intraoperative OCT chassis and ophthalmic surgical microscope focal plane.
  • the outputs of the plurality of camera assemblies are provide to a stereo vision system 38 configured to determine a three-dimensional location of each of the plurality of markers from the determined two-dimensional positions at the plurality of camera assemblies.
  • the stereo vision system 38 triangulates for the 3-D position of a point in space based on the disparity in its projection in the image planes of the plurality of camera assemblies. Assuming proper calibration, the three-dimensional position of the markers relative to one camera can be determined and then transformed, if needed, to a global coordinate system
  • An instrument tracking component 42 is configured to determine a position of a working tip of the surgical instrument according to the determined three- dimensional locations of the plurality of markers.
  • two or more markers are placed along an axis of the surgical instrument, such that an orientation and position of the instrument can be determined, and a position of the working tip can be determined from the determined orientation and position of the instrument and known properties of the instrument.
  • two or more markers are placed along an axis of the surgical instrument and another marker is placed on a opposite side of the instrument from the two or more markers to allow for the determination to better account for the deviation of the markers from the centerline of the instrument.
  • a drive control 44 is configured to instruct the drive system to adjust the at least one of the position and the orientation of the at least one scanner mirror 22 such that the field of view of the OCT scanner system 20 is determined by the position of the working tip of the surgical instrument.
  • the position of the working tip of the surgical instrument can be transformed from a coordinate system associated with the stereo vision system to a coordinate system associated with the OCT scanner system 20 via an appropriate coordinate transform.
  • the drive system 24 is instructed to adjust the at least one scanner mirror 22 such that the OCT field of view remains centered on the working tip of the instrument. In another implementation, the drive system 24 is instructed to adjust the at least one scanner mirror 22 such that the OCT field of view remains at a predetermined displacement from the position of the working tip. Accordingly, the surgical procedure can be performed without the need to continuously reposition the field of view of the OCT scanner.
  • FIG. 2 illustrates one example of a surgical instrument 50 having a working tip 52 and a plurality of markers 54, 55, and 58 and illustrated in two opposing views.
  • each of the plurality of markers 54, 55, and 58 include light emitting diodes (LED) and configured to emit light at a wavelength of around nine hundred forty nanometers with a one hundred and sixty degree beam angle.
  • Two axial markers 54 and 55 also include collars for affixing the LEDs to the instrument and an off-axis marker 58 is attached to a cap placed on a free end of the instrument.
  • the two axial markers 54 and 55 were radially offset from the axis of the instrument, and the off-axis marker 58 is used to compensate for the thickness of the instrument at each axial marker position.
  • FIGS. 3 and 4 example methodologies will be better appreciated with reference to FIGS. 3 and 4. While, for purposes of simplicity of explanation, the methodologies of FIGS. 3 and 4 are shown and described as executing serially, it is to be understood and appreciated that the present invention is not limited by the illustrated order, as some actions could in other examples occur in different orders and/or concurrently from that shown and described herein.
  • FIG. 3 illustrates one example of a method 100 for adjusting a field of view of microscope-integrated intraoperative optical coherence tomography (OCT) scanner system to track the tip of a surgical instrument.
  • OCT optical coherence tomography
  • respective two- dimensional projections of the positions of a plurality of markers attached to a surgical instrument are determined at each of a plurality of camera assemblies.
  • the markers can emit or reflect light of a particular wavelength
  • the cameras can be fitted with appropriate filters to detect light only around that particular wavelength, with the location of each marker within the camera coordinate system representing the two-dimensional projection of the marker position.
  • a three-dimensional location of each of the plurality of markers is determined from the determined two-dimensional positions at the plurality of camera assemblies. Using a known relationship among the cameras, reflected as calibration parameters, such as a relative pose between two cameras, the
  • determined two-dimensional positions can be used to determine a point in three- dimensional space within a coordinate domain associated with one of the cameras.
  • two cameras giving three dimensional locations represented as vectors mi eft and m ri g h t, the three dimensional position ⁇ X, Y, Z) can be determined in a coordinate frame of the right camera from the calibration parameters, such that:
  • S x is a scaling parameter based on pixel length for the right camera in a ⁇ direction
  • S y is a scaling parameter based on pixel length for the right camera in a / direction
  • is a skew angle of a given pixel in the right camera
  • ⁇ xo, yo represent the pixel coordinates of the right camera origin
  • R is a rotation matrix relating the coordinate systems of the two cameras
  • t is a translation vector relating the coordinate systems of the two cameras.
  • a position of a working tip of the surgical instrument is determined according to the determined three-dimensional locations of the plurality of markers.
  • a directional vector is defined by two markers positioned to be parallel to a longitudinal instrument axis. The tip was then defined as a point along a vector parallel to the directional vector that intersected the location of a marker on a free tip of the instrument, for example, from a known length of the instrument. The two body markers were radially offset from the axis of the instrument and a marker at the free-tip of the instrument is used to compensate for the thickness of the instrument at each axial marker position.
  • Instrument orientation was defined as the angle between the directional vector and the y-axis of the OCT coordinate system.
  • a triangulation position error of each active marker and a desired orientation resolution determines a separation distance between the body markers. For example, for a desired orientation resolution of 0.9 ° and a triangulation position error of 150 ⁇ , the axial markers should be separated by 19.1 mm. In practice, a value 2.5 times that can be used to account for addition error sources, such as variability in the placement of the markers and ensure significant orientation resolution.
  • At 108 controlling at least one of the position and the orientation of at least one scanner mirror associated with the OCT scanner system such that the field of view of the OCT scanner system is determined by the position of the working tip of the surgical instrument.
  • the field of view can be centered on the working tip or slightly displaced from the working tip.
  • An orientation of the field of view can also be controlled according to the instrument position, such that sequential B-scans are aligned parallel and perpendicular to the instrument axis.
  • Each determined position is transformed into the OCT coordinate system, and the scanner mirror or mirrors are controlled to maintain the position and orientation of the OCT scanner system field of view.
  • FIG. 4 illustrates a method 150 for calibrating a stereo vision system for instrument tracking in a microscope-integrated intraoperative optical coherence tomography (OCT) scanner system.
  • OCT optical coherence tomography
  • a model comprising a plurality of markers in a known arrangement, is moved to a plurality of locations within a field of view of a plurality of camera assemblies.
  • the model comprises four markers forming four corners of a square having a side length of 17.5 millimeters.
  • a plurality of images of the model are captured at each of the plurality of camera assemblies, such that each camera generates a plural set of two-dimensional locations for each marker.
  • [Rt] is a 3x4 matrix that describes the rotation and translation between the cameras.
  • [R t] will become an identity matrix.
  • the intrinsic parameters describe the properties of the image sensor that depend mainly on the shape of its pixels, and the pixel coordinates that represent the origin, which is not necessarily at the center of the sensor.
  • the intrinsic parameters matrix can be represented as:
  • S x is a scaling parameter based on pixel length in a x direction
  • S y is a scaling parameter based on pixel length in a / direction
  • is a skew angle of a given pixel in the camera
  • (x 0 , yo) represent the pixel coordinates of the camera origin.
  • a second calibration step is performed to determine a transform from a coordinate domain associated with the reference camera to a coordinate system associated with the OCT scanner system.
  • a model including at least one marker, is moved within the field of view of the OCT scanner system and the plurality of cameras at 1 58. It will be appreciated that this can be the same model as that used in 1 52, or a different model. In one implementation, this can be done with a surgical instrument appropriately affixed with markers, such as the instrument illustrated in FIG. 2. In one implementation, the model is moved to four known positions in the OCT coordinate system, with two lying on an x-axis of the OCT coordinate system and two lying on a y-axis of the coordinate system. At 1 60, a plurality of images of the model are captured at each of the plurality of camera assemblies.
  • the transform from the coordinate domain associated with the reference camera to the coordinate system associated with the OCT scanner system is determined from the plurality of images of the model captured at 160.
  • the coordinate transformation is implemented as a three-dimensional translation and rotation.
  • the three-dimensional position of the marker or markers can be determined from the captured images and arranged as a 3xM matrix, X s tereo, in which each of M determined positions are stored as ordered triplets representing coordinates in the camera reference plane.
  • X mo tor is a 3xM matrix representing the position of the model within the OCT coordinate frame. From this data, a 3x3 rotation matrix, R, is determined such that:
  • FIG. 5 is a schematic block diagram illustrating an exemplary system 200 of hardware components capable of implementing examples of the systems and methods disclosed in FIGS. 1 -4, such as the stereo vision system 38, the instrument tracking component 42, and the drive control illustrated in FIG. 1 .
  • the system 200 can include various systems and subsystems.
  • the system 200 can be a personal computer, a laptop computer, a workstation, a computer system, an appliance, an application-specific integrated circuit (ASIC), a server, a server blade center, a server farm, etc.
  • ASIC application-specific integrated circuit
  • the system 200 can includes a system bus 202, a processing unit 204, a system memory 206, memory devices 208 and 210, a communication interface 212 (e.g., a network interface), a communication link 214, a display 216 (e.g., a video screen), and an input device 218 (e.g., a keyboard and/or a mouse).
  • the system bus 202 can be in communication with the processing unit 204 and the system memory 206.
  • the additional memory devices 208 and 210 such as a hard disk drive, server, stand alone database, or other non-volatile memory, can also be in communication with the system bus 202.
  • the system bus 202 interconnects the processing unit 204, the memory devices 206-210, the communication interface 212, the display 216, and the input device 218. In some examples, the system bus 202 also interconnects an additional port (not shown), such as a universal serial bus (USB) port.
  • USB universal serial bus
  • the processing unit 204 can be a computing device and can include an application-specific integrated circuit (ASIC).
  • the processing unit 204 executes a set of instructions to implement the operations of examples disclosed herein.
  • the processing unit can include a processing core.
  • the additional memory devices 206, 208 and 210 can store data, programs, instructions, database queries in text or compiled form, and any other information that can be needed to operate a computer.
  • the memories 206, 208 and 210 can be implemented as computer-readable media (integrated or removable) such as a memory card, disk drive, compact disk (CD), or server accessible over a network.
  • the memories 206, 208 and 210 can comprise text, images, video, and/or audio, portions of which can be available in formats
  • system 200 can access an external data source or query source through the communication interface 212, which can communicate with the system bus 202 and the communication link 214.
  • system 200 can be used to implement one or more parts of a therapeutic delivery system in accordance with the present invention.
  • Computer executable logic for implementing the diagnostic system resides on one or more of the system memory 206, and the memory devices 208, 210 in accordance with certain examples.
  • the processing unit 204 executes one or more computer executable instructions originating from the system memory 206 and the memory devices 208 and 210.
  • the term "computer readable medium” as used herein refers to a medium that participates in providing instructions to the processing unit 204 for execution, and can, in practice, refer to multiple, operatively connected apparatuses for storing machine executable instructions.

Landscapes

  • Health & Medical Sciences (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Engineering & Computer Science (AREA)
  • Surgery (AREA)
  • Physics & Mathematics (AREA)
  • General Health & Medical Sciences (AREA)
  • Medical Informatics (AREA)
  • Veterinary Medicine (AREA)
  • Biomedical Technology (AREA)
  • Heart & Thoracic Surgery (AREA)
  • Molecular Biology (AREA)
  • Animal Behavior & Ethology (AREA)
  • Public Health (AREA)
  • Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
  • General Physics & Mathematics (AREA)
  • Biophysics (AREA)
  • Ophthalmology & Optometry (AREA)
  • Radiology & Medical Imaging (AREA)
  • Pathology (AREA)
  • Oral & Maxillofacial Surgery (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Theoretical Computer Science (AREA)
  • Signal Processing (AREA)
  • Optics & Photonics (AREA)
  • Gynecology & Obstetrics (AREA)
  • Robotics (AREA)
  • Multimedia (AREA)
  • Chemical & Material Sciences (AREA)
  • Analytical Chemistry (AREA)
  • Quality & Reliability (AREA)
  • Length Measuring Devices By Optical Means (AREA)
  • Microscoopes, Condenser (AREA)

Abstract

Systems and methods are provided for a microscope-integrated intraoperative scanner system having automated tracking of an instrument tip. A scanning mirror is configured such that a field of view of the OCT system is determined by a position or orientation of the scanning mirror. A drive system is configured to control the scanning mirror. Camera assemblies are configured to determine respective two-dimensional projections of the positions of markers attached to a surgical instrument. A stereo vision system is configured to determine a three-dimensional location of each of the markers from the determined two-dimensional positions. An instrument tracking component is configured to determine a position of a working tip of the surgical instrument according to the determined three-dimensional locations. A drive control is configured to instruct the drive system to adjust the scanner mirror to control the field of view of the OCT system.

Description

INSTRUMENT TRACKING IN OCT-ASSISTED SURGERY
RELATED APPLICATIONS
[0001] This application claims priority from U.S. Patent Application Serial No. 62/158,006, filed 7 May 2015, which is incorporated herein in its entirety.
TECHNICAL FIELD
[0002] This invention relates to medical systems, and more particularly, to instrument tracking in OCT-assisted surgery.
BACKGROUND
[0003] Retinal Optical Coherence Tomography (OCT) allows the visualization of high-resolution cross-sectional images of tissue microstructure and is the gold- standard for ophthalmic diagnosis. Recently, intraoperative OCT has been used perioperatively to image pre- and post-operative manipulations to verify the completion of surgical goals and aid clinical decision-making. However,
perioperative imaging requires interruption of surgery and precludes real-time surgical guidance. This limitation was overcome by the recent development of microscope-integrated intraoperative OCT systems (iOCT), which allows live cross- sectional imaging concurrent with surgery. A major limitation to the clinical utility of intraoperative imaging is real-time visualization of instrument-tissue interactions. In iOCT, visualization of surgical maneuvers may be performed by spatial compounding under the guidance of a Heads~Up Display (HUD) unit. Volumetric scans around the instrument tip are sparsely sampled and spatially compounded across the
instrument's cross-section to visualize instrument-tissue interactions. However, spatial compounding inherently trades-off temporal resolution and field-of-view (FOV), which generally limits video-rate visualization to the tip of surgical
instruments. Another limitation of spatial compounding is the OCT FOV needs to be precisely aligned with the surgical instrument and along its projected axis of motion. SUMMARY OF THE INVENTION
[0004] In accordance with an aspect of the present invention, a microscope- integrated intraoperative optical coherence tomography (OCT) scanner system having automated tracking of an instrument tip is provided. At least one scanning mirror is configured such that a field of view of the OCT scanner system is
determined by at least one of a position and an orientation of the at least one scanning mirror. A drive system is configured to control the at least one of the position and the orientation of the at least one scanning mirror. A plurality of camera assemblies are configured to determine respective two-dimensional projections of the positions of a plurality of markers attached to a surgical instrument. A stereo vision system is configured to determine a three-dimensional location of each of the plurality of markers from the determined two-dimensional positions at the plurality of camera assemblies. An instrument tracking component is configured to determine a position of a working tip of the surgical instrument according to the determined three- dimensional locations of the plurality of markers. A drive control is configured to instruct the drive system to adjust the at least one of the position and the orientation of the at least one scanner mirror such that the field of view of the OCT scanner system is determined from a position of the working tip of the surgical instrument.
[0005] In accordance with another aspect of the present invention, a method is provided for adjusting a field of view of microscope-integrated intraoperative optical coherence tomography (OCT) scanner system to track the tip of a surgical instrument. Respective two-dimensional projections of the positions of a plurality of markers attached to a surgical instrument is determined. A three-dimensional location of each of the plurality of markers is determined from the determined two- dimensional positions at the plurality of camera assemblies and a known relationship among the plurality of camera assemblies. A position of a working tip of the surgical instrument is determined according to the determined three-dimensional locations of the plurality of markers. At least one of the position and the orientation of at least one scanner mirror associated with the OCT scanner system are controlled such that the field of view of the OCT scanner system is dependent on the position of the working tip of the surgical instrument. [0006] In accordance with yet another aspect of the present invention, a method is provided for adjusting a field of view of microscope-integrated
intraoperative optical coherence tomography (OCT) scanner system to track the tip of a surgical instrument, the method. A model comprising a plurality of markers in a known arrangement is moved within a field of view of a plurality of camera
assemblies. A plurality of images of the model are captured at each of the plurality of camera assemblies, such that each camera generates a plural set of two- dimensional locations for each marker. A relative pose between the plurality of camera assemblies is determined from the plural set of two-dimensional locations for each marker at each of the plurality of cameras and the known arrangement of the plurality of markers. Respective two-dimensional projections of the positions of a plurality of markers attached to a surgical instrument are determined. A three- dimensional location of each of the plurality of markers is determined from the determined two-dimensional positions at the plurality of camera assemblies and the relative pose. A position of a working tip of the surgical instrument is determined according to the determined three-dimensional locations of the plurality of markers. At least one of the position and the orientation of at least one scanner mirror associated with the OCT scanner system is controlled such that the field of view of the OCT scanner system is determined by the position of the working tip of the surgical instrument.
BRIEF DESCRIPTION OF THE DRAWINGS
[0007] FIG. 1 illustrates a microscope-integrated intraoperative optical coherence tomography scanner system having automated tracking of an instrument tip during a surgical procedure;
[0008] FIG. 2 illustrates one example of a surgical instrument having a working tip and a plurality of markers and illustrated in two opposing views;
[0009] FIG. 3 illustrates one example of a method for adjusting a field of view of microscope-integrated intraoperative optical coherence tomography (OCT) scanner system to track the tip of a surgical instrument; [0010] FIG. 4 illustrates a method for calibrating a stereo vision system for instrument tracking in a microscope-integrated intraoperative optical coherence tomography (OCT) scanner system; and
[0011] FIG. 5 is a schematic block diagram illustrating an exemplary system of hardware components capable of implementing examples of the systems and methods disclosed in FIGS. 1 -4.
DETAILED DESCRIPTION
[0012] In accordance with an aspect of the present invention, systems and methods are provided for tracking an instrument tip and adjusting a field of view of an optical coherence tomography (OCT) system to follow the instrument tip. In one implementation, a stereo-vision system, comprising two or more cameras tracking a set of active or passive markers, are used to determine multiple positions on or near a free tip of the instrument and, from this information, determine a position of the working tip of the instrument. The position and/or orientations of a set of scanning mirrors associated with the OCT scanner can be altered to maintain the working tip of the instrument within the field of view of the OCT scanner.
[0013] To this end, FIG. 1 illustrates a microscope-integrated intraoperative optical coherence tomography (OCT) scanner system 10 having automated tracking of an instrument tip during a surgical procedure. The scanner system 10 includes a surgical microscope 12, and an OCT scanning unit 20 configured to image a target location in conjunction with at least one optical component associated with the surgical microscope 12, such as an objective lens.
[0014] The OCT scanning unit 20 includes at least one scanning mirror 22 configured such that a field of view of the OCT scanner system is determined by at least one of a position and an orientation of the at least one scanning mirror. A drive system 24 configured to control the at least one of the position and the orientation of the at least one scanning mirror. Current generation tabletop and microscope- integrated OCT systems utilize actuating mirrors for two-dimensional lateral scanning of their imaging beam. In combination with the axial sectioning capabilities of OCT, this yields three-dimensional tomograms of the tissues of interest. These actuating mirrors are generally paired galvanometer scanners, but may consist of resonant scanners and MEMS scanners or a combination thereof. To reduce system complexity, conventional OCT systems position paired scanners in close proximity, but do not optically image the angular scan fields between the mirror faces to the back aperture of their imaging objective or pupil plane.
[0015] A plurality of camera assemblies 32 and 33 configured to determine respective two-dimensional projections of the positions of a plurality of markers attached to a surgical instrument. The camera assemblies 32 and 33 can include appropriate filters to limit the response of the camera assemblies to a band of wavelengths within one of the infrared, ultraviolet, or visible range, depending on the wavelength of emission or reflection associated with the plurality of markers. It will be appreciated that, in the illustrated implementation 1 0, two camera assemblies 32 and 33 are shown, but in practice, additional cameras may be employed to improve the accuracy of the location of the working tip of the instrument. It will be
appreciated that the markers can be active, that is, light emitting, or passive, that is, light reflecting. Where passive markers are used, the system 10 can include an appropriate light source (not shown).
[0016] Depth resolution in stereo vision, Az , is directly proportional to the square of the imaging distance, z, and inversely proportional to both the focal length of and separation distance between the cameras, conventionally defined as the baseline. For a fixed imaging distance and focal length, the baseline can be maximized in order to
achieve optimal depth resolution. It will be appreciated, however, that available baseline in an intraoperative imaging system can be limited, and in one
implementation, the upper limit of the baseline was set to one hundred millimeters as not exceed the body width of the intraoperative OCT system. The cameras were pointed inwards in a converging stereo setup with a tilt that maximized the stereo vision FOY overlap at a desired imaging distance of approximately nineteen centimeters. This imaging distance was set by the axial distance between the bottom of the intraoperative OCT chassis and ophthalmic surgical microscope focal plane. [0017] The outputs of the plurality of camera assemblies are provide to a stereo vision system 38 configured to determine a three-dimensional location of each of the plurality of markers from the determined two-dimensional positions at the plurality of camera assemblies. Essentially, the stereo vision system 38 triangulates for the 3-D position of a point in space based on the disparity in its projection in the image planes of the plurality of camera assemblies. Assuming proper calibration, the three-dimensional position of the markers relative to one camera can be determined and then transformed, if needed, to a global coordinate system
associated with the system 10.
[0018] An instrument tracking component 42 is configured to determine a position of a working tip of the surgical instrument according to the determined three- dimensional locations of the plurality of markers. In one implementation, two or more markers are placed along an axis of the surgical instrument, such that an orientation and position of the instrument can be determined, and a position of the working tip can be determined from the determined orientation and position of the instrument and known properties of the instrument. In another implementation, two or more markers are placed along an axis of the surgical instrument and another marker is placed on a opposite side of the instrument from the two or more markers to allow for the determination to better account for the deviation of the markers from the centerline of the instrument.
[0019] A drive control 44 is configured to instruct the drive system to adjust the at least one of the position and the orientation of the at least one scanner mirror 22 such that the field of view of the OCT scanner system 20 is determined by the position of the working tip of the surgical instrument. To this end, the position of the working tip of the surgical instrument can be transformed from a coordinate system associated with the stereo vision system to a coordinate system associated with the OCT scanner system 20 via an appropriate coordinate transform. In one
implementation, the drive system 24 is instructed to adjust the at least one scanner mirror 22 such that the OCT field of view remains centered on the working tip of the instrument. In another implementation, the drive system 24 is instructed to adjust the at least one scanner mirror 22 such that the OCT field of view remains at a predetermined displacement from the position of the working tip. Accordingly, the surgical procedure can be performed without the need to continuously reposition the field of view of the OCT scanner.
[0020] FIG. 2 illustrates one example of a surgical instrument 50 having a working tip 52 and a plurality of markers 54, 55, and 58 and illustrated in two opposing views. In the illustrated implementation, each of the plurality of markers 54, 55, and 58 include light emitting diodes (LED) and configured to emit light at a wavelength of around nine hundred forty nanometers with a one hundred and sixty degree beam angle. Two axial markers 54 and 55 also include collars for affixing the LEDs to the instrument and an off-axis marker 58 is attached to a cap placed on a free end of the instrument. The two axial markers 54 and 55 were radially offset from the axis of the instrument, and the off-axis marker 58 is used to compensate for the thickness of the instrument at each axial marker position.
[0021] In view of the foregoing structural and functional features described above in FIGS. 1 and 2, example methodologies will be better appreciated with reference to FIGS. 3 and 4. While, for purposes of simplicity of explanation, the methodologies of FIGS. 3 and 4 are shown and described as executing serially, it is to be understood and appreciated that the present invention is not limited by the illustrated order, as some actions could in other examples occur in different orders and/or concurrently from that shown and described herein.
[0022] FIG. 3 illustrates one example of a method 100 for adjusting a field of view of microscope-integrated intraoperative optical coherence tomography (OCT) scanner system to track the tip of a surgical instrument. At 102, respective two- dimensional projections of the positions of a plurality of markers attached to a surgical instrument are determined at each of a plurality of camera assemblies. For example, the markers can emit or reflect light of a particular wavelength, and the cameras can be fitted with appropriate filters to detect light only around that particular wavelength, with the location of each marker within the camera coordinate system representing the two-dimensional projection of the marker position.
[0023] At 104, a three-dimensional location of each of the plurality of markers is determined from the determined two-dimensional positions at the plurality of camera assemblies. Using a known relationship among the cameras, reflected as calibration parameters, such as a relative pose between two cameras, the
determined two-dimensional positions can be used to determine a point in three- dimensional space within a coordinate domain associated with one of the cameras. In an example two cameras, giving three dimensional locations represented as vectors mieft and mright, the three dimensional position {X, Y, Z) can be determined in a coordinate frame of the right camera from the calibration parameters, such that:
Figure imgf000009_0001
SyY
+ y0 Eq. right v SXX SyY m 0
X =— +— Eq.
J right right
[0024] where Sx is a scaling parameter based on pixel length for the right camera in a ^ direction, Sy is a scaling parameter based on pixel length for the right camera in a / direction, Θ is a skew angle of a given pixel in the right camera, and {xo, yo) represent the pixel coordinates of the right camera origin, R is a rotation matrix relating the coordinate systems of the two cameras, and t is a translation vector relating the coordinate systems of the two cameras.
[0025] At 1 06, a position of a working tip of the surgical instrument is determined according to the determined three-dimensional locations of the plurality of markers. In one implementation, to calculate the working-tip position, a directional vector is defined by two markers positioned to be parallel to a longitudinal instrument axis. The tip was then defined as a point along a vector parallel to the directional vector that intersected the location of a marker on a free tip of the instrument, for example, from a known length of the instrument. The two body markers were radially offset from the axis of the instrument and a marker at the free-tip of the instrument is used to compensate for the thickness of the instrument at each axial marker position. Instrument orientation was defined as the angle between the directional vector and the y-axis of the OCT coordinate system. A triangulation position error of each active marker and a desired orientation resolution determines a separation distance between the body markers. For example, for a desired orientation resolution of 0.9 ° and a triangulation position error of 150 μη , the axial markers should be separated by 19.1 mm. In practice, a value 2.5 times that can be used to account for addition error sources, such as variability in the placement of the markers and ensure significant orientation resolution.
[0026] At 108, controlling at least one of the position and the orientation of at least one scanner mirror associated with the OCT scanner system such that the field of view of the OCT scanner system is determined by the position of the working tip of the surgical instrument. For example, the field of view can be centered on the working tip or slightly displaced from the working tip. An orientation of the field of view can also be controlled according to the instrument position, such that sequential B-scans are aligned parallel and perpendicular to the instrument axis. Each determined position is transformed into the OCT coordinate system, and the scanner mirror or mirrors are controlled to maintain the position and orientation of the OCT scanner system field of view.
[0027] To maintain an OCT agnostic platform, conventional sawtooth scanner drive waveforms, generated from an iOCT system, are relayed through a data acquisition board to add voltage and phase offsets calculated from the stereo vision system to track the tip position and orientation of surgical instruments. At the beginning of each imaging session, one second of the drive signal for each galvanometer was sampled at 100 KS/s on two analog channels and stored in internal buffers. The zero-crossings in the stored sampled signals were determined to identify the start of each scan trajectory. For each channel, the corresponding output buffer outputs contiguous chunks of samples from the stored drive signal corresponding to a single B-scan. The output buffer looped over the stored signal circularly, starting and ending at a zero crossing to avoid discontinuities in the output signal. The modified output signals were then output to their respective galvanometer scanner drivers to maintain the desired position and orientation of the OCT field of view.
[0028] FIG. 4 illustrates a method 150 for calibrating a stereo vision system for instrument tracking in a microscope-integrated intraoperative optical coherence tomography (OCT) scanner system. As discussed previously, using either of a pair of stereo vision cameras as a reference frame, the 3-D coordinates of point M relative to that camera can be calculated using similar triangles. This process, however, assumes a perfectly parallel pair of cameras and ignores the contribution of the image sensor in the image projection. In the practical case, there will be an arbitrary rotation and translation between the cameras, referred to as the relative pose, and the contribution of the image sensor for each camera, the intrinsic parameters, that needs to be accounted for.
[0029] To this end, at 152 a model, comprising a plurality of markers in a known arrangement, is moved to a plurality of locations within a field of view of a plurality of camera assemblies. In one implementation, the model comprises four markers forming four corners of a square having a side length of 17.5 millimeters. At 154, a plurality of images of the model are captured at each of the plurality of camera assemblies, such that each camera generates a plural set of two-dimensional locations for each marker.
[0030] At 156, intrinsic parameters for each of the camera assemblies and a relative pose between the camera assemblies are determined from the plural set of two-dimensional locations for each marker. To this end, a homography matrix, H, is generated for each camera as:
H = A[Rt] Eq.
1
[0031] where A represents the intrinsic parameters matrix, and [Rt] is a 3x4 matrix that describes the rotation and translation between the cameras. For a reference camera, [R t] will become an identity matrix.
[0032] The intrinsic parameters describe the properties of the image sensor that depend mainly on the shape of its pixels, and the pixel coordinates that represent the origin, which is not necessarily at the center of the sensor. For a given camera, the intrinsic parameters matrix can be represented as:
Figure imgf000012_0001
2
[0033] where Sx is a scaling parameter based on pixel length in a x direction, Sy is a scaling parameter based on pixel length in a / direction, Θ is a skew angle of a given pixel in the camera, and (x0, yo) represent the pixel coordinates of the camera origin.
[0034] Once the homography matrix for each camera has been determined, a second calibration step is performed to determine a transform from a coordinate domain associated with the reference camera to a coordinate system associated with the OCT scanner system. A model, including at least one marker, is moved within the field of view of the OCT scanner system and the plurality of cameras at 1 58. It will be appreciated that this can be the same model as that used in 1 52, or a different model. In one implementation, this can be done with a surgical instrument appropriately affixed with markers, such as the instrument illustrated in FIG. 2. In one implementation, the model is moved to four known positions in the OCT coordinate system, with two lying on an x-axis of the OCT coordinate system and two lying on a y-axis of the coordinate system. At 1 60, a plurality of images of the model are captured at each of the plurality of camera assemblies.
[0035] At 1 62, the transform from the coordinate domain associated with the reference camera to the coordinate system associated with the OCT scanner system is determined from the plurality of images of the model captured at 160. The coordinate transformation is implemented as a three-dimensional translation and rotation. In one implementation, the three-dimensional position of the marker or markers can be determined from the captured images and arranged as a 3xM matrix, Xstereo, in which each of M determined positions are stored as ordered triplets representing coordinates in the camera reference plane. Xmotor is a 3xM matrix representing the position of the model within the OCT coordinate frame. From this data, a 3x3 rotation matrix, R, is determined such that:
X R X Eq .
3
[0036] An appropriate optimization algorithm, such as least squares
optimization, can be used to solve for the rotation matrix R.
[0037] Even cameras of the same model from the same manufacturer can have significant variation in their intrinsic parameters, leading to inaccurate positions for the instrument tip in the absence of such calibration. It will be appreciated that the calibration procedure of FIG. 4 allows for a significant increase in the accuracy of the system, which is allows for the submillimeter precision necessary for OCT assisted surgery.
[0038] FIG. 5 is a schematic block diagram illustrating an exemplary system 200 of hardware components capable of implementing examples of the systems and methods disclosed in FIGS. 1 -4, such as the stereo vision system 38, the instrument tracking component 42, and the drive control illustrated in FIG. 1 . The system 200 can include various systems and subsystems. The system 200 can be a personal computer, a laptop computer, a workstation, a computer system, an appliance, an application-specific integrated circuit (ASIC), a server, a server blade center, a server farm, etc.
[0039] The system 200 can includes a system bus 202, a processing unit 204, a system memory 206, memory devices 208 and 210, a communication interface 212 (e.g., a network interface), a communication link 214, a display 216 (e.g., a video screen), and an input device 218 (e.g., a keyboard and/or a mouse). The system bus 202 can be in communication with the processing unit 204 and the system memory 206. The additional memory devices 208 and 210, such as a hard disk drive, server, stand alone database, or other non-volatile memory, can also be in communication with the system bus 202. The system bus 202 interconnects the processing unit 204, the memory devices 206-210, the communication interface 212, the display 216, and the input device 218. In some examples, the system bus 202 also interconnects an additional port (not shown), such as a universal serial bus (USB) port.
[0040] The processing unit 204 can be a computing device and can include an application-specific integrated circuit (ASIC). The processing unit 204 executes a set of instructions to implement the operations of examples disclosed herein. The processing unit can include a processing core.
[0041] The additional memory devices 206, 208 and 210 can store data, programs, instructions, database queries in text or compiled form, and any other information that can be needed to operate a computer. The memories 206, 208 and 210 can be implemented as computer-readable media (integrated or removable) such as a memory card, disk drive, compact disk (CD), or server accessible over a network. In certain examples, the memories 206, 208 and 210 can comprise text, images, video, and/or audio, portions of which can be available in formats
comprehensible to human beings.
[0042] Additionally or alternatively, the system 200 can access an external data source or query source through the communication interface 212, which can communicate with the system bus 202 and the communication link 214.
[0043] In operation, the system 200 can be used to implement one or more parts of a therapeutic delivery system in accordance with the present invention.
Computer executable logic for implementing the diagnostic system resides on one or more of the system memory 206, and the memory devices 208, 210 in accordance with certain examples. The processing unit 204 executes one or more computer executable instructions originating from the system memory 206 and the memory devices 208 and 210. The term "computer readable medium" as used herein refers to a medium that participates in providing instructions to the processing unit 204 for execution, and can, in practice, refer to multiple, operatively connected apparatuses for storing machine executable instructions.
[0044] What have been described above are examples of the present invention. It is, of course, not possible to describe every conceivable combination of components or methodologies for purposes of describing the present invention, but one of ordinary skill in the art will recognize that many further combinations and permutations of the present invention are possible. Accordingly, the present invention is intended to embrace all such alterations, modifications, and variations that fall within the scope of the appended claims.

Claims

CLAIMS What is claimed is:
1 . A microscope-integrated intraoperative optical coherence tomography (OCT) scanner system having automated tracking of an instrument tip, the system comprising:
at least one scanning mirror configured such that a field of view of the OCT scanner system is determined by at least one of a position and an orientation of the at least one scanning mirror;
a drive system configured to control the at least one of the position and the orientation of the at least one scanning mirror;
a plurality of camera assemblies configured to determine respective two- dimensional projections of the positions of a plurality of markers attached to a surgical instrument;
a stereo vision system configured to determine a three-dimensional location of each of the plurality of markers from the determined two-dimensional positions at the plurality of camera assemblies;
an instrument tracking component configured to determine a position of a working tip of the surgical instrument according to the determined three-dimensional locations of the plurality of markers; and
a drive control configured to instruct the drive system to adjust the at least one of the position and the orientation of the at least one scanner mirror such that the field of view of the OCT scanner system is determined from a position of the working tip of the surgical instrument.
2. The microscope-integrated intraoperative OCT scanner system of claim 1 , the stereo vision system being configured to calculate a first ray from an origin of a first camera of the plurality of camera assemblies to a given one of the plurality of markers, calculate a second ray from a center point of a second camera of the plurality of camera assemblies to the given one of the plurality of markers, find a shortest line segment connecting the first ray and the second ray, and determining the three-dimensional location of the given one of the plurality of markers as a midpoint of the shortest line segment connecting the first ray and the second ray.
3. The microscope-integrated intraoperative OCT scanner system of claim 1 , wherein each of the plurality of cameras are within one hundred millimeters of one another.
4. The microscope-integrated intraoperative OCT scanner system of claim 1 , wherein at least a subset of the plurality of markers are aligned along a longitudinal axis of the surgical instrument and instrument tracking component is configured to determine an orientation of the instrument from the position of the plurality of markers.
5. The microscope-integrated intraoperative OCT scanner system of claim 4, wherein the surgical instrument comprises at least two of the plurality of markers aligned along a line parallel with a longitudinal axis of the surgical instrument and at least one marker of the plurality of markers that is not aligned along the line parallel with a longitudinal axis of the surgical instrument.
6. The microscope-integrated intraoperative OCT scanner system of claim 1 , wherein the instrument tracking component is configured to transform the position of a working tip of the surgical instrument from a coordinate system associated with the stereo vision system to a coordinate system associated with the OCT scanner system via an appropriate coordinate transform.
7. The microscope-integrated intraoperative OCT scanner system of claim 1 , wherein the markers are active markers implemented as infrared light emitting diodes.
8. The microscope-integrated intraoperative OCT scanner system of claim 1 , wherein the drive control is configured to instruct the drive system to adjust the at least one of the position and the orientation of the at least one scanner mirror such that the field of view of the OCT scanner system is oriented perpendicular to an axis of the surgical instrument.
9. A method for adjusting a field of view of microscope-integrated intraoperative optical coherence tomography (OCT) scanner system to track the tip of a surgical instrument, the method comprising:
determining respective two-dimensional projections of the positions of a plurality of markers attached to a surgical instrument;
determining a three-dimensional location of each of the plurality of markers from the determined two-dimensional positions at the plurality of camera assemblies and a known relationship among the plurality of camera assemblies;
determining a position of a working tip of the surgical instrument according to the determined three-dimensional locations of the plurality of markers; and
controlling at least one of the position and the orientation of at least one scanner mirror associated with the OCT scanner system such that the field of view of the OCT scanner system is determined by the position of the working tip of the surgical instrument.
10. The method of claim 9, further comprising:
moving a model comprising a plurality of markers in a known arrangement within a field of view of the plurality of camera assemblies;
capturing a plurality of images of the model at each of the first and second camera assemblies, such that each camera generates a plural set of two- dimensional locations for each marker; and
determining at least a portion of the known relationship among the plurality of camera assemblies as a relative pose from the plural set of two-dimensional locations for each marker at each of the first and second cameras and the known arrangement of the plurality of markers.
1 1 . The method of claim 10, further comprising:
moving a marker to a plurality of known locations within a field of view of the OCT scanner system;
capturing a set of images of the marker at each of the plurality of camera assemblies; and
determining the coordinate transform from the sets of images captured at each of the plurality of camera assemblies.
12. The method of claim 1 1 , further comprising:
recording each of the known locations within the coordinate system associated with the OCT scanner system as three-dimensional locations in a first matrix, Xmotor;
determining three-dimensional locations of the marker in the coordinate system associated with the plurality of camera assemblies as a second matrix,
XstereoJ &nd
solving the relationship Xmotor = fix X5(mo for the coordinate transform, R, using an appropriate optimization algorithm.
13. The method of claim 9, wherein controlling at least one of the position and the orientation of at least one scanner mirror comprises controlling at least one of the position and the orientation of at least one scanner mirror associated with the OCT scanner system instruct the drive system to adjust the at least one of the position and the orientation of the at least one scanner mirror such that the field of view of the OCT scanner system is oriented perpendicular to an axis of the surgical instrument.
14. The method of claim 9, further comprising transforming the position of a working tip of the surgical instrument from a coordinate system associated with the stereo vision system to a coordinate system associated with the OCT scanner system via an appropriate coordinate transform.
15. The method of claim 9, wherein the markers are active markers implemented as infrared light emitting diodes.
16. The method of claim 9, further comprising transforming the position of the working tip of the surgical instrument from a coordinate system associated with the plurality of camera assemblies to a coordinate system associated with the OCT scanner system via a coordinate transform.
17. A method for adjusting a field of view of microscope-integrated intraoperative optical coherence tomography (OCT) scanner system to track the tip of a surgical instrument, the method comprising:
moving a model comprising a plurality of markers in a known arrangement within a field of view of a plurality of camera assemblies;
capturing a plurality of images of the model at each of the plurality of camera assemblies, such that each camera generates a plural set of two-dimensional locations for each marker;
determining a relative pose between the plurality of camera assemblies from the plural set of two-dimensional locations for each marker at each of the plurality of cameras and the known arrangement of the plurality of markers;
determining respective two-dimensional projections of the positions of a plurality of markers attached to a surgical instrument;
determining a three-dimensional location of each of the plurality of markers from the determined two-dimensional positions at the plurality of camera assemblies and the relative pose;
determining a position of a working tip of the surgical instrument according to the determined three-dimensional locations of the plurality of markers; and
controlling at least one of the position and the orientation of at least one scanner mirror associated with the OCT scanner system such that the field of view of the OCT scanner system is determined by the position of the working tip of the surgical instrument.
18. The method of claim 17, further comprising transforming the position of the working tip of the surgical instrument from a coordinate system associated with the plurality of camera assemblies to a coordinate system associated with the OCT scanner system via a coordinate transform.
19. The method of claim 18, further comprising:
moving a marker to a plurality of known locations within a field of view of the OCT scanner system;
capturing a set of images of the marker at each of the plurality of camera assemblies; and
determining the coordinate transform from the sets of images captured at each of the plurality of camera assemblies.
20. The method of claim 19, further comprising:
recording each of the known locations within the coordinate system
associated with the OCT scanner system as three-dimensional locations in a first matrix, Xmotor;
determining three-dimensional locations of the marker in the coordinate system associated with the plurality of camera assemblies as a second matrix,
XstereoJ &nd
solving the relationship Xmotor = fix X5(mo for the coordinate transform, R, using an appropriate optimization algorithm.
PCT/US2016/031452 2015-05-07 2016-05-09 Instrument tracking in oct-assisted surgery Ceased WO2016179582A1 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US201562158006P 2015-05-07 2015-05-07
US62/158,006 2015-05-07

Publications (1)

Publication Number Publication Date
WO2016179582A1 true WO2016179582A1 (en) 2016-11-10

Family

ID=56080463

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/US2016/031452 Ceased WO2016179582A1 (en) 2015-05-07 2016-05-09 Instrument tracking in oct-assisted surgery

Country Status (2)

Country Link
US (1) US10045831B2 (en)
WO (1) WO2016179582A1 (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111251189A (en) * 2019-11-12 2020-06-09 长沙长泰机器人有限公司 Visual positioning method for casting polishing

Families Citing this family (20)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2015183994A1 (en) 2014-05-28 2015-12-03 Santec Corporation Non-invasive optical measurement of blood analyte
JP2016073409A (en) * 2014-10-03 2016-05-12 ソニー株式会社 Information processing apparatus, information processing method, and operation microscope apparatus
DE102015002729A1 (en) * 2015-02-27 2016-09-01 Carl Zeiss Meditec Ag Ophthalmic laser therapy device and method for generating corneal access incisions
US20160331584A1 (en) * 2015-05-14 2016-11-17 Novartis Ag Surgical tool tracking to control surgical system
US10677580B2 (en) 2016-04-27 2020-06-09 Santec Corporation Optical coherence tomography system using polarization switching
IL245560B2 (en) 2016-05-09 2024-05-01 Elbit Systems Ltd Optical coherence tomography (OCT) images are localized for surgical procedures in ophthalmology
US20190117459A1 (en) 2017-06-16 2019-04-25 Michael S. Berlin Methods and Systems for OCT Guided Glaucoma Surgery
US20180360655A1 (en) * 2017-06-16 2018-12-20 Michael S. Berlin Methods and systems for oct guided glaucoma surgery
CN109589179B (en) * 2017-10-02 2023-01-17 吕孝宇 Mixed reality system and method for determining spatial coordinates of a dental instrument
US10657726B1 (en) * 2017-10-02 2020-05-19 International Osseointegration Ai Research And Training Center Mixed reality system and method for determining spatial coordinates of dental instruments
US11213200B2 (en) * 2018-03-22 2022-01-04 Santec Corporation Topographical imaging using combined sensing inputs
US11278360B2 (en) * 2018-11-16 2022-03-22 Globus Medical, Inc. End-effectors for surgical robotic systems having sealed optical components
US12029501B2 (en) 2019-01-11 2024-07-09 Vanderbilt University Automated instrument-tracking and adaptive image sampling
DE102020108345A1 (en) 2020-03-26 2021-09-30 Carl Zeiss Meditec Ag Reading in codes using a surgical microscope camera
DE102021202384B3 (en) 2021-03-11 2022-07-14 Carl Zeiss Meditec Ag Microscope system, medical instrument and calibration method
EP4319678A4 (en) * 2021-04-06 2025-02-26 Digital Surgery Systems, Inc. SYSTEM, METHOD AND APPARATUS FOR TRACKING A TOOL VIA A DIGITAL SURGICAL MICROSCOPE
DE102021206971A1 (en) * 2021-07-02 2023-01-05 Carl Zeiss Meditec Ag Method and system for optically detecting the position of at least one instrument in an operating room
US12313883B2 (en) 2023-03-03 2025-05-27 santec Holdings Corporation Photonic beam steering device with wavelength sweep
WO2025034761A1 (en) * 2023-08-07 2025-02-13 Evident Vascular, Inc. Systems and methods of manually assisted movement for spatial alignment lengthwise measurements in intravascular ultrasound imaging
CN118766591B (en) * 2024-09-05 2024-12-24 图湃(北京)医疗科技有限公司 Intraoperative OCT imaging position tracking device, surgical microscope system and storage medium

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20120184846A1 (en) * 2011-01-19 2012-07-19 Duke University Imaging and visualization systems, instruments, and methods using optical coherence tomography
US20130060146A1 (en) * 2010-04-28 2013-03-07 Ryerson University System and methods for intraoperative guidance feedback
US20140221822A1 (en) * 2013-02-04 2014-08-07 The Cleveland Clinic Foundation Instrument depth tracking for oct-guided procedures

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6741359B2 (en) * 2002-05-22 2004-05-25 Carl Zeiss Meditec, Inc. Optical coherence tomography optical scanner

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20130060146A1 (en) * 2010-04-28 2013-03-07 Ryerson University System and methods for intraoperative guidance feedback
US20120184846A1 (en) * 2011-01-19 2012-07-19 Duke University Imaging and visualization systems, instruments, and methods using optical coherence tomography
US20140221822A1 (en) * 2013-02-04 2014-08-07 The Cleveland Clinic Foundation Instrument depth tracking for oct-guided procedures

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111251189A (en) * 2019-11-12 2020-06-09 长沙长泰机器人有限公司 Visual positioning method for casting polishing
CN111251189B (en) * 2019-11-12 2021-07-27 长沙长泰机器人有限公司 Visual positioning method for casting polishing

Also Published As

Publication number Publication date
US10045831B2 (en) 2018-08-14
US20160324593A1 (en) 2016-11-10

Similar Documents

Publication Publication Date Title
US10045831B2 (en) Instrument tracking in OCT-assisted surgery
JP7561221B2 (en) Live 3D holographic guidance and navigation for performing interventional procedures
US12219228B2 (en) Stereoscopic visualization camera and integrated robotics platform
US20220079686A1 (en) 3d navigation system and methods
JP6042468B2 (en) Microsurgery system and navigation method based on guidance by optical coherence tomography
JP7837556B2 (en) System and method for overlaying a virtual image onto a real-time image
US10426345B2 (en) System for generating composite images for endoscopic surgery of moving and deformable anatomy
US11546569B2 (en) System and methods for correcting image data of distinct images and generating and stereoscopic three-dimensional images
US11389249B2 (en) Localized optical coherence tomography images for ophthalmological surgical procedures
El-Haddad et al. Automated stereo vision instrument tracking for intraoperative OCT guided anterior segment ophthalmic surgical maneuvers
US20180125586A1 (en) System and method for providing a contour video with a 3d surface in a medical navigation system
CN112292092B (en) System and method for measuring distance using a stereo endoscope
CA3029348C (en) Intraoperative medical imaging method and system
US20240081921A1 (en) System and method for verification of conversion of locations between coordinate systems
US10827162B1 (en) Augmented optical imaging system for use in medical procedures
US20170258528A1 (en) Medical imaging system for illuminating tissue samples using three-dimensional structured illumination microscopy
Figl et al. A fully automated calibration method for an optical see-through head-mounted operating microscope with variable zoom and focus
CN104280886A (en) Microscopic system and microscopic method based on in-situ three-dimensional enhanced display
JP2016016331A (en) Ophthalmic surgery system
Weiss et al. Fast 5DOF needle tracking in iOCT
CN117413060A (en) Augmented reality systems for real space navigation and surgical systems using the same
JP2020530367A (en) Ophthalmic equipment
US11717150B2 (en) Ophthalmologic apparatus, and ophthalmologic information processing apparatus
US20150348311A1 (en) Method and apparatus for determining a surface topography of a body in a coordinate system fixed in space and/or fixed on the body
Lin et al. A cranial-feature-based registration scheme for robotic micromanipulation using a microscopic stereo camera system

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 16725006

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 16725006

Country of ref document: EP

Kind code of ref document: A1