WO2015187620A1 - Navigation chirurgicale avec stéréovision et procédés associés - Google Patents

Navigation chirurgicale avec stéréovision et procédés associés Download PDF

Info

Publication number
WO2015187620A1
WO2015187620A1 PCT/US2015/033672 US2015033672W WO2015187620A1 WO 2015187620 A1 WO2015187620 A1 WO 2015187620A1 US 2015033672 W US2015033672 W US 2015033672W WO 2015187620 A1 WO2015187620 A1 WO 2015187620A1
Authority
WO
WIPO (PCT)
Prior art keywords
model
stereo image
stereo
annotating data
data
Prior art date
Application number
PCT/US2015/033672
Other languages
English (en)
Inventor
David W. Roberts
Keith D. Paulsen
Alexander Hartov
Songbai Ji
Frederic Leblond
Brian W. Pogue
Scott C. Davis
Dax Kepshire
Original Assignee
The Trustees Of Dartmouth College
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by The Trustees Of Dartmouth College filed Critical The Trustees Of Dartmouth College
Publication of WO2015187620A1 publication Critical patent/WO2015187620A1/fr
Priority to US15/367,243 priority Critical patent/US10568535B2/en
Priority to US16/785,882 priority patent/US11129562B2/en
Priority to US17/486,839 priority patent/US11690558B2/en
Priority to US18/204,091 priority patent/US12114988B2/en

Links

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/20—Image signal generators
    • H04N13/204—Image signal generators using stereoscopic image cameras
    • H04N13/239—Image signal generators using stereoscopic image cameras using two 2D image sensors having a relative position equal to or related to the interocular distance
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T19/00—Manipulating 3D models or images for computer graphics
    • G06T19/006—Mixed reality
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B90/00—Instruments, implements or accessories specially adapted for surgery or diagnosis and not covered by any of the groups A61B1/00 - A61B50/00, e.g. for luxation treatment or for protecting wound edges
    • A61B90/10—Instruments, implements or accessories specially adapted for surgery or diagnosis and not covered by any of the groups A61B1/00 - A61B50/00, e.g. for luxation treatment or for protecting wound edges for stereotaxic surgery, e.g. frame-based stereotaxis
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B90/00—Instruments, implements or accessories specially adapted for surgery or diagnosis and not covered by any of the groups A61B1/00 - A61B50/00, e.g. for luxation treatment or for protecting wound edges
    • A61B90/36—Image-producing devices or illumination devices not otherwise provided for
    • A61B90/361—Image-producing devices, e.g. surgical cameras
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B90/00—Instruments, implements or accessories specially adapted for surgery or diagnosis and not covered by any of the groups A61B1/00 - A61B50/00, e.g. for luxation treatment or for protecting wound edges
    • A61B90/36—Image-producing devices or illumination devices not otherwise provided for
    • A61B90/37—Surgical systems with images on a monitor during operation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/30—Determination of transform parameters for the alignment of images, i.e. image registration
    • G06T7/33—Determination of transform parameters for the alignment of images, i.e. image registration using feature-based methods
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
    • H04N13/106—Processing image signals
    • H04N13/111—Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
    • H04N13/106—Processing image signals
    • H04N13/172—Processing image signals image signals comprising non-image signal components, e.g. headers or format information
    • H04N13/183—On-screen display [OSD] information, e.g. subtitles or menus
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B90/00—Instruments, implements or accessories specially adapted for surgery or diagnosis and not covered by any of the groups A61B1/00 - A61B50/00, e.g. for luxation treatment or for protecting wound edges
    • A61B90/36—Image-producing devices or illumination devices not otherwise provided for
    • A61B2090/364—Correlation of different images or relation of image positions in respect to the body
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B90/00—Instruments, implements or accessories specially adapted for surgery or diagnosis and not covered by any of the groups A61B1/00 - A61B50/00, e.g. for luxation treatment or for protecting wound edges
    • A61B90/36—Image-producing devices or illumination devices not otherwise provided for
    • A61B2090/364—Correlation of different images or relation of image positions in respect to the body
    • A61B2090/365—Correlation of different images or relation of image positions in respect to the body augmented reality, i.e. correlating a live optical image with another image
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B90/00—Instruments, implements or accessories specially adapted for surgery or diagnosis and not covered by any of the groups A61B1/00 - A61B50/00, e.g. for luxation treatment or for protecting wound edges
    • A61B90/36—Image-producing devices or illumination devices not otherwise provided for
    • A61B2090/364—Correlation of different images or relation of image positions in respect to the body
    • A61B2090/367—Correlation of different images or relation of image positions in respect to the body creating a 3D dataset from 2D images using position information
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B90/00—Instruments, implements or accessories specially adapted for surgery or diagnosis and not covered by any of the groups A61B1/00 - A61B50/00, e.g. for luxation treatment or for protecting wound edges
    • A61B90/36—Image-producing devices or illumination devices not otherwise provided for
    • A61B90/37—Surgical systems with images on a monitor during operation
    • A61B2090/371—Surgical systems with images on a monitor during operation with simultaneous use of two cameras
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B90/00—Instruments, implements or accessories specially adapted for surgery or diagnosis and not covered by any of the groups A61B1/00 - A61B50/00, e.g. for luxation treatment or for protecting wound edges
    • A61B90/36—Image-producing devices or illumination devices not otherwise provided for
    • A61B90/37—Surgical systems with images on a monitor during operation
    • A61B2090/372—Details of monitor hardware
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10004—Still image; Photographic image
    • G06T2207/10012—Stereo images
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10072—Tomographic images
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30004—Biomedical image processing
    • G06T2207/30016—Brain
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2210/00—Indexing scheme for image generation or computer graphics
    • G06T2210/41—Medical

Definitions

  • a system for generating a photographic image of a surgical field includes a stereo image capture device for capturing a stereo image stream from two cameras imaging the surgical field; and a stereo viewer for displaying the stereo image stream to a surgeon.
  • the system also includes a stereo image to 3D surface extraction module, implemented as machine readable instructions stored in memory of a computer, that is capable of, when the instructions are executed by a processor of the computer, generating a first 3D model of the surgical field from the stereo image streams; a registration module, implemented as machine readable instructions stored in the memory, that is capable of, when the instructions are executed by the processor, co -registering annotating data with the first 3D model; and a stereo image enhancer, implemented as machine readable instructions stored in the memory, that is capable of, when the instructions are executed by the processor, graphically overlaying at least part of the annotating data onto the stereo image stream to form an enhanced stereo image stream for display by the stereo viewer, wherein the enhanced stereo stream enhances the surgeon' s perception of the surgical field.
  • a system configured for graphically representing and displaying a first and second annotating data in relation to a surgical field, including an interactive display device; a registration module, implemented as machine readable instruction stored in a memory of a computer, that is capable of, when the instructions are executed by a processor of the computer, co -registering the first and second annotating data; and a stereo image enhancer, implemented as machine readable instruction stored in the memory, that is capable of, when the instructions are executed by the processor, graphically overlaying at least part of the annotating data onto the stereo image stream to form an enhanced stereo image stream for display by the stereo viewer, wherein the enhanced stereo stream enhances the surgeon's perception of the surgical field.
  • a registration module implemented as machine readable instruction stored in a memory of a computer, that is capable of, when the instructions are executed by a processor of the computer, co -registering the first and second annotating data
  • a stereo image enhancer implemented as machine readable instruction stored in the memory, that is capable of, when the instructions are executed by the processor,
  • a method for surgical navigation with stereo vision including capturing a stereo image pair of a surgical field; generating a 3D surface model from the stereo image pair; registering preoperatively captured annotating data with the 3D surface model; generating an enhanced stereo image based upon the stereo image pair and the annotating data; wherein the annotating data is graphically overlaid onto the stereo image pair.
  • FIG. 1 shows one exemplary system for surgical navigation with stereovision, in an embodiment.
  • FIG. 2 shows the stereo image enhancer of FIG. 1 in further exemplary detail.
  • FIG. 3 shows a portion of the stereo imaging system of FIG. 1 configured with stereo hyperspectral cameras for capturing a stereo hyperspectral image stream, in an embodiment.
  • FIG. 4 is a flowchart illustrating one exemplary method for surgical navigation with stereo vision, in an embodiment.
  • FIG. 5 shows examples of the annotating data of FIG. 1.
  • FIG. 6A is a perspective view of one exemplary implantable device model that is stored within the system of FIG. 1, in an embodiment.
  • FIG. 6B is a perspective view of one exemplary implantable device model that is stored within the system of FIG. 1 in an embodiment.
  • FIG. 7 shows one exemplary system with an interactive stereo display device for interacting with a surgeon to analyze and/or create a surgical procedure for a patient, in an embodiment.
  • FIG. 8 shows the system of FIG. 1 further configured for use in positioning an implantable device during a surgical procedure, in an embodiment.
  • FIG. 9 shows one exemplary optical system for surgical navigation with stereovision, in an embodiment.
  • FIG. 10A is a perspective view of the 3D model of FIG. 1 and the implantable device model of FIG. 6B.
  • FIG. 10B is a 2D representation of the enhanced stereo image stream of the surgical field corresponding to FIG. 10A as generated by the system of FIG. 1.
  • FIG. 11 is a perspective view of the 3D model and the annotating data 3D model generated from the annotating data of FIG. 1, where the annotating data is based upon MRI tomography data.
  • FIG. 1 shows one exemplary system 100 for surgical navigation with stereovision.
  • System 100 includes a stereo viewer 102, a stereo image annotator or enhancer 104, and a stereo image capture device 106.
  • System 100 is used to aid a surgeon during an operation on a person 180.
  • the surgeon is performing a procedure on a brain 182 of person 180.
  • system 100 may be used to aid procedures on other parts of a person's body, and may be used when performing procedures on animals as well as on humans.
  • Stereo image capture device 106 includes illumination sources 114, 116 and at least two cameras 110, 112, that cooperate to capture a high quality stereo image stream 107 of a surgical field 184 (i.e., the area where the procedure is performed).
  • System 100 may be calibrated using one or more calibration targets used in surgical field 184, as taught by PCT/US2009/045082.
  • Stereo image enhancer 104 receives stereo image stream 107 from capture device 106 and generates an enhanced stereo image stream 103 from stereo image stream 107 by adding information from annotating data 108.
  • Annotating data 108 is for example a previously stored imaging data set such as one of (a) preoperative,
  • intraoperative and postoperative radiological studies selected from the group including MRI, fMRI, CT, SPECT, and PET, and (b) physiological studies selected from the group including EEG, evoked potentials, and magnetoencephalography (MEG).
  • EEG electroencephalography
  • At least three reference points 118, 120, 122 may be attached to patient 180 for the duration of the procedure and the location of each reference point is defined as a location within a patent-centered coordinate system of patient 180.
  • a tracker 129 determines, relative to itself, locations of reference points 118, 120, 122 attached to patient 180 and reference points 124, 126, 128 attached to stereo image capture device 106.
  • Registration module 132 uses these locations to determine a view reference 131 (e.g., spatial relationship including relative position and orientation) between patient 180 and stereo image capture device 106.
  • View reference 131 defines a spatial relationship of stereo image stream 107 relative to surgical field 184, and may also incorporate functions (e.g., zoom) of optics within cameras 110 and 112.
  • Tracker 129 is in some embodiments a commercially available operating room tracking device that determines positions of reference points or transponders attached to equipment in the operating room, including stereo image capture device 106, and may include transponders attached to the patient. Tracker 129 may utilize one or both of optical and electromagnetic tracking as known in the art.
  • registration module 132 may automatically determine a spatial relationship between stereo image capture device 106 and patient 180, or to a frame attached to patent 180.
  • 3D model 134 allows each pixel within stereo image stream 107 to be located within the patient-centered coordinate system. By determining this relationship between pixels of stereo image stream 107 and the patient-centered reference system, image post-processing techniques may be used to evaluate captured images.
  • registration module 132 In a particular embodiment where registration module 132
  • the registration module determines features of patient bone structure from preoperative CT scans and/or MRI scans, and registers this bone structure to features of bone structure determined from other CT scans and MRI scans for the same patient as well as to bone structure determined from 3D surfaces extracted from the stereo images.
  • an inconsistency such as a scan with significant bony differences from other scans associated with the patient, or features of facial 3D surfaces that are inconsistent with features of bone structure extracted from the 3D surfaces, is found while
  • stereo image stream 107 is recorded throughout a surgical procedure, and the recorded stereo image stream is effectively continuously geotagged by attaching an encoded position and orientation of image capture device 106 as determined by tracker 129 in real-time.
  • the recorded image stream is available for documentation, and image pairs from the image stream are sampled and processed as indicated herein for extraction of 3D surface models and for co-registration of, and co- presentation of, data from other image sources, as described herein, periodically throughout a surgical procedure.
  • Stereo image enhancer 104 may be controlled, in a particular embodiment by the surgeon or a surgical assistant, to graphically enhance stereo image stream 107 with selected information from annotating data 108 such that the surgeon has an enhanced view of surgical field 184 through stereo viewer 102.
  • Enhancer 104 includes a stereo image to 3D surface module 130 that generates a 3D model 134 of surgical field 184 from stereo image stream 107.
  • 3D model 134 includes a reference 135 that defines a spatial relationship of 3D model 134 relative to the patient-centered coordinate system of patient 180, in a particular embodiment the spatial relationship is determined from tracker 129 based upon reference points 118, 120, 122.
  • an operating-room centered coordinate system may be used, wherein the determined spatial relationship between patient 180 and stereo capture device 106 allows conversion of coordinates from one coordinate system to another according to a calibration. .
  • Registration module 132 operates to co-register annotating data 108 (based upon a reference 109 associated, or included with, annotating data 108) and 3D model 134 (based upon determined reference 135) and generate graphical information from annotating data 108 that corresponds to surgical field 184 as imaged within image stream 107.
  • annotating data 108 is a tomographic image generated from a MRI scan
  • that image may be scaled and oriented with respect to the surgical field viewed by the surgeon through stereo viewer 102 and graphically added to enhanced stereo image stream 103.
  • Enhancer 104 thereby enhances the view of surgical field 184 seen by the surgeon by adding information from other sources and co -registering this information to a conventional optical view; thus the surgeon assimilates the information without the need to move or avert his/her eyes from the surgical field.
  • the surgeon may view a corresponding view of 3D model 134 and a 3D model of annotating data 108, and may view data from the 3D model 134 with
  • a mapping between raw image coordinates and patient coordinates is established and, and images transformed from raw coordinate systems into the patient centered coordinate system, by registration module 132 for each image..
  • Registration module 132 may identify features within stereo image stream 107 and/or 3D model 134 that correspond to features identified within annotating data 108, wherein registration module 130 automatically aligns graphical images generated from annotating data 108 with stereo image stream 107. This automatic alignment may include one or more of shifting, rotating, warping, and scaling of annotating data 108 to align corresponding features between annotating data 108 and stereo image stream 107. Registration module 132 may also identify tissue shift within 3D model 134 as compared to annotating data 108, wherein registration module 132 may shift, rotate, warp, and/or scale annotating data 108 to correspond to 3D model 134.
  • Stereo image to 3D surface module 130 continually updates 3D model 134 based upon stereo image stream 107 and registration module 132 continually compares 3D module 134 to annotating data 108, shifting, rotating, warping, and/or scaling annotating data 108 to match 3D model 134.
  • registration module 132 utilizes a mechanical model of tissue, instruments, and structure within surgical field 184 such that accurate manipulation of annotating data 108 may be made corresponding to detected movement within 3D model 134.
  • enhancer 104 In one embodiment, where enhanced data 108 does not include imagery (i.e., EEG and/or evoked potentials), enhancer 104 generates contour lines mapped to 3D model 134 and corresponded to information within enhanced data 108. [0035] In another embodiment, where enhanced data 108 includes a 3D model or data derived in a voxel-based form (e.g., MRI data), enhancer 104 may generate one or more cross sections through an indicated incision plane and map that cross section to the incision plane within 3D model 134 such that the surgeon views the corresponding 3D structures beneath the incision plane prior to, and after, making the actual incision.
  • imagery i.e., EEG and/or evoked potentials
  • enhancer 104 may generate one or more cross sections through an indicated incision plane and map that cross section to the incision plane within 3D model 134 such that the surgeon views the corresponding 3D structures beneath the incision plane prior to, and after, making the actual incision.
  • system 100 makes information of enhanced data 108 available within the same field of view as surgical field 184, thereby lessening need for the surgeon to look away from stereo viewer 102 or from surgical field 184.
  • System 100 includes an interactive controller 170 adapted to select one of a plurality of predefined stored views 160 of 3D model 134 and annotating data 108.
  • controller 170 is controlled by a surgeon's foot.
  • controller 170 is controlled by the surgeon's eye movement, wherein controller 170 may be implemented within stereo viewer 102.
  • controller 170 may be implemented within stereo viewer 102.
  • controller 170 is controlled by a surgeon's voice using a trained speech interpretation interface.
  • Stored views 160 define a plurality of views that combine annotating data 108 with stereo image stream 107 to generate enhanced stereo image stream 103. For example, one view may select a particular annotating data 108 for enhancing stereo image stream 103. Each view within stored views 160 also defines which portions of annotating data 108 to select for combining with stereo image stream 107. For example, where a surgical incision is planned (see planned surgical procedures 760 of FIG. 7), stored view 160 may define a view that graphically indicates planned incisions within enhanced stereo image stream 103, and may further define a
  • annotating data 3D model 222 for display with the planned incision to indicate tissue structures beneath the intended point of incision. This provides the surgeon with additional information relative to the next immediate step in the planned operational procedure.
  • Stored views 160 may also define an alternate viewing point for display within stereo viewer 102, wherein the stereo image enhancer 104 generates the alternate view based upon 3D model 134 and annotating data 108.
  • a surgeon may elect to temporarily view a side image of a patient's head to learn additional information from MRI tomography data stored within annotating data 108 in relation to surgical field 184.
  • FIG. 2 shows stereo image enhancer 104 of FIG. 1 in further exemplary detail.
  • Registration module 132 includes an alignment refiner 202 that utilizes a shifting/rotating/scaling/warping module 204 to improve registration of annotating data 108 with 3D model 134.
  • Registration module 132 may also include a deformation modeler 206 that generates a 3D mechanical model 208 of annotating data 108 to model deformation of tissue within annotating data 108 based upon one or more of gravity, retraction, incision, and so on. Deformation modeler 206 thereby determines an appropriate deformation of annotating data 108 to match determined deformation of tissue within 3D model 134 and surgical field 184.
  • a portion of a patient's skull is removed for surgery, resulting in reduced pressure and support for soft brain tissues.
  • This reduced support allows the brain tissue to reposition as a result of gravity and reduced support.
  • Deformation modeler 206 uses 3D mechanical model 208 to estimate such deformation and alignment refiner 202 uses
  • shifting/rotating/scaling/warping module 204 to apply the estimated deformation to annotating data 108 such that annotating data 108 better aligns to 3D model 134.
  • deformation modeler 206 uses 3D mechanical model 208 to estimate deformation of surrounding tissue to better align annotating data 108 to 3D model 134.
  • 3D mechanical model 208 includes learning capability and is continually updated based upon detected deformation of actual tissue during the surgery.
  • stereo pairs captured using the stereo image capture device 106 are processed to extract a 3D surface model, which in turn is used to determine an actual surface configuration that is fed back to mechanical model 208; in a particular embodiment these 3D surface models are used to tune coefficients of a neural network implementation of a mechanical model 208.
  • a mechanical property extraction routine 231 updates 3D mechanical model 208 to better predict the outcome of the procedure.
  • Stereo image enhancer 104 may also include an annotating data 3D model generator 220 that generates an annotating data 3D model 222 from annotating data 108.
  • annotating data 3D model 222 is generated to represent a model of the scanned tissue.
  • annotating data 3D model generator 220 may generate annotating data 3D model 222 using contours based upon 3D model 134 or known shape of the patient.
  • Electrical activity may be recorded by EEG electrodes at particular times during a preoperative or an intraoperative EEG study, and may include recorded evoked potentials.
  • Preoperative studies may use conventional scalp electrodes, while intraoperative studies may use any combination of scalp, cortical surface, and deep electrodes appropriate to the surgical procedure and stage of surgery at which EEG study is desired.
  • This electrical activity is mapped onto a 3D model, for example as extracted from a CT scan or MRI imaging study, of the subject's brain.
  • the resulting 3D maps of electrical activity are processed by annotating data 3D model generator 220 to generate annotating data 3D model 222 that is then co-registered with 3D model 134 extracted from stereo image stream 107.
  • EEG activity associated with an epileptic focus may be used to enhance the surgeon' s view of the surgical field.
  • EEG activity associated with evoked potentials such as potentials evoked when the subject speaks, may thus be added to the surgeon's view of the surgical field.
  • EEG activity associated with particular areas such as Broca's area critical for speech, or visual cortex associated with the fovea, may be incorporated within enhanced stereo image stream 103.
  • Mechanical property extraction routine 231 updates 3D mechanical model 208 based upon observed displacement of tissue, such as when an indenter is place on the brain for example.
  • Stereo image enhancer 104 may also include a physiological tissue property analyzer 230 that processes intraoperative visual information for determination of tissue physiological properties, including but not limited to blood flow and
  • System 100 may have an integral tissue classifier based upon these physiological properties.
  • analyzer 230 may map and display tissue classifications and/or the presence or absence of particular degrees of oxygenation, light scattering, and absorbance parameters. Such physiological properties may be indicated within enhanced stereo image stream 103 and/or output separately. Since tumor cells may have different average sizes than normal cells, and many tumors are hypoxic, this information may be used to locate tumors.
  • Intraoperative functional neuroimaging is of use in isolating structures that require preservation, or in locating epileptic foci.
  • Registration module 132 may first align each model 134, 208, 222 and annotating data 108 based upon references 135, 209, 223, and 109 that defines a relationship of the model/data to a patient-centered reference system for example, and thereby to one another. Registration module 132 may then use alignment refiner 202 to better register two or more models together, such as by deforming annotating data 108 to align identifiable features within annotating data 108 with corresponding features within 3D model 134.
  • An image generator 250 then generates enhanced stereo image stream 103 based upon stereo image stream 107 and rendering of one or more of 3D model 134, 3D mechanical model 208, annotating data 3D model 222, and annotating data 108.
  • 3D model 134, 3D mechanical model 208, annotating data 3D model 222 and annotating data 108 may be graphically rendered and overlaid onto stereo image stream 107 to form enhanced stereo image stream 103.
  • FIG. 3 shows a portion of stereo imaging system 100 of FIG. 1 configured with stereo hyperspectral cameras 310, 312 for capturing a stereo
  • each camera 310, 312 has a monochrome sensor and a controllable color filter 311, 313, wherein the filter is controlled to selectively capture images of a particular color band.
  • each camera 310, 312 has a high resolution sensor that is fitted with pattered filters having a plurality (e.g., 64) of color bands, wherein the camera simultaneously captures images in each of the color bands.
  • pattered filters having a plurality (e.g., 64) of color bands, wherein the camera simultaneously captures images in each of the color bands.
  • These cameras operate similarly to conventional color imagers that typically have Bayer filters for separating the captured image into three color bands (red, green, and blue). However, in this embodiment, the image is separated into 64 (or more) distinct color bands (at the expense of resolution).
  • each camera 310, 312 has a scanning slit with a dispersing element (e.g., prism or grating) that disperses the image from each point along the slit across multiple pixels of a sensor array, and thereby provides a spectrum, including
  • a dispersing element e.g., prism or grating
  • An optical system is configured such that the slit traverses a field of view, and a hyperspectral cube is captured of the entire field of view.
  • cameras 310, 312 replace camera 110, 112, wherein processing within stereo image capture device 306 recombines stereo
  • hyperspectral image steam 307 to form stereo image stream 107.
  • stereo hyperspectral image stream 307 and stereo image stream 107 are generated from the same captured images, co-registration is ensured.
  • cameras 310, 312 are included with cameras 110, 112 such that image streams 107 and 307 are generated independently.
  • a hyperspectral image processor 302 receives stereo hyperspectral image stream 307 and generates a 3D hyperspectral model 304 with a reference 305.
  • Fluorescence depth processor 310 operates to generate a 3D
  • fluorescence model 312 with a reference 313 from hyperspectral stream 307; in an embodiment fluorescence depth processor 310 determines depth in tissue of fluorophores of 3D fluorescence model 312 by determining observed fluorescent emissions spectra - the intensity of observed fluorescent emissions in hyperspectral stream 307 at two or more wavelengths.
  • the depth processor 310 models spectral shift due to absorption and scattering of light by tissue at those two or more wavelengths relative to known emissions spectra of the fluorophores.
  • each fluorophore emits light with a known emissions spectra or distribution of intensity of emitted light at the two or more wavelengths, and tissue absorbs and scatters light at each wavelength differently, deeper fluorophores generally have a greater spectral shift between observed fluorescent emissions spectra and the known emissions spectra of the fluorophores.
  • Registration module 132 operates to co-register two or more models 134, 208, 222, 304, and 312, and image generator 205 operates to graphically combine at least part of one or more models 134, 208, 222, 304, and 312 with stereo image stream 107 to form enhanced stereo image stream 103.
  • Captured hyperspectral/fluorescent image streams, and/or determined 3D models therefrom, may be stored for use as annotating data 108 in a following procedure/operation.
  • a first part of a surgical procedure may collect hyperspectral/fluorescence data from a patient, wherein that data is stored within system 100 and used (e.g., as annotating data 108) within a subsequent part of the procedure.
  • FIG. 4 is a flowchart illustrating one exemplary method 400 for surgical navigation with stereo vision, in an embodiment.
  • method 400 captures a stereo image stream of a surgical field.
  • stereo image capture device 106 captures stereo image stream 107 of surgical field 184.
  • method 400 generates a 3D surface model from the stereo image stream.
  • stereo image to 3D surface module 130 generates 3D model 134 based upon stereo image stream 107.
  • method 400 generates a 3D enhancing model from annotating data.
  • annotating data 3D model generator 220 generates annotating data 3D model 222 from annotating data 108.
  • step 408 registers the 3D enhancing model with the 3D surface model.
  • registration module 132 aligns annotating data 3D model 222 with 3D model 134 based upon reference 223 of model 222 and reference 135 of model 134.
  • method 400 adjusts registration of the 3D enhancing model based upon matched features between the 3D enhancing model and the 3D surface model; in an embodiment features are matched between these 3D models by manual tagging, in an alternate embodiment common features of the models are matched automatically.
  • alignment refiner 202 is invoked by registration model 132 to better register/align identifiable features within annotating data 108 to corresponding features within 3D model 134.
  • alignment refiner 202 invokes deformation modeler 206 to annotate a 3D mechanical model 208 with annotating data 108 and then deform 3D mechanical model 208 based upon alignment of corresponding identified features of 3D model 134 to determine post- deformation locations of features found in annotating data 108. Alignment refiner 202 then invokes shifting/rotating/scaling/warping module 204 to apply the identified changes of 3D mechanical model 208 to annotating data 3D model 222.
  • step 412 method 400 generates an enhanced stereo image stream based upon the stereo image stream and the annotating data 3D model.
  • image generator 250 generates enhanced image stream 103 based upon stereo image stream 107 and at least part of enhanced data 3D model 222.
  • step 414 method 400 outputs the enhanced stereo image stream.
  • image generator 250 outputs enhanced stereo image stream 103 to stereo viewer 102.
  • FIG. 5 shows examples of annotating data 108 of FIG. 1.
  • annotating data 108 is a previously created set of data from one of radiological studies and physiological studies of the patient.
  • Each data set may be independently modeled by generator 220 to generate an independent annotating data 3D model 222 that may be included at least in part within enhanced stereo image stream 103.
  • annotating data 108 may include one or more of :a recorded hyperspectral image stream 502 that has an associated reference 503, a recorded fluorescence image stream 504 that has an associated reference 505, a recorded fluorescence difference image stream, a recorded hyperspectral image model 506 that has an associated reference 507, a recorded fluorescence image model 508 that has an associated reference 509, a recorded stereo image stream 510 that has an associated reference 511, a recorded 3D model 512 that has an associated reference 513, MRI data 514 that has an associated reference 515, an fMRI image 516 that has an associated reference 517, a CT image 518 that has an associated reference 519, a single-photon emission computed tomography (SPECT) image 520 that has an associated reference 521, a PET image 522 that has an associated reference 523, an EEG data set 524 that has an associated reference 525, an SPECT image 520 that has an associated reference 521, a PET image 522 that has an associated reference 523, an E
  • FIG. 6A shows a perspective view of one exemplary implantable device model 600 that is stored within system 100.
  • Model 600 represents an electrode array that may be implanted subdural, where positioning of the device during surgery is critical to successful operation.
  • FIG. 6B shows a perspective view of a model 650 of one exemplary implantable device model 650 that is stored within system 100.
  • Model 650 represents a deep brain probe, deep brain stimulator, electrode array, or other device, that should be inserted into a brain such that a tip 652 of the probe is positioned within a target area of the brain. Positioning of the device during surgery is critical to successful operation and is further complicated in that a surgeon is unable to visually see the location of tip 652 with respect to the target area.
  • a current position of the implantable device is determined by triangulation in the 3D images taken by the 3D imaging device using reference marks on the rear or top of the implantable device and known positions of the 3D imaging device.
  • a current 3D model of tissue is rendered and displayed.
  • the current 3D model is displayable from either the point of view of a tip of the implantable device, or from a point of view of a surgeon with a representation of the implantable device, including a representation of portions of the device that are hidden from view by tissue.
  • the system provides for superimposition of modeled locations, as displaced during surgery, of features extracted from annotating data, including a target area for the probe tip in the brain.
  • a surgeon viewing the target area from the point of view of the probe tip can aim the probe at the target area, and by viewing the probe and target area as viewed from a different angle, can judge how much further the probe should be inserted to reach the target area.
  • FIG. 7 shows one exemplary system 700 with an interactive stereo display device 701 (e.g., stereo viewer 102 of FIG. 1, a 3D television type display (for use with 3D glasses), touch screen, keyboard and mouse, and so on) for interacting with a surgeon to analyze and/or create a surgical procedure for a patient.
  • an interactive stereo display device 701 e.g., stereo viewer 102 of FIG. 1, a 3D television type display (for use with 3D glasses), touch screen, keyboard and mouse, and so on
  • a surgeon for analyze and/or create a surgical procedure for a patient.
  • system 700 is used outside of an operating theater to plan a surgical procedure using 3D models of the patient. Recorded data 190 from a previous surgery and/or annotating data 108 captured during a previous procedure on the patient are co- registered and presented for interaction with the surgeon.
  • System 700 has a computer 702 with a processor 704 that is communicatively coupled with a memory 706. Memory 706 is shown storing s registration module 732 and an image generator 750.
  • Registration module 732 and image generator 750 have similar functionality to registration module 132 and image generator 250 of system 100, FIGs. 1 and 2.
  • Registration module 732 and image generator 750 are at least partly implemented as machine readable instructions that are loaded and executed by processor 704 to perform the functionality described herein.
  • Registration module 732 co-registers two or more models to present a coherent stereovision display, that, if desired by the surgeon, shows features of derived from multiple preoperative imaging modalities such as MRI, CT, and ultrasound imaging to a surgeon using system 700.
  • system 700 allows a surgeon to plan an operation by manipulating views (e.g., rendered images that may include appropriate shading and simulated lighting) of one or more 3D models 734, 600, 722, defining the planned surgical procedures 760, such as making incisions, positioning an implantable device, and preselecting the appropriate views (stored within stored views 160) of annotating data 108 to support the operation.
  • views e.g., rendered images that may include appropriate shading and simulated lighting
  • the surgeon positions a model of an implantable device relative to a 3D model of a patient's brain generated from MRI data.
  • the surgeon defines one or more views of the modeled implant and modelled MRI data (and any other annotating data 108 that may provide useful information) that will provide additional guidance to the surgeon during the operation.
  • Implantable device model 600, planned surgical procedures 760, and stored views 160 are then used within system 100 to provide additional support to the surgeon during the operation.
  • the surgeon selects previously recorded data 190 and annotating data 108 for a patient and operates system 700 to generate a view of at least a portion of the patient based upon an indicated surgical field (e.g., surgical field 184, FIG. 1).
  • the surgeon may adjust the field of view to
  • Registration module 732 aligns (co -registers) a 3D model 734 generated from recorded data 190 with an annotating data 3D model 722 generated from annotating data 108 such that the generated view of 3D model 734 and annotating data 3D model 722 are consistent.
  • annotating data 3D model 722 is based upon MRI tomographic images
  • the surgeon may select a particular cross section or portion of annotating data 3D model 722 for display within stereo viewer 102 during the operation.
  • registration module 732 aligns 3D model 734 and annotating data 3D model 722 based upon reference 735 and reference 723, respectively.
  • Registration module 732 may also include an alignment refiner (similar to alignment refiner 202, FIG. 2) to ensure correct alignment of 3D model 722 with 3D model 734.
  • Planned surgical procedures 760 define surgical activities that the surgeon intends to perform on the patient. These activities are selected from the group including resection, retraction, trajectory determination, and surgical corridor, and may be defined singly or in combination. These surgical activities are thus defined
  • System 700 may include a 3D mechanical model 708 (similar to 3D mechanical model 208 of FIG. 2) that is used to model tissue in at least part of the surgical field and/or annotating data 3D model 722 such that tissue displacement resulting from planned surgical procedures 760 are predicted and portrayed correctly.
  • 3D mechanical model 708 similar to 3D mechanical model 208 of FIG. 2 that is used to model tissue in at least part of the surgical field and/or annotating data 3D model 722 such that tissue displacement resulting from planned surgical procedures 760 are predicted and portrayed correctly.
  • FIG. 8 shows system 100 of FIG. 1 further configured for use in positioning an implantable device during a surgical procedure based upon implantable device model 600 and associated reference 601, where reference 601 is defined by system 700 of FIG. 7 as the intended position of the implantable device within the patient.
  • system 100 invokes stereo image to 3D surface module 130 to generate 3D model 134 based upon stereo image stream 107.
  • Registration module 132 then aligns (co -registers) annotating data 3D model 122 and implantable device model 600 with 3D model 134 based upon references 123, 601, and 135.
  • System 100 may then refine this alignment using alignment refiner 202 (not shown in FIG. 8) as described above.
  • System 100 may also include an implant recognition module 802 that operates to detect the location of an implant within 3D image stream 107 based upon implanted device model 600.
  • Implant recognition module 802 may determine position and orientation of the implant even when only a part of the implant is visible within stereo image stream 107.
  • the implant includes identification and orientation markings that aid recognition and orientation by implant recognition module 802.
  • system 100 may generate graphical guides to aid the surgeon's positioning and insertion of the device.
  • system 100 provides guidance for more accurate positioning of the implantable devices as compared to using conventional placement means.
  • system 100 may provide warnings (visual and/or audio) to the surgeon where the device is approaching such critical areas.
  • Visual indications may include 3D graphical display of target and critical structures and a 3d graphical representation of the implantable device as it moves through the tissue during the procedure.
  • system 100 may record one or both of stereo image stream 107 and enhanced stereo image steam 107 throughout the surgical procedure/operation.
  • System 700 of FIG. 7 may they be used to perform a post-operative analysis of the surgical procedure based upon the recorded information and annotating data 108, wherein system 700 automatically refines alignment and uses 3D mechanical model 702 to allow for tissue deformation resulting from activities during the operational procedure.
  • Recorded data 190 thereby provide documentation of the surgical procedure for future reference and analysis alone or co-registered with other imaging data sets of annotating data 108, even where the annotating data 108 was not used during the surgery - such as when obtained post- surgery.
  • Recorded data 190 and annotating data 108 may be used for postoperative use of intraoperatively acquired functional information (including but not limited to motor, sensory, speech, memory, and other cognitive function). For example, recorded data 190 may be co-registered with other annotating data 108 acquired postoperatively.
  • Recorded data 190 e.g., hyperspectral stereo imaging data
  • annotating data 108 may be used to determine abnormal tissue physiological properties, including but limited to blood flow and tissue oxygenation, and to concentrations of selective and nonselective fluorescent optical biomarkers.
  • Recorded data 190 may be analyzed using system 700 together with subsequently acquired and potentially co-registered imaging data sets, either
  • FIG. 10A is a perspective view of 3D model 134 and implantable device model 650 illustrating a tip 1000 of a partially positioned device corresponding to model 650 in relation to a target area 1004 within brain 182.
  • FIG. 10B is a 2D
  • enhanced stereo image stream 103 i.e., the surgeon's view
  • 3D view resulting from enhanced stereo image stream 103 provides significantly more information that can be shown in the 2D example of FIG. 10B.
  • FIGs. 10A and 10B are best viewed together with the following description.
  • Target area 1004 is defined within planned surgical procedures 760 for example, and may include a desired entry point 1005 and/or path for the implantable device.
  • a view of models 134, 650 and target area 1004 is generated corresponding to the surgeon's view of surgical area 184 and added to enhanced stereo image stream 103 such that the surgeon sees, in 3D, a representation 1052 (e.g., a rendered image) of tip 652 and target area 1004 in relation to surgical field 184.
  • the surgeon may also view models 134, 650 and target area 1004 from other perspectives during the activity of inserting the implantable device.
  • Implantable device 1002 may have one or more visibly detectable markings to facilitate determining orientation of device 1002 by implant recognition module 802.
  • FIG. 11 is a perspective view of 3D model 134 and annotating data 3D model 222 generated from annotating data 108 based upon MRI tomography data.
  • model 650 and target area 1004 are not shown in the example of FIG. 11, but could be selectively added by the surgeon to show progress of tip 652 towards target area 1004.
  • a section of annotating data 3D model 222 is superimposed onto a cut-away of 3D model 134 to emphasize the inner structures of tissue below the surface of 3D model 134.
  • System 100 generates enhanced stereo image stream 103 to superimpose the hidden tissue structure onto the surgeon's view of surgical field 184. Since model 222 is aligned with 3d model 134 and corrected based upon 3D mechanical model 208, system 100 enhances the surgeons view with a very accurate 3D portrayal of the tissue normally hidden from the surgeon's view.
  • FIG. 9 shows one exemplary system 900 for surgical navigation with stereovision.
  • System 900 includes an optical surgical microscope 902 and a stereo image enhancer 904 that receives a stereo image stream 907 of a surgical field 984 from cameras 910 and 912.
  • System 900 operates similarly to system 100 of FIG. 1 to enhance a surgeon's view of a surgical field 984.
  • At least three reference points 918, 920, 922 may be attached to patient 980 for the duration of the procedure and the location of each reference point is defined as a location within a patent-centered coordinate system of patient 980.
  • tracker 129 determines locations of reference points 918, 920, 922 attached to patient 980 and reference points 924, 926, 928 attached to microscope 902.
  • Stereo image enhancer 904 uses these locations to determine a view reference (e.g., spatial relationship including relative position and orientation) between patient 980 and microscope 902 to define a reference of stereo image stream 907 captured by cameras 910, 912.
  • Stereo image enhancer 904 generates and sends an enhancing image stream 903 to image projectors 914, 916 that project the images from stereo image stream 903 into optical paths 932, 934, respectively.
  • optical path 932 includes a first beam splitter 942 that diverts a portion of light from surgical field 984 to camera 910 and a beam combiner 946 that adds light from image projector 914 to optical path 932.
  • optical path 934 includes a first beam splitter 944 that diverts a portion of light from surgical field 984 to camera 912 and a beam combiner 948 that adds light from image projector 916 to optical path 934.
  • a system designated A for generating a photographic image of a surgical field including a stereo image capture device for capturing a stereo image stream from two cameras imaging the surgical field; and a stereo viewer for displaying the stereo image stream to a surgeon.
  • the system also includes a stereo image to 3D surface extraction module, implemented as machine readable instructions stored in memory of a computer, that is capable of, when the instructions are executed by a processor of the computer, generating a first 3D model of the surgical field from the stereo image streams; a registration module, implemented as machine readable instructions stored in the memory, that is capable of, when the instructions are executed by the processor, co- registering annotating data with the first 3D model; and a stereo image enhancer, implemented as machine readable instructions stored in the memory, that is capable of, when the instructions are executed by the processor, graphically overlaying at least part of the annotating data onto the stereo image stream to form an enhanced stereo image stream for display by the stereo viewer, wherein the enhanced stereo stream enhances the surgeon's perception of the surgical field.
  • a system designated AA including the system designated A, wherein the registration module aligns the annotating data with the first 3D model.
  • a system designated AB including the system designated A or AA further including a 3D model generator for generating a second 3D model from the annotating data, wherein the registration module co-registers the second 3D model with the first 3D model; and an image generator configured to generate a graphical image of the second 3D model based upon the surgical field.
  • a system designated AC including the system designated A, AA or AB wherein the annotating data comprises one of a preoperative, an intraoperative and a postoperative radiological study selected from the group consisting of magnetic resonance imaging (MRI), functional magnetic resonance imaging (fMRI), x-ray computed tomography (CT), Single-photon emission computed tomography (SPECT), and positron emission tomography (PET).
  • MRI magnetic resonance imaging
  • fMRI functional magnetic resonance imaging
  • CT x-ray computed tomography
  • SPECT Single-photon emission computed tomography
  • PET positron emission tomography
  • a system designated AD including the system designated AB or AC, the image generator configured to generate a cross sectional view of the second 3D model based upon an indicated surgical plane within the surgical field.
  • a system designated AE including the system designated A, AA, AB, AC, or AD wherein the annotating data further comprises physiological study data selected from the group including EEG, evoked potentials, and magnetoencephalography (MEG).
  • EEG EEG
  • evoked potentials EEG
  • MEG magnetoencephalography
  • a system designated AF including the system designated A, AA, AB, AC, AD or AE, the registration module further including an alignment refiner configured to adjust the registration of the annotating data with the 3D model based upon matching of features within the 3D model and features within the annotating data.
  • a system designated AF including the system designated A, AA, AB, AC, AD, AE, or AF, the registration module further including a deformation modeler configured to deform the annotating data based upon a determined tissue deformation within the surgical field.
  • a system designated AG including the system designated AF, the tissue deformation resulting from one or more of skull removal, retraction, and incision.
  • a system designated AH including the system designated A, AA, AB, AC, AD, AE, AF, or AG and further including a first controllable filter coupled with a first of the two cameras to filter light entering the first of the two cameras from the surgical field; a first controllable light source for illuminating the surgical field with light of a determined fluorescent stimulus wavelength; and a fluorescence depth processor, implemented as machine readable instructions stored in the memory, that is capable of, when the instructions are executed by the processor, generating a 3D fluorescence model; wherein the stereo image enhancer graphically overlays at least a portion of the 3D fluorescence model onto the enhanced stereo image stream.
  • a system designated AK including the system designated AH, further including a second controllable filter coupled with a second of the two cameras to filter light entering the second of the two cameras from the surgical field; and a hyperspectral image processor, implemented as machine readable instructions stored in the memory, that is capable of, when the instructions are executed by the processor, controlling the first and second filters to capture a stereo hyperspectral image stream and generate a 3D hyperspectral model; wherein the stereo image enhancer graphically overlays at least a portion of the 3D hyperspectral model onto the enhanced stereo image stream.
  • a system designated AL including the system designated A, AA, AB, AC, AD, AE, AF, or AG, further including a first and a second hyperspectral imaging camera for capturing a stereo hyperspectral image stream of the surgical field; and a hyperspectral image processor, implemented as machine readable instructions stored in the memory, that is capable of, when the instructions are executed by the processor, generating a 3D hyperspectral model from the stereo hyperspectral image stream; wherein the stereo image enhancer graphically overlays at least a portion of the 3D hyperspectral model onto the enhanced stereo image stream.
  • a system designated B for graphically representing and displaying a first and second annotating data in relation to a surgical field including an interactive display device; a registration module, implemented as machine readable instruction stored in a memory of a computer, that is capable of, when the instructions are executed by a processor of the computer, co-registering the first and second annotating data; and a stereo image enhancer, implemented as machine readable instruction stored in the memory, that is capable of, when the instructions are executed by the processor, graphically overlaying at least part of the annotating data onto the stereo image stream to form an enhanced stereo image stream for display by the stereo viewer, wherein the enhanced stereo stream enhances the surgeon' s perception of the surgical field.
  • a registration module implemented as machine readable instruction stored in a memory of a computer, that is capable of, when the instructions are executed by a processor of the computer, co-registering the first and second annotating data
  • a stereo image enhancer implemented as machine readable instruction stored in the memory, that is capable of, when the instructions are executed by the processor, graphically
  • a system designated BA including the system designated B, further configured to review a recorded surgical procedure that is co-registered with post operatively acquired imaging and/or psychological data.
  • a method designated C for surgical navigation with stereo vision including capturing a stereo image pair of a surgical field; generating a 3D surface model from the stereo image pair; registering preoperatively captured annotating data with the 3D surface model; generating an enhanced stereo image based upon the stereo image pair and the annotating data; wherein the annotating data is graphically overlaid onto the stereo image pair.
  • a method designated CA including the method designated C, the step of registering further including generating a 3D model based upon the annotating data; and registering the 3D model to the 3D surface model.
  • a method designated CB including the method designated C or CA, further including adjusting the annotating data based upon matched features of the 3D surface model and features of the annotating data.
  • a method designated CC including the method designated C, CA, or
  • the step of adjusting comprising one or more of shifting, rotating, warping, and scaling the annotating data.
  • a method designated CD including the method designated C or CA, , the step of adjusting including generating a mechanical model of the tissue within the surgical field to determine the adjustment of the enhanced data.
  • a method designated CE including the method designated C, CA, CB, or CC, further including recording the stereo image pair.
  • a method designated CF including the method designated C, CA, CB, or CC, further including recording the enhanced stereo image pair.
  • a method designated CG including the method designated C, CA, CB, or CC, wherein the annotating data comprises one of a preoperative, an intraoperative and a postoperative radiological study selected from the group consisting of magnetic resonance imaging (MRI), functional magnetic resonance imaging (fMRI), x-ray computed tomography (CT), Single-photon emission computed tomography (SPECT), and positron emission tomography (PET).
  • MRI magnetic resonance imaging
  • fMRI functional magnetic resonance imaging
  • CT x-ray computed tomography
  • SPECT Single-photon emission computed tomography
  • PET positron emission tomography
  • a method designated CH including the method designated C, CA, CB,
  • annotating data further comprises a physiological study selected from the group including EEG, evoked potentials, and
  • MEG magnetoencephalography

Landscapes

  • Engineering & Computer Science (AREA)
  • Health & Medical Sciences (AREA)
  • Surgery (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
  • General Health & Medical Sciences (AREA)
  • Pathology (AREA)
  • Biomedical Technology (AREA)
  • Heart & Thoracic Surgery (AREA)
  • Medical Informatics (AREA)
  • Molecular Biology (AREA)
  • Animal Behavior & Ethology (AREA)
  • Oral & Maxillofacial Surgery (AREA)
  • Public Health (AREA)
  • Veterinary Medicine (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Gynecology & Obstetrics (AREA)
  • Radiology & Medical Imaging (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Human Computer Interaction (AREA)
  • Computer Graphics (AREA)
  • Computer Hardware Design (AREA)
  • General Engineering & Computer Science (AREA)
  • Software Systems (AREA)
  • Magnetic Resonance Imaging Apparatus (AREA)
  • Measuring And Recording Apparatus For Diagnosis (AREA)

Abstract

La présente invention concerne un système de guidage chirurgical comprenant deux caméras permettant d'obtenir des flux d'images stéréo d'un champ chirurgical ; et une visionneuse stéréo. Le système comporte un module d'extraction de surface 3D qui génère un premier modèle 3D du champ chirurgical à partir des flux d'images stéréo ; un module d'enregistrement servant à enregistrer des données d'annotation conjointement avec le premier modèle 3D ; et un rehausseur d'images stéréo permettant une superposition graphique d'au moins une partie des données d'annotation sur le flux d'images stéréo pour former un flux d'images stéréo rehaussé à des fins d'affichage, le flux stéréo rehaussé améliorant la perception d'un chirurgien du champ chirurgical. Le module d'enregistrement comporte un logiciel d'affinement d'alignement permettant d'ajuster l'enregistrement des données d'annotation avec le modèle 3D sur la base d'une mise en correspondance de caractéristiques du modèle 3D et de caractéristiques des données d'annotation ; et, dans un mode de réalisation, un logiciel de modélisation de déformation servant à déformer les données d'annotation sur la base d'une déformation tissulaire déterminée.
PCT/US2015/033672 2008-05-22 2015-06-02 Navigation chirurgicale avec stéréovision et procédés associés WO2015187620A1 (fr)

Priority Applications (4)

Application Number Priority Date Filing Date Title
US15/367,243 US10568535B2 (en) 2008-05-22 2016-12-02 Surgical navigation with stereovision and associated methods
US16/785,882 US11129562B2 (en) 2008-05-22 2020-02-10 Surgical navigation with stereovision and associated methods
US17/486,839 US11690558B2 (en) 2011-01-21 2021-09-27 Surgical navigation with stereovision and associated methods
US18/204,091 US12114988B2 (en) 2008-05-22 2023-05-31 Surgical navigation with stereovision and associated methods

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US201462006786P 2014-06-02 2014-06-02
US62/006,786 2014-06-02

Related Parent Applications (9)

Application Number Title Priority Date Filing Date
PCT/US2009/045082 Continuation-In-Part WO2009143491A2 (fr) 2008-05-22 2009-05-22 Système et procédé pour calibrer une opération chirurgicale guidée par image
US12/994,044 Continuation-In-Part US9052384B2 (en) 2008-05-22 2009-05-22 System and method for calibration for image-guided surgery
US99404411A Continuation-In-Part 2008-05-22 2011-01-21
US14/345,029 Continuation-In-Part US9655545B2 (en) 2011-09-15 2012-09-17 Apparatus for measuring in-vivo mechanical properties of biological tissues
PCT/US2012/055755 Continuation-In-Part WO2013040555A2 (fr) 2008-05-22 2012-09-17 Appareil permettant de mesurer les propriétés mécaniques in vivo de tissus biologiques
PCT/US2013/020352 Continuation-In-Part WO2013103870A1 (fr) 2008-05-22 2013-01-04 Procédé et appareil pour étalonnage de système de mappage de surface tridimensionnel stéréo-optique
PCT/US2013/022266 Continuation-In-Part WO2013109966A1 (fr) 2008-05-22 2013-01-18 Procédé et appareil pour imagerie quantitative hyperspectrale par fluorescence et réflectance pour guidage chirurgical
US14/373,443 Continuation-In-Part US20140378843A1 (en) 2012-01-20 2013-01-18 Method And Apparatus For Quantitative Hyperspectral Fluorescence And Reflectance Imaging For Surgical Guidance
PCT/US2013/024400 Continuation-In-Part WO2013116694A1 (fr) 2008-05-22 2013-02-01 Procédé et appareil de détermination d'un décalage des tumeurs durant une opération chirurgicale à l'aide d'un système de cartographie de surface stéréo-optique en trois dimensions

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US15/367,243 Continuation-In-Part US10568535B2 (en) 2008-05-22 2016-12-02 Surgical navigation with stereovision and associated methods

Publications (1)

Publication Number Publication Date
WO2015187620A1 true WO2015187620A1 (fr) 2015-12-10

Family

ID=54767260

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/US2015/033672 WO2015187620A1 (fr) 2008-05-22 2015-06-02 Navigation chirurgicale avec stéréovision et procédés associés

Country Status (1)

Country Link
WO (1) WO2015187620A1 (fr)

Cited By (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9789338B1 (en) * 2016-04-13 2017-10-17 Vision Rt Ltd. Patient monitoring system
US10499997B2 (en) 2017-01-03 2019-12-10 Mako Surgical Corp. Systems and methods for surgical navigation
US10568535B2 (en) 2008-05-22 2020-02-25 The Trustees Of Dartmouth College Surgical navigation with stereovision and associated methods
US11510600B2 (en) 2012-01-04 2022-11-29 The Trustees Of Dartmouth College Method and apparatus for quantitative and depth resolved hyperspectral fluorescence and reflectance imaging for surgical guidance
US11564639B2 (en) 2013-02-13 2023-01-31 The Trustees Of Dartmouth College Method and apparatus for medical imaging using differencing of multiple fluorophores
US11937951B2 (en) 2013-02-13 2024-03-26 The Trustees Of Dartmouth College Method and apparatus for medical imaging using differencing of multiple fluorophores

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20070236514A1 (en) * 2006-03-29 2007-10-11 Bracco Imaging Spa Methods and Apparatuses for Stereoscopic Image Guided Surgical Navigation
WO2013109966A1 (fr) * 2012-01-20 2013-07-25 The Trustees Of Dartmouth College Procédé et appareil pour imagerie quantitative hyperspectrale par fluorescence et réflectance pour guidage chirurgical

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20070236514A1 (en) * 2006-03-29 2007-10-11 Bracco Imaging Spa Methods and Apparatuses for Stereoscopic Image Guided Surgical Navigation
WO2013109966A1 (fr) * 2012-01-20 2013-07-25 The Trustees Of Dartmouth College Procédé et appareil pour imagerie quantitative hyperspectrale par fluorescence et réflectance pour guidage chirurgical

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
NAJIB ET AL.: "Transcranial Brain Stimulation: Clinical Applications and Future Directions.", NEUROSURG CLIN. N. AM., April 2011 (2011-04-01), Retrieved from the Internet <URL:http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3547606/pdf/nihms263566.pdf> [retrieved on 20150731] *
VIGNERON ET AL.: "2D XFEM-Based Modeling of Retraction and Successive Resections for Preoperative Image Update.", COMPUT AIDED SURG., 2009, Retrieved from the Internet <URL:http://www.ncbi.nim.nih.gov/pmc/articies/PMC3843511/pdf/nihms-524531.pdf> [retrieved on 20150731] *

Cited By (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10568535B2 (en) 2008-05-22 2020-02-25 The Trustees Of Dartmouth College Surgical navigation with stereovision and associated methods
US11129562B2 (en) 2008-05-22 2021-09-28 The Trustees Of Dartmouth College Surgical navigation with stereovision and associated methods
US12114988B2 (en) 2008-05-22 2024-10-15 The Trustees Of Dartmouth College Surgical navigation with stereovision and associated methods
US11510600B2 (en) 2012-01-04 2022-11-29 The Trustees Of Dartmouth College Method and apparatus for quantitative and depth resolved hyperspectral fluorescence and reflectance imaging for surgical guidance
US11857317B2 (en) 2012-01-04 2024-01-02 The Trustees Of Dartmouth College Method and apparatus for quantitative and depth resolved hyperspectral fluorescence and reflectance imaging for surgical guidance
US11564639B2 (en) 2013-02-13 2023-01-31 The Trustees Of Dartmouth College Method and apparatus for medical imaging using differencing of multiple fluorophores
US11937951B2 (en) 2013-02-13 2024-03-26 The Trustees Of Dartmouth College Method and apparatus for medical imaging using differencing of multiple fluorophores
US9789338B1 (en) * 2016-04-13 2017-10-17 Vision Rt Ltd. Patient monitoring system
US10499997B2 (en) 2017-01-03 2019-12-10 Mako Surgical Corp. Systems and methods for surgical navigation
US11707330B2 (en) 2017-01-03 2023-07-25 Mako Surgical Corp. Systems and methods for surgical navigation

Similar Documents

Publication Publication Date Title
US12114988B2 (en) Surgical navigation with stereovision and associated methods
US11857317B2 (en) Method and apparatus for quantitative and depth resolved hyperspectral fluorescence and reflectance imaging for surgical guidance
US11656448B2 (en) Method and apparatus for quantitative hyperspectral fluorescence and reflectance imaging for surgical guidance
WO2015187620A1 (fr) Navigation chirurgicale avec stéréovision et procédés associés
CN108430306B (zh) 用于使用激光散斑对比度成像快速检查脉管系统和微粒流的系统和方法
CN105263398B (zh) 手术成像系统
CA2959215C (fr) Imagerie multispectrale synchronisee
WO2016048911A1 (fr) Système d&#39;imagerie hyperspectrale
WO2015023990A1 (fr) Procédé et appareil pour imagerie hyperspectrale par fluorescence et réflectance quantitative et résolue en profondeur pour le guidage chirurgical
KR101531620B1 (ko) 뇌의 실영상에 ｎｂｓ 기능 데이터를 오버레이하기 위한 방법 및 시스템
JP6972049B2 (ja) 脈管叢構造の弾性マッピングを用いた画像処理方法および画像処理装置
KR20130108320A (ko) 관련 애플리케이션들에 대한 일치화된 피하 해부구조 참조의 시각화
US20230386153A1 (en) Systems for medical image visualization
WO2023021450A1 (fr) Dispositif d&#39;affichage stéréoscopique et loupe numérique pour dispositif d&#39;affichage proche de l&#39;œil à réalité augmentée
CN110720985A (zh) 一种多模式引导的手术导航方法和系统
US11690558B2 (en) Surgical navigation with stereovision and associated methods
KR101977650B1 (ko) 증강 현실을 이용한 의료 영상 처리 장치 및 의료 영상 처리 방법
Zagzoog et al. Perspective review on applications of optics in skull base surgery
Le OPTICAL NAVIGATION TECHNIQUES FOR MINIMALLY INVASIVE ROBOTIC SURGERIES
Árnadóttir et al. 3D printing to advance neurosurgery planning
CN118369732A (zh) 解剖扫描、瞄准和可视化
Martinez Escobar Human factors and performance considerations of visual spatial skills in medical context tasks
CA2892298A1 (fr) Systeme d&#39;environnement chirurgical a realite augmentee

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 15803748

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 15803748

Country of ref document: EP

Kind code of ref document: A1