WO2013093761A3 - Overlay and motion compensation of structures from volumetric modalities onto video of an uncalibrated endoscope - Google Patents

Overlay and motion compensation of structures from volumetric modalities onto video of an uncalibrated endoscope Download PDF

Info

Publication number
WO2013093761A3
WO2013093761A3 PCT/IB2012/057393 IB2012057393W WO2013093761A3 WO 2013093761 A3 WO2013093761 A3 WO 2013093761A3 IB 2012057393 W IB2012057393 W IB 2012057393W WO 2013093761 A3 WO2013093761 A3 WO 2013093761A3
Authority
WO
WIPO (PCT)
Prior art keywords
image
endoscope
structures
overlay
projection
Prior art date
Application number
PCT/IB2012/057393
Other languages
French (fr)
Other versions
WO2013093761A2 (en
Inventor
Aleksandra Popovic
Haytham Elhawary
Original Assignee
Koninklijke Philips N.V.
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Koninklijke Philips N.V. filed Critical Koninklijke Philips N.V.
Priority to EP12823006.7A priority Critical patent/EP2793680A2/en
Priority to CN201280063699.5A priority patent/CN104010560A/en
Priority to JP2014548289A priority patent/JP2015506188A/en
Priority to US14/363,840 priority patent/US20140347353A1/en
Publication of WO2013093761A2 publication Critical patent/WO2013093761A2/en
Publication of WO2013093761A3 publication Critical patent/WO2013093761A3/en

Links

Classifications

    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
    • A61B1/00002—Operational features of endoscopes
    • A61B1/00004—Operational features of endoscopes characterised by electronic signal processing
    • A61B1/00009—Operational features of endoscopes characterised by electronic signal processing of image signals during a use of endoscope
    • A61B1/000094—Operational features of endoscopes characterised by electronic signal processing of image signals during a use of endoscope extracting biological structures
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
    • A61B1/00002—Operational features of endoscopes
    • A61B1/00043—Operational features of endoscopes provided with output arrangements
    • A61B1/00045—Display arrangement
    • A61B1/0005—Display arrangement combining images e.g. side-by-side, superimposed or tiled
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
    • A61B1/313—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor for introducing through surgical openings, e.g. laparoscopes
    • A61B1/3137—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor for introducing through surgical openings, e.g. laparoscopes for examination of the interior of blood vessels
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T11/00—2D [Two Dimensional] image generation
    • G06T11/003—Reconstruction from projections, e.g. tomography
    • G06T11/006—Inverse problem, transformation from projection-space into object-space, e.g. transform methods, back-projection, algebraic methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T11/00—2D [Two Dimensional] image generation
    • G06T11/60—Editing figures and text; Combining figures or text
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/30—Determination of transform parameters for the alignment of images, i.e. image registration
    • G06T7/33—Determination of transform parameters for the alignment of images, i.e. image registration using feature-based methods
    • G06T7/344—Determination of transform parameters for the alignment of images, i.e. image registration using feature-based methods involving models
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B90/00—Instruments, implements or accessories specially adapted for surgery or diagnosis and not covered by any of the groups A61B1/00 - A61B50/00, e.g. for luxation treatment or for protecting wound edges
    • A61B90/36—Image-producing devices or illumination devices not otherwise provided for
    • A61B2090/364—Correlation of different images or relation of image positions in respect to the body
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B34/00—Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
    • A61B34/30—Surgical robots
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2200/00—Indexing scheme for image data processing or generation, in general
    • G06T2200/32—Indexing scheme for image data processing or generation, in general involving image mosaicing
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10068—Endoscopic image
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10072—Tomographic images
    • G06T2207/10081—Computed x-ray tomography [CT]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20072—Graph-based image processing
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30004—Biomedical image processing
    • G06T2207/30048—Heart; Cardiac
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30004—Biomedical image processing
    • G06T2207/30101—Blood vessel; Artery; Vein; Vascular
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2210/00—Indexing scheme for image generation or computer graphics
    • G06T2210/41—Medical

Landscapes

  • Health & Medical Sciences (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Engineering & Computer Science (AREA)
  • Surgery (AREA)
  • Physics & Mathematics (AREA)
  • Molecular Biology (AREA)
  • Veterinary Medicine (AREA)
  • Optics & Photonics (AREA)
  • Pathology (AREA)
  • Radiology & Medical Imaging (AREA)
  • Biophysics (AREA)
  • Biomedical Technology (AREA)
  • Heart & Thoracic Surgery (AREA)
  • Medical Informatics (AREA)
  • Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
  • Animal Behavior & Ethology (AREA)
  • General Health & Medical Sciences (AREA)
  • Public Health (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Signal Processing (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Algebra (AREA)
  • Mathematical Analysis (AREA)
  • Mathematical Optimization (AREA)
  • Mathematical Physics (AREA)
  • Pure & Applied Mathematics (AREA)
  • Endoscopes (AREA)
  • Image Processing (AREA)
  • Apparatus For Radiation Diagnosis (AREA)

Abstract

A method, system, and program product are provided for overlaying structures from volumetric modalities onto video of an uncalibrat- ed endoscope. The method comprises: determining correspondence (310) of a plurality of point pairs between a structure on a 3D preoperative image and the structure on a 2D endoscope video image; deriving a projection matrix for translation (320) from the 3D image to the 2D image using assumed camera parameters and the corresponding point pairs; rendering a 2D projection (330) of the structure from the 3D image onto the 2D endoscope image using the derived projection matrix; determining a homography matrix; and warping the rendered structure projection (340), using the homography matrix.
PCT/IB2012/057393 2011-12-21 2012-12-17 Overlay and motion compensation of structures from volumetric modalities onto video of an uncalibrated endoscope WO2013093761A2 (en)

Priority Applications (4)

Application Number Priority Date Filing Date Title
EP12823006.7A EP2793680A2 (en) 2011-12-21 2012-12-17 Overlay and motion compensation of structures from volumetric modalities onto video of an uncalibrated endoscope
CN201280063699.5A CN104010560A (en) 2011-12-21 2012-12-17 Overlay and motion compensation of structures from volumetric modalities onto video of uncalibrated endoscope
JP2014548289A JP2015506188A (en) 2011-12-21 2012-12-17 Video overlay and motion compensation of uncalibrated endoscopes of structures from volumetric modalities
US14/363,840 US20140347353A1 (en) 2011-12-21 2012-12-17 Overlay and motion compensation of structures from volumetric modalities onto video of an uncalibrated endoscope

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US201161578427P 2011-12-21 2011-12-21
US61/578,427 2011-12-21

Publications (2)

Publication Number Publication Date
WO2013093761A2 WO2013093761A2 (en) 2013-06-27
WO2013093761A3 true WO2013093761A3 (en) 2013-08-08

Family

ID=47678905

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/IB2012/057393 WO2013093761A2 (en) 2011-12-21 2012-12-17 Overlay and motion compensation of structures from volumetric modalities onto video of an uncalibrated endoscope

Country Status (5)

Country Link
US (1) US20140347353A1 (en)
EP (1) EP2793680A2 (en)
JP (1) JP2015506188A (en)
CN (1) CN104010560A (en)
WO (1) WO2013093761A2 (en)

Families Citing this family (29)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11399900B2 (en) * 2012-06-21 2022-08-02 Globus Medical, Inc. Robotic systems providing co-registration using natural fiducials and related methods
CN104427927B (en) 2012-06-28 2018-04-13 皇家飞利浦有限公司 For visualization of blood vessels and the navigation guided by fibre optical sensor of monitoring
JP6049202B2 (en) * 2013-10-25 2016-12-21 富士フイルム株式会社 Image processing apparatus, method, and program
CN105934216B (en) 2014-01-24 2019-09-17 皇家飞利浦有限公司 Robot guides system, control unit and device
WO2015118423A1 (en) 2014-02-04 2015-08-13 Koninklijke Philips N.V. Visualization of depth and position of blood vessels and robot guided visualization of blood vessel cross section
WO2015121765A1 (en) * 2014-02-12 2015-08-20 Koninklijke Philips N.V. Robotic control of surgical instrument visibility
EP3169264A1 (en) 2014-07-15 2017-05-24 Koninklijke Philips N.V. Image integration and robotic endoscope control in x-ray suite
KR20170110089A (en) * 2015-01-30 2017-10-10 톰슨 라이센싱 Method and apparatus for generating an initial superpixel label map for an image
US10013808B2 (en) 2015-02-03 2018-07-03 Globus Medical, Inc. Surgeon head-mounted display apparatuses
WO2018002347A1 (en) * 2016-06-30 2018-01-04 Koninklijke Philips N.V. Registering tomographic imaging and endoscopic imaging
CN110049742B (en) 2016-12-07 2023-01-03 皇家飞利浦有限公司 Image-guided motion scaling for robot control
US10417738B2 (en) 2017-01-05 2019-09-17 Perfect Corp. System and method for displaying graphical effects based on determined facial positions
US20190254753A1 (en) 2018-02-19 2019-08-22 Globus Medical, Inc. Augmented reality navigation systems for use with robotic surgical systems and methods of their use
JP7146949B2 (en) * 2018-05-31 2022-10-04 オーリス ヘルス インコーポレイテッド Image-based airway analysis and mapping
US11026585B2 (en) 2018-06-05 2021-06-08 Synaptive Medical Inc. System and method for intraoperative video processing
EP3618005A1 (en) 2018-08-30 2020-03-04 Koninklijke Philips N.V. Image processing system and method
GB2579843A (en) * 2018-12-18 2020-07-08 Continental Automotive Gmbh Method and apparatus for calibrating the extrinsic parameter of an image sensor
US11190803B2 (en) * 2019-01-18 2021-11-30 Sony Group Corporation Point cloud coding using homography transform
US11992373B2 (en) 2019-12-10 2024-05-28 Globus Medical, Inc Augmented reality headset with varied opacity for navigated robotic surgery
US11464581B2 (en) 2020-01-28 2022-10-11 Globus Medical, Inc. Pose measurement chaining for extended reality surgical navigation in visible and near infrared spectrums
US11382699B2 (en) 2020-02-10 2022-07-12 Globus Medical Inc. Extended reality visualization of optical tool tracking volume for computer assisted navigation in surgery
US11207150B2 (en) 2020-02-19 2021-12-28 Globus Medical, Inc. Displaying a virtual model of a planned instrument attachment to ensure correct selection of physical instrument attachment
US11607277B2 (en) 2020-04-29 2023-03-21 Globus Medical, Inc. Registration of surgical tool with reference array tracked by cameras of an extended reality headset for assisted navigation during surgery
US11382700B2 (en) 2020-05-08 2022-07-12 Globus Medical Inc. Extended reality headset tool tracking and control
US11153555B1 (en) 2020-05-08 2021-10-19 Globus Medical Inc. Extended reality headset camera system for computer assisted navigation in surgery
US11510750B2 (en) 2020-05-08 2022-11-29 Globus Medical, Inc. Leveraging two-dimensional digital imaging and communication in medicine imagery in three-dimensional extended reality applications
US11737831B2 (en) 2020-09-02 2023-08-29 Globus Medical Inc. Surgical object tracking template generation for computer assisted navigation during surgical procedure
US20240289947A1 (en) * 2023-02-27 2024-08-29 Cilag Gmbh International False color overlay heatmaps with adjustable sensitivity settings
WO2024180445A1 (en) * 2023-02-27 2024-09-06 Cilag Gmbh International Adaptive overlay stabilization of false color overlay heatmaps

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20090324062A1 (en) * 2008-06-25 2009-12-31 Samsung Electronics Co., Ltd. Image processing method
US20110282151A1 (en) * 2008-10-20 2011-11-17 Koninklijke Philips Electronics N.V. Image-based localization method and system
WO2012156873A1 (en) * 2011-05-18 2012-11-22 Koninklijke Philips Electronics N.V. Endoscope segmentation correction for 3d-2d image overlay

Family Cites Families (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP3078085B2 (en) * 1991-03-26 2000-08-21 オリンパス光学工業株式会社 Image processing apparatus and image processing method
US6859549B1 (en) * 2000-06-07 2005-02-22 Nec Laboratories America, Inc. Method for recovering 3D scene structure and camera motion from points, lines and/or directly from the image intensities
SE519884C2 (en) * 2001-02-02 2003-04-22 Scalado Ab Method for zooming and producing a zoomable image
JP3975736B2 (en) * 2001-12-07 2007-09-12 ソニー株式会社 Image processing apparatus, image processing method, storage medium, and computer program
AU2003223085A1 (en) * 2002-04-17 2003-10-27 Super Dimension Ltd. Endoscope structures and techniques for navigating to a target in branched structure
WO2004114224A1 (en) * 2003-06-20 2004-12-29 Nippon Telegraph And Telephone Corporation Virtual visual point image generating method and 3-d image display method and device
US7885011B1 (en) * 2009-08-19 2011-02-08 General Electric Company Objective for optical imaging systems
US8675926B2 (en) * 2010-06-08 2014-03-18 Microsoft Corporation Distinguishing live faces from flat surfaces
KR101677561B1 (en) * 2010-12-08 2016-11-18 한국전자통신연구원 Image registration device and image registration method thereof

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20090324062A1 (en) * 2008-06-25 2009-12-31 Samsung Electronics Co., Ltd. Image processing method
US20110282151A1 (en) * 2008-10-20 2011-11-17 Koninklijke Philips Electronics N.V. Image-based localization method and system
WO2012156873A1 (en) * 2011-05-18 2012-11-22 Koninklijke Philips Electronics N.V. Endoscope segmentation correction for 3d-2d image overlay

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
XIANWANG WANG ET AL: "Endoscopic Video Texture Mapping on Pre-Built 3-D Anatomical Objects Without Camera Tracking", IEEE TRANSACTIONS ON MEDICAL IMAGING, IEEE SERVICE CENTER, PISCATAWAY, NJ, US, vol. 29, no. 6, 1 June 2010 (2010-06-01), pages 1213 - 1223, XP011294913, ISSN: 0278-0062 *

Also Published As

Publication number Publication date
CN104010560A (en) 2014-08-27
US20140347353A1 (en) 2014-11-27
EP2793680A2 (en) 2014-10-29
JP2015506188A (en) 2015-03-02
WO2013093761A2 (en) 2013-06-27

Similar Documents

Publication Publication Date Title
WO2013093761A3 (en) Overlay and motion compensation of structures from volumetric modalities onto video of an uncalibrated endoscope
EP2153409A4 (en) Camera pose estimation apparatus and method for augmented reality imaging
GB2483213A (en) 3D image generation
WO2012032412A3 (en) Stereoscopic (3d) panorama creation on handheld device
WO2011143073A3 (en) Method, system, and apparatus for pressure image registration
WO2010101434A3 (en) Apparatus and method of generating panoramic image and computer-readable recording medium storing program for executing the method
WO2014079879A3 (en) Method of 3d reconstruction and 3d panoramic mosaicing of a scene
WO2007149322A3 (en) System and method for displaying images
FR2976107B1 (en) METHOD FOR LOCATING A CAMERA AND 3D RECONSTRUCTION IN A PARTIALLY KNOWN ENVIRONMENT
WO2011029209A3 (en) Method and apparatus for generating and processing depth-enhanced images
WO2013119408A3 (en) Method and system for automatic 3-d image creation
WO2007084267A3 (en) A method for rectifying stereoscopic display systems
EP2512145A3 (en) Full reference system for predicting subjective quality of three-dimensional video
WO2006127713A3 (en) A fast 2d-3d image registration method with application to continuously guided endoscopy
GB0910117D0 (en) Method and system for filming
WO2009157708A3 (en) Method and apparatus for processing 3d video image
EP2725548A3 (en) Image processing apparatus and method
MX2011011799A (en) Systems and methods for the autonomous production of videos from multi-sensored data.
WO2011031026A3 (en) 3d avatar service providing system and method using background image
WO2009049750A3 (en) Method for calibrating an assembly using at least one omnidirectional camera and an optical display unit
EP2688286A3 (en) Robust two dimensional panorama generation using light field camera capture
WO2013029722A3 (en) Method for representing surroundings
MX2014007075A (en) System for filming a video movie.
EP3997864A4 (en) Real-time image generation in moving scenes
WO2017058424A8 (en) Range imaging

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 12823006

Country of ref document: EP

Kind code of ref document: A2

WWE Wipo information: entry into national phase

Ref document number: 14363840

Country of ref document: US

ENP Entry into the national phase

Ref document number: 2014548289

Country of ref document: JP

Kind code of ref document: A

REEP Request for entry into the european phase

Ref document number: 2012823006

Country of ref document: EP

WWE Wipo information: entry into national phase

Ref document number: 2012823006

Country of ref document: EP