USRE49930E1 - Methods and systems for computer-aided surgery using intra-operative video acquired by a free moving camera - Google Patents
Methods and systems for computer-aided surgery using intra-operative video acquired by a free moving camera Download PDFInfo
- Publication number
- USRE49930E1 USRE49930E1 US17/548,202 US201617548202A USRE49930E US RE49930 E1 USRE49930 E1 US RE49930E1 US 201617548202 A US201617548202 A US 201617548202A US RE49930 E USRE49930 E US RE49930E
- Authority
- US
- United States
- Prior art keywords
- camera
- coordinates
- points
- image
- video
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active, expires
Links
- 238000000034 method Methods 0.000 title claims abstract description 148
- 238000001356 surgical procedure Methods 0.000 title claims abstract description 45
- 210000003484 anatomy Anatomy 0.000 claims abstract description 112
- 239000003550 marker Substances 0.000 claims abstract description 111
- 238000012545 processing Methods 0.000 claims abstract description 77
- 230000000007 visual effect Effects 0.000 claims abstract description 55
- 210000001519 tissue Anatomy 0.000 claims abstract description 16
- 238000013519 translation Methods 0.000 claims abstract description 16
- 238000003745 diagnosis Methods 0.000 claims abstract description 13
- 239000000523 sample Substances 0.000 claims description 85
- 238000005259 measurement Methods 0.000 claims description 30
- 230000009466 transformation Effects 0.000 claims description 27
- 230000006870 function Effects 0.000 claims description 22
- 210000001264 anterior cruciate ligament Anatomy 0.000 claims description 19
- 238000003780 insertion Methods 0.000 claims description 19
- 230000037431 insertion Effects 0.000 claims description 19
- 230000003190 augmentative effect Effects 0.000 claims description 15
- 239000012636 effector Substances 0.000 claims description 11
- 230000033001 locomotion Effects 0.000 claims description 8
- 239000000203 mixture Substances 0.000 claims description 8
- 230000000399 orthopedic effect Effects 0.000 claims description 7
- 238000002271 resection Methods 0.000 claims description 7
- 238000013150 knee replacement Methods 0.000 claims description 6
- 230000009471 action Effects 0.000 claims description 5
- 230000007547 defect Effects 0.000 claims description 5
- 210000002967 posterior cruciate ligament Anatomy 0.000 claims description 5
- 238000011541 total hip replacement Methods 0.000 claims description 5
- 210000000845 cartilage Anatomy 0.000 claims description 4
- 238000005286 illumination Methods 0.000 claims description 4
- 238000002355 open surgical procedure Methods 0.000 claims description 4
- 239000003826 tablet Substances 0.000 claims description 4
- 238000009472 formulation Methods 0.000 claims description 3
- 230000035515 penetration Effects 0.000 claims description 3
- 210000000323 shoulder joint Anatomy 0.000 claims description 3
- 239000004053 dental implant Substances 0.000 claims description 2
- 238000004519 manufacturing process Methods 0.000 claims description 2
- 239000002184 metal Substances 0.000 claims description 2
- 229910052751 metal Inorganic materials 0.000 claims description 2
- 238000004164 analytical calibration Methods 0.000 claims 4
- 238000002224 dissection Methods 0.000 claims 1
- 210000000988 bone and bone Anatomy 0.000 abstract description 37
- 238000001514 detection method Methods 0.000 abstract description 18
- 210000000515 tooth Anatomy 0.000 abstract description 3
- 238000003860 storage Methods 0.000 description 21
- 230000008569 process Effects 0.000 description 16
- 210000000689 upper leg Anatomy 0.000 description 11
- 238000013459 approach Methods 0.000 description 10
- 238000013461 design Methods 0.000 description 9
- 239000007943 implant Substances 0.000 description 9
- 210000000056 organ Anatomy 0.000 description 8
- 238000004891 communication Methods 0.000 description 7
- 210000003127 knee Anatomy 0.000 description 7
- 238000002591 computed tomography Methods 0.000 description 6
- 238000005516 engineering process Methods 0.000 description 6
- 230000003287 optical effect Effects 0.000 description 6
- 238000005094 computer simulation Methods 0.000 description 5
- 239000013598 vector Substances 0.000 description 5
- 230000008901 benefit Effects 0.000 description 4
- 238000005553 drilling Methods 0.000 description 4
- 210000001624 hip Anatomy 0.000 description 4
- 210000003041 ligament Anatomy 0.000 description 4
- 239000011159 matrix material Substances 0.000 description 4
- 238000013179 statistical model Methods 0.000 description 4
- 238000011882 arthroplasty Methods 0.000 description 3
- 230000006378 damage Effects 0.000 description 3
- 238000010586 diagram Methods 0.000 description 3
- 238000006073 displacement reaction Methods 0.000 description 3
- 230000007170 pathology Effects 0.000 description 3
- 230000002085 persistent effect Effects 0.000 description 3
- 238000006467 substitution reaction Methods 0.000 description 3
- 230000003247 decreasing effect Effects 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 208000014674 injury Diseases 0.000 description 2
- 210000000629 knee joint Anatomy 0.000 description 2
- 230000004044 response Effects 0.000 description 2
- 230000011664 signaling Effects 0.000 description 2
- 238000010561 standard procedure Methods 0.000 description 2
- 230000001360 synchronised effect Effects 0.000 description 2
- 210000002303 tibia Anatomy 0.000 description 2
- 230000008733 trauma Effects 0.000 description 2
- 238000002604 ultrasonography Methods 0.000 description 2
- 238000012800 visualization Methods 0.000 description 2
- 208000025674 Anterior Cruciate Ligament injury Diseases 0.000 description 1
- 241001653121 Glenoides Species 0.000 description 1
- 241000567769 Isurus oxyrinchus Species 0.000 description 1
- 238000012084 abdominal surgery Methods 0.000 description 1
- 210000000588 acetabulum Anatomy 0.000 description 1
- 230000004075 alteration Effects 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 210000001188 articular cartilage Anatomy 0.000 description 1
- 230000003416 augmentation Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 239000000560 biocompatible material Substances 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 210000004204 blood vessel Anatomy 0.000 description 1
- 239000000872 buffer Substances 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 210000000080 chela (arthropods) Anatomy 0.000 description 1
- 239000011248 coating agent Substances 0.000 description 1
- 238000000576 coating method Methods 0.000 description 1
- 238000007906 compression Methods 0.000 description 1
- 230000006835 compression Effects 0.000 description 1
- 238000005520 cutting process Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 210000002391 femur head Anatomy 0.000 description 1
- 238000002594 fluoroscopy Methods 0.000 description 1
- 230000004927 fusion Effects 0.000 description 1
- 210000004394 hip joint Anatomy 0.000 description 1
- 210000004095 humeral head Anatomy 0.000 description 1
- 208000015181 infectious disease Diseases 0.000 description 1
- 238000007689 inspection Methods 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 230000005865 ionizing radiation Effects 0.000 description 1
- 210000001503 joint Anatomy 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 239000000463 material Substances 0.000 description 1
- 150000002739 metals Chemical class 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 238000012829 orthopaedic surgery Methods 0.000 description 1
- 238000002559 palpation Methods 0.000 description 1
- 210000003049 pelvic bone Anatomy 0.000 description 1
- 238000004321 preservation Methods 0.000 description 1
- 238000007639 printing Methods 0.000 description 1
- 238000011084 recovery Methods 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 238000007670 refining Methods 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 238000005316 response function Methods 0.000 description 1
- 210000002832 shoulder Anatomy 0.000 description 1
- 238000011272 standard treatment Methods 0.000 description 1
- 230000026676 system process Effects 0.000 description 1
- 238000012549 training Methods 0.000 description 1
- 230000001052 transient effect Effects 0.000 description 1
- 238000001429 visible spectrum Methods 0.000 description 1
- 238000011179 visual inspection Methods 0.000 description 1
- XLYOFNOQVPJJNP-UHFFFAOYSA-N water Substances O XLYOFNOQVPJJNP-UHFFFAOYSA-N 0.000 description 1
Images
Classifications
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B34/00—Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
- A61B34/20—Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B34/00—Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
- A61B34/10—Computer-aided planning, simulation or modelling of surgical operations
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B34/00—Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
- A61B34/25—User interfaces for surgical systems
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B34/00—Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
- A61B34/30—Surgical robots
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B90/00—Instruments, implements or accessories specially adapted for surgery or diagnosis and not covered by any of the groups A61B1/00 - A61B50/00, e.g. for luxation treatment or for protecting wound edges
- A61B90/36—Image-producing devices or illumination devices not otherwise provided for
- A61B90/361—Image-producing devices, e.g. surgical cameras
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/03—Arrangements for converting the position or the displacement of a member into a coded form
- G06F3/0304—Detection arrangements using opto-electronic means
- G06F3/0317—Detection arrangements using opto-electronic means in co-operation with a patterned surface, e.g. absolute position or relative movement detection for an optical mouse or pen positioned with respect to a coded surface
- G06F3/0321—Detection arrangements using opto-electronic means in co-operation with a patterned surface, e.g. absolute position or relative movement detection for an optical mouse or pen positioned with respect to a coded surface by optically sensing the absolute position with respect to a regularly patterned surface forming a passive digitiser, e.g. pen optically detecting position indicative tags printed on a paper sheet
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T19/00—Manipulating 3D models or images for computer graphics
- G06T19/006—Mixed reality
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/50—Depth or shape recovery
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/60—Analysis of geometric attributes
- G06T7/62—Analysis of geometric attributes of area, perimeter, diameter or volume
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/70—Determining position or orientation of objects or cameras
- G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/80—Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
- G06T7/85—Stereo camera calibration
-
- G—PHYSICS
- G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
- G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- G16H20/00—ICT specially adapted for therapies or health-improving plans, e.g. for handling prescriptions, for steering therapy or for monitoring patient compliance
- G16H20/30—ICT specially adapted for therapies or health-improving plans, e.g. for handling prescriptions, for steering therapy or for monitoring patient compliance relating to physical therapies or activities, e.g. physiotherapy, acupressure or exercising
-
- G—PHYSICS
- G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
- G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- G16H20/00—ICT specially adapted for therapies or health-improving plans, e.g. for handling prescriptions, for steering therapy or for monitoring patient compliance
- G16H20/40—ICT specially adapted for therapies or health-improving plans, e.g. for handling prescriptions, for steering therapy or for monitoring patient compliance relating to mechanical, radiation or invasive therapies, e.g. surgery, laser therapy, dialysis or acupuncture
-
- G—PHYSICS
- G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
- G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- G16H30/00—ICT specially adapted for the handling or processing of medical images
- G16H30/20—ICT specially adapted for the handling or processing of medical images for handling medical images, e.g. DICOM, HL7 or PACS
-
- G—PHYSICS
- G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
- G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- G16H30/00—ICT specially adapted for the handling or processing of medical images
- G16H30/40—ICT specially adapted for the handling or processing of medical images for processing medical images, e.g. editing
-
- G—PHYSICS
- G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
- G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- G16H50/00—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics
- G16H50/50—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics for simulation or modelling of medical disorders
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/20—Image signal generators
- H04N13/204—Image signal generators using stereoscopic image cameras
- H04N13/207—Image signal generators using stereoscopic image cameras using a single 2D image sensor
- H04N13/221—Image signal generators using stereoscopic image cameras using a single 2D image sensor using the relative movement between cameras and objects
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/20—Image signal generators
- H04N13/204—Image signal generators using stereoscopic image cameras
- H04N13/246—Calibration of cameras
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B34/00—Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
- A61B34/20—Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
- A61B2034/2046—Tracking techniques
- A61B2034/2055—Optical tracking systems
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B34/00—Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
- A61B34/20—Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
- A61B2034/2046—Tracking techniques
- A61B2034/2055—Optical tracking systems
- A61B2034/2057—Details of tracking cameras
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B34/00—Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
- A61B34/20—Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
- A61B2034/2046—Tracking techniques
- A61B2034/2065—Tracking using image or pattern recognition
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B34/00—Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
- A61B34/30—Surgical robots
- A61B2034/301—Surgical robots for introducing or steering flexible instruments inserted into the body, e.g. catheters or endoscopes
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B34/00—Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
- A61B34/30—Surgical robots
- A61B2034/302—Surgical robots specifically adapted for manipulations within body cavities, e.g. within abdominal or thoracic cavities
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B90/00—Instruments, implements or accessories specially adapted for surgery or diagnosis and not covered by any of the groups A61B1/00 - A61B50/00, e.g. for luxation treatment or for protecting wound edges
- A61B90/36—Image-producing devices or illumination devices not otherwise provided for
- A61B2090/364—Correlation of different images or relation of image positions in respect to the body
- A61B2090/365—Correlation of different images or relation of image positions in respect to the body augmented reality, i.e. correlating a live optical image with another image
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B90/00—Instruments, implements or accessories specially adapted for surgery or diagnosis and not covered by any of the groups A61B1/00 - A61B50/00, e.g. for luxation treatment or for protecting wound edges
- A61B90/39—Markers, e.g. radio-opaque or breast lesions markers
- A61B2090/3937—Visible markers
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B90/00—Instruments, implements or accessories specially adapted for surgery or diagnosis and not covered by any of the groups A61B1/00 - A61B50/00, e.g. for luxation treatment or for protecting wound edges
- A61B90/39—Markers, e.g. radio-opaque or breast lesions markers
- A61B2090/3937—Visible markers
- A61B2090/3945—Active visible markers, e.g. light emitting diodes
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B90/00—Instruments, implements or accessories specially adapted for surgery or diagnosis and not covered by any of the groups A61B1/00 - A61B50/00, e.g. for luxation treatment or for protecting wound edges
- A61B90/39—Markers, e.g. radio-opaque or breast lesions markers
- A61B2090/3983—Reference marker arrangements for use with image guided surgery
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B90/00—Instruments, implements or accessories specially adapted for surgery or diagnosis and not covered by any of the groups A61B1/00 - A61B50/00, e.g. for luxation treatment or for protecting wound edges
- A61B90/39—Markers, e.g. radio-opaque or breast lesions markers
- A61B2090/3991—Markers, e.g. radio-opaque or breast lesions markers having specific anchoring means to fixate the marker to the tissue, e.g. hooks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30204—Marker
- G06T2207/30208—Marker matrix
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30244—Camera pose
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2219/00—Indexing scheme for manipulating 3D models or images for computer graphics
- G06T2219/004—Annotating, labelling
Definitions
- the disclosure generally relates to the field of computer-aided surgery, and in particular, but not by way of limitation, the disclosed embodiments refer to computer aided-navigation in camera guided procedures of surgery and diagnosis in anatomical regions with rigid tissues such as bone, which includes arthroscopy of knee, hip, or shoulder, and open surgery in orthopedics and dentistry in which case a camera must be used to observe the operating field.
- One or more embodiments can also be employed in any other application domain, such as industrial inspection, that uses a camera system to visualize a work space that comprises rigid, non-deformable parts.
- MIS procedures aim to minimize damage to healthy tissue by accessing targeted organs and anatomical cavities through relatively small size incisions. Since the workspace is not fully exposed, the surgeon typically carries the medical procedure using as guidance video acquired by a camera system that is inserted into the cavity. MIS procedures are being increasingly adopted in different medical specialties, such as orthopedics, abdominal surgery, urology, neurosurgery, and ENT, just to name a few.
- Arthroscopy is a MIS procedure for treatment of damaged joints in which instruments and endoscopic camera (the arthroscope) are inserted into the articular cavity through small incisions (the surgical ports). Arthroscopy, as opposed to conventional open surgery, largely preserves the integrity of the articulation, which is beneficial for the patient in terms of reduction of trauma, risk of infection and recovery time.
- arthroscopic procedures are relatively difficult to execute because of indirect visualization and limited maneuverability inside the joint, with novices having to undergo a long training period and experts often making mistakes of clinical consequences. This is a scenario where computer-assistive technologies for safely guiding the surgeon throughout the procedure can make a difference, both in terms of improving clinical outcome and in terms of decreasing the surgeon learning curve.
- a system for Computer-Aided Surgery comprises two distinct stages: (i) an offline step in which the procedure is planned leading to some sort of computational model that can either be a three-dimensional (3D) pre-operative image of the patient's organ (e.g. CT-Scan), a statistical bone model, or a set of guidelines for inferring meaningful locations with respect to anatomical landmarks; and (ii) an intra-operative navigation step in which the computer guides the surgeon throughout the procedure for the execution to be done as defined.
- 3D three-dimensional
- the intra-operative navigation usually passes by overlying the pre-operative computational model with the actual bone, and by localizing in real-time the tools and instruments with respect to each other, and with respect to the targeted organ.
- the technology to accomplish this task is Optical-Tracking (OT) that consists in using a stationary stereo head, henceforth called base station, for tracking a set of markers that are rigidly attached to instruments and/or bone.
- the stereo head comprises two infrared (IR) cameras that track a set of point markers that are rigidly attached to the object of interest.
- the position of each marker is estimated by simple triangulation and, since their relative arrangement is known ‘a priori’, the 3D pose of the object of interest is computed in the reference frame of the base station.
- IR infrared
- the surgical navigation solutions that are currently available for Orthopedics, Neurosurgery, and ENT invariably rely in OT.
- the typical workflow passes by the surgeon to rigidly attach a tool marker to patient and/or targeted organ, which is followed by pin pointing anatomical landmarks with a calibrated tracked probe.
- the 3D position of these landmarks is determined in the coordinate system of the base station and the pre-operative computational model is registered with the patient. From this point on, it is possible to determine in real-time the pose of instruments with respect to patient and plan, which enables the system to safely guide the surgeon throughout the procedure.
- the O-arm from Medtronic® combines OT with a CT-scanner that enables the acquiring of the 3D pre-operative model of patient's anatomy in the Operating Room (OR) before starting the procedure, which avoids the surgeon performing explicit registration.
- the system that is being developed by 7D Surgical® goes in the same direction with the 3D model being obtained using multi-view reconstruction and structured light to avoid the ionizing radiation of CT-scanning. Nevertheless, these systems still rely in conventional OT to know the relative position between instruments and anatomy after registration has been accomplished.
- OT has proved to be an effective way of obtaining real-time 3D information in the OR, which largely explains the fact of being transversally used across different systems and solutions.
- the technology has several drawbacks that preclude a broader dissemination of surgical navigation: (i) it requires a significant investment in capital equipment, namely in acquiring the base station; (ii) it disrupts normal surgical workflow by changing the OR layout to accommodate additional equipment, by forcing the surgeon to work with instruments with bulky tool markers attached, and by constraining the team movements due to the need of preserving lines of sight between base station and tool markers; and (iii) it is not well suited to be used in MIS procedures because organs and tissues are occluded which avoids placing marker tools that can be observed from the outside by the base station.
- OT based navigation in arthroscopic procedures always requires opening additional incisions such that the marker tool attached to the bone protrudes through patient skin.
- Electromagnetic Tracking is currently used in some surgical navigation systems with the advantage of not requiring preservation of a line of sight.
- it has the problem of being vulnerable to electromagnetic interference caused by nearby metals and devices, being in practice less reliable and accurate than OT.
- OT Electromagnetic Tracking
- it still requires additional capital equipment, namely a base station, and the need of attaching coil markers with hanging wires to organs makes it non amenable to MIS procedures.
- the embodiments in the disclosure provide a new concept for computer-assisted procedures of surgery and diagnosis that target rigid, non-deformable anatomical parts such as bone, tissue, or teeth.
- the disclosure describes attaching small visual markers to instruments and anatomy of interest (e.g. bone surface), with each marker having a printed known pattern for detection and unique identification in images acquired by a free-moving camera, and a geometry that enables estimating its rotation and translation with respect to the camera using solely image processing techniques.
- VTIAC Visual-Tracking Inside the Anatomical Cavity
- the disclosure discloses the apparatus for VTIAC and the required initial calibration procedures, it describes how to use VTIAC to perform very accurate 3D measurements inside the anatomical cavity, and it shows how to use augmented reality, virtual reality, or robotics to provide real-time guidance to the surgeon after registering a pre-operative 3D plan.
- VTIAC is specially well suited for arthroscopy where the already existing monocular arthroscope acts as the free-moving camera that provides the video input.
- VTIAC can be successfully employed in any clinical procedure that targets anatomical regions with rigid parts, such as open orthopaedic surgery or dentistry, in which case the operating field must be observed by a camera that can either be attached to a tool or handheld.
- the disclosure describes illustrative implementations in knee arthroscopy and spine surgery that by no means limit the range of possible clinical applications.
- FIG. 1 A is an embodiment of a representation of the computation of the World Marker (WM) 3D pose C in the camera reference frame using the homography relation H C that maps WM points into image points.
- WM World Marker
- FIG. 1 B is an embodiment of a representation of a tool with a Tool Marker (TM) and WM being simultaneously seen by the camera.
- TM Tool Marker
- Both the rigid transform between the WM and the camera (C), and between the TM and the camera ( ⁇ circumflex over (T) ⁇ ) can be computed using the homography relations.
- the rigid transform between WM and TM (T) can be easily computed from the previous transforms.
- FIG. 1 C is an embodiment of a representation of the 3D reconstruction of points, curves, or sparse mesh in the surface of the rigid tissue using a touch-probe with a TM that is calibrated such that the tip P T is known.
- a touch-probe with a TM that is calibrated such that the tip P T is known.
- FIG. 1 D is an embodiment of a representation of the operation of overlying a surgical plan with the patient's anatomy by using the 3D reconstruction results in a suitable 3D registration algorithm that provides the rigid transformation M that maps surgical plan coordinates into WM coordinates.
- FIG. 1 E is an embodiment of a representation of VTIAC assisted execution of a surgical procedure where the surgical plan is overlaid into images in real-time (Augmented Reality) for the purpose of guiding the surgeon in positioning and/or orienting an instrument.
- FIG. 2 A is an example of some of the free moving cameras that can be used with the VTIAC.
- Top is an endoscopic lens than can be attached to a camera-head for visualization during arthroscopy
- bottom is a generic handheld camera that can be employed for VTIAC intra-operative navigation in open surgery with direct visual access.
- FIG. 2 B is an embodiment of a World Marker (WM) that is in this particular case a screw-like object that comprises one planar facet with a known visual pattern that defines the world system of coordinates.
- WM World Marker
- FIG. 2 C is an embodiment of a touch-probe with a Tool Marker (TM) attached that defines a local system of coordinates in which the position of the tip P T is known.
- TM Tool Marker
- FIG. 2 D is an embodiment of a surgical instrument with a Tool Marker (TM) attached to the surgical instruments for execution of the clinical procedure.
- the surgical tool can be any tool used during the operation, including but not limited to powered tools (e.g. resection bur), drill guides, or any other instrument used for resection or palpation.
- the TM defines a local system of coordinates in which the position of instrument's relevant parts—such as tip, symmetry axis, or even a complete CAD model of the instrument—are known.
- FIG. 2 E is a schematic of an embodiment of the image processing system, that may correspond to or may be part of a computer and/or any other computing device, that receives as input images and video acquired by the camera, as well as commands that may be entered by a button panel, keyboard, camera buttons, foot switch and/or any other input interface, and that outputs the processing results to a display.
- FIG. 2 F is an embodiment of a representation of the tool calibration process, which includes finding the coordinates P T of a particular tool point in the TM reference frame.
- the calibration operation can be quickly carried in simultaneous with the initial calibration of the camera without requiring the acquisition of additional calibration frames.
- the rigid transformation ⁇ maps coordinates in the grid reference frame into coordinates in the camera reference frame.
- FIG. 3 A is an embodiment of a planar pattern template (synthetic image), where the quadrilateral shape and high contrast enable fast detection of the corners, the sharp corners provide accurate point detection (P 1 , P 2 , P 3 and P 4 ), and a bitmap binary code allows visual identification.
- FIG. 3 B is an embodiment of the result of detection and identification of a planar marker in close-range operation using a wide FOV camera that introduces very strong radial distortion.
- the correct detection and point correspondences (u 1 , u 2 , u 3 and u 4 against P 1 , P 2 , P 3 and P 4 of the template pattern) is not accurate enough for the demands of the medical applications, due to the compression effect of the radial distortion in the periphery of the image.
- FIG. 3 C is an embodiment of the representation of the camera pose estimation using the detection on the highly distorted image (C 0 ) and the camera pose estimation (C) after computing the pose update ⁇ .
- This pose update is computed by minimizing the photo-geometric error between the template pattern and the detection on the image.
- FIG. 3 D is an embodiment of a representation of the warping operation for the minimization of the photo-geometric error between the template pattern and the current marker detection in the image.
- the pose update ⁇ encodes the increments in rotation ⁇ R and in translation ⁇ t , that are estimated taking into account the amount of distortion in the image for increased accuracy.
- FIG. 4 A is an embodiment of the VTIAC reconstructing points in 3D in ACL reconstruction.
- the tip of the probe is positioned such that both WM and TM are in the FOV of the camera.
- Points position in 3D are memorized by the system in the reference frame of the WM.
- the figure shows the Augmented Reality (AR) view, where information is overlaid, and in the top right corner the Virtual Reality (VR) view where points are displayed in 3D.
- AR Augmented Reality
- VR Virtual Reality
- FIG. 4 B is an embodiment of the VTIAC reconstructing curves in 3D in ACL reconstruction where the surgeon uses the touch-probe to outline the curves. These particular curves are the inter-condyle contours that are overlaid in image and shown in 3D in the VR window in the top right corner of image.
- FIG. 4 C is an embodiment of the VTIAC reconstructing a dense surface by randomly grasping the bone surface with an instrumented tool. On the upper right corner, the reconstructed points in 3D are shown, along with a graphical representation of the touch probe.
- FIG. 4 D is an embodiment of using the VTIAC to measure distance between points. By pinpointing points with a calibrated tool, TM coordinates are transferred to the WM reference frame, and distances are computed with sub-millimetric accuracy.
- FIG. 4 E is an embodiment of using the VTIAC to measure shape and/or area of a region by establishing a closed contour on the bone surface. This shape can then be analyzed to infer metrics such as area of tissue to remove or volume of implants needed for surgery.
- FIG. 5 A is an embodiment of using the VTIAC to find the location of the femoral tunnel in ACL reconstruction. This is accomplished by measuring the length of the notch ceiling and placing the ACL footprint at 1 ⁇ 3 the length counting from the posterior end.
- FIG. 5 B is an embodiment of using the VTIAC to correctly orient the drill tool for opening the femoral tunnel in ACL reconstruction. This is accomplished by using the VR view to anticipate in real-time the position and exit point of the tunnel depending on the pose of the drill tool that is determined at each frame instant from arthroscopic video.
- FIG. 6 A is an embodiment of using the VTIAC to pinpoint points for the purpose of registration with a pre-operative model of the patient's anatomy in Placing Pedicle Screws (PPS) in spine surgery.
- PPS Pedicle Screws
- fiducial points on the vertebra are indicated to the surgeon (green point in the upper right corner VR view) so he/she can touch the correspondent point in the patient's anatomy.
- the point correspondences are then used for 3D registration.
- the touch-probe is rigidly attached to the camera (CamT) which means that only the WM is required to be visible in images
- FIG. 6 B is an embodiment or using VTIAC to reconstruct a sparse mesh on the surface of the vertebra for the purpose of registration with a pre-operative model during PPS.
- the touch-probe is independent of the camera and both WM and TM must be kept in the image.
- FIG. 6 C is an embodiment of using the VTIAC for the measurement of the angle of the insertion in PPS in spine surgery.
- Virtual representation of the insertion guide is presented in the upper right corner (VR view) and the angle measurement between instrumented Kirschner wire and guide is presented in the AR view.
- the VTIAC system can provide visual alerts when the angle of penetration is considered acceptable.
- FIG. 6 D is an embodiment of using the VTIAC for the visual inspection of the perforation trajectory for PPS.
- the virtual representation of the instrument is augmented along its axis so the surgeon can see the outcome of the perforation for each position of the tool.
- FIG. 6 E is an embodiment of using the VTIAC for guiding a perforation or insertion of instrument during PPS.
- the optical axis of the virtual camera is aligned with the desired line of perforation or insertion.
- the penetration depth displayed is relative to the bone surface at the entry point.
- FIG. 7 A is a prototype of the embodiment of a Camera Tool (CamT) including a small-camera mounted on a custom made hand-holder that can either be used as a free-hand camera, or coupled with touch-probe such that camera, holder, and probe become a single rigid body.
- P C is the tool tip that can be calibrated with respect to the camera.
- FIG. 7 B is an embodiment of using the CamT to reconstruct points in the bone surface.
- the 3D coordinates P C of the tip of the probe are known in the camera reference frame and therefore they can be referenced to the WM for measurements computation, as if it was a tool with a TM.
- FIG. 7 C is an embodiment of a representation of the CamT calibration using a single image of the calibration pattern to simultaneously calibrate the camera and the tool tip position P G .
- FIG. 7 D is an embodiment of a representation of using the VTIAC with a laser pointer instead of a touch probe for contactless measurements.
- the lases pointer is instrumented with a TM that is in the FOV of the camera.
- the laser pointer produces a colored dot on the bone/tissue surface (point of incidence), that is seen in the image (as well as the TM and WM).
- the point is reconstructed in 3D by intersecting the line L T of the beam with the back-projection line B x of the image point x where the point of light incidence is projected.
- FIG. 7 E is an embodiment of a representation of using the VTIAC with a Laser Rangefinder (LRF), or other equivalent device or technology relying in Time-of-Flight (ToF) principles, that is able to measure distances along the direction of the beam line L T .
- LRF Laser Rangefinder
- ToF Time-of-Flight
- S T and d T are known in the local reference frame of TM.
- the surgeon orients the LRF such that the beam becomes incident with the point of interest in the anatomy, and acquires in a synchronous manner the distance measurement ⁇ and an image where both WM and TM are visible.
- the point of interest can be outside the camera FOV.
- FIG. 7 F is an embodiment of a representation of the use of multiple WM to increase the area of operation.
- the surgeon can fix an auxiliary visual marker (WM′) on a convenient location on the bone surface, and move the camera such that both WM and WM′ lie in the FOV.
- WM′ auxiliary visual marker
- FIG. 7 G is an embodiment of a representation of using the VTIAC to provide control commands to a robotic system.
- the robotic end effector is instrumented with a TM and the anatomy has a WM attached such that relative 3D pose can be computed from video acquired by the free-moving camera.
- the VTIAC uses the visual feedback to guide the positioning of the end-effector by sending commands to the robot.
- the robotic arm can also provide feedback of the actuators state that can be fused with visual feedback for more accurate closed loop control.
- FIG. 8 is a schematic diagram of an embodiment of an image processing system.
- FIG. 9 is a schematic diagram of an embodiment of a computing device.
- One or more embodiments disclosed herein applies to camera-guided orthopedic MIS procedures, namely arthroscopy, that is used as illustrative example throughout most of the description.
- the application of the presently disclosed embodiments can include other surgical procedures and clinical specialties where the operating field comprises rigid, non-deformable parts and surfaces.
- the application of the disclosed embodiments requires a camera system for visualizing the anatomical scene that might already exist (e.g. arthroscopy) or be added (e.g. open orthopedic surgery).
- One or more embodiments in the disclosure provide a surgical navigation scheme for arthroscopy and other procedures using a conventional camera and with scenes that comprise rigid surfaces.
- the surgical navigation scheme will be referred to as Visual-Tracking Inside the Anatomical Cavity (VTIAC).
- VTIAC Visual-Tracking Inside the Anatomical Cavity
- the disclosure relates to attaching small, recognizable visual markers to instruments and rigid anatomy (e.g. bones) and use the free-moving camera, that is the arthroscope in case of arthroscopic procedures, to estimate their relative rotation and translation (the relative 3D pose).
- the relative 3D pose is determined by estimating the plane-to-image homography that is factorized to obtain the rotation and translation between plane and camera reference frames.
- the marker attached to the bone surface serves as absolute reference with all measurements being expressed in its coordinate system (world coordinates).
- VTIAC can be used to obtain 3D information about the bone surface, register a pre-operative computational model, and ultimately solve the navigation issues by providing guidance using augmented reality, virtual reality, or robotic actuation.
- VTIAC introduces many differences relatively to other embodiments of OT/ET in the context of computer-aided surgery in general and arthroscopy in particular.
- the global world reference frame instead of being the external stereo head (the base station), is substituted by the system of coordinates of the WM that is inside the articular cavity. This avoids issues related to preserving lines of sight in the OR, as well as the need of having marker tools protruding through patient skin.
- the approach relies on processing the video acquired by a free-moving camera, which means that in the case of arthroscopy there is no need of investing in additional capital equipment that provides alternative sensing modalities.
- measurements are performed in the images acquired at close range inside the anatomical cavity, which dramatically increases spatial and/or metric accuracy with respect to OT or ET.
- the plane-to-image homography may be a factor in the VTIAC approach for surgical navigation.
- the projection of a plane into a perspective image may be described by a 3 ⁇ 3 matrix transformation (the homography) that encodes the plane rotation and translation (the plane 3D pose) in camera coordinates.
- the homography has been broadly used in the field of Computer Vision for several different purposes, ranging from camera calibration to visual tracking, and passing by 3D motion estimation.
- an OT system the MicronTracker® developed by Claronav®
- the approach herein described differs from MicronTracker® in that the tracking is performed by a moving monocular camera as opposed to a stationary stereo setup.
- the base station is the external stereo setup, which raises the issues about line of sight inherent to conventional OT, in VTIAC, measurements are carried out with respect to the WM that is rigidly attached to the surface inside the articular joint or anatomical cavity.
- inventions may be used to determine the relative pose between a laparoscope and an intra-operative ultrasound (US) probe or laser projector.
- the embodiments attach a printed planar pattern to the probe and/or projector that is viewed by the laparoscope. This enables estimation of the plane-to-image homography and determination of the relative pose of the probe and/or projector in camera coordinates.
- VTIAC provides a much broader range of functionalities that arise from using a World Marker (WM) attached to the bone surface.
- WM World Marker
- VTIAC not only provides the relative pose of tools and devices that are inserted into the anatomical cavity, but it also enables the reconstruction of points and contours on the surface of the organ of interest that are pin-pointed by the surgeon.
- This information can be used for a multitude of purposes such as metric measurements, registration of pre-operative models, or guidance using augmented reality, that are seamlessly supported by the framework.
- measurements are typically represented in camera coordinates, which means that it is not possible to relate or integrate information across frames because the laparoscope is in constant motion.
- VTIAC VTIAC
- all measurements are stored in the coordinate system of the WM that works as an absolute reference across time and space. Thus, the visual tracking process can even be discontinued, and the 3D information obtained till that moment becomes readily available as soon as the WM is redetected in the images acquired by the moving camera.
- Section 2 provides an overview of the concepts behind the VTIAC
- Section 3 provides details on the apparatus and calibration of the necessary tools to be used with the system
- Section 4 provides a description of the visual markers' accurate detection under high radial distortion
- Section 5 details the estimation of 3D pose from the detection of markers in the image and practical capabilities of the system
- Section 6 provides an overview of the operation flow of the VTIAC system for operation during surgery
- Section 7 provides extensions and variations on the tools and methods presented before.
- VTIAC In order to better illustrate the usefulness of VTIAC, two embodiments that can be applied to design a navigation system for the arthroscopic reconstruction of the Anterior Cruciate Ligament (ACL) and for Placing Pedicle Screws (PPS) in spine surgery are presented (sections 8 and 9). These procedures are mere examples that do not limit in any way the potential applications of VTIAC. As stated in the following sections, the VTIAC can be applied to a multitude of arthroscopic procedures, as well as open procedures and including dentistry surgery.
- ACL Anterior Cruciate Ligament
- PPS Pedicle Screws
- points are represented by their vectors of coordinates and vectors are denoted by a bold letter (e.g., P, x).
- the rigid displacement between coordinate frames is represented by a 4 ⁇ 4 matrix in the Special Euclidean Group (SE(3)) where the left upper 3 ⁇ 3 submatrix is a rotation matrix and 3 ⁇ 1 right upper submatrix is a translation vector.
- SE(3) Special Euclidean Group
- Matrices are typically denoted by plain capital letters (e.g., C, T).
- VTIAC Visual-Tracking Inside the Anatomical Cavity
- the free-moving camera is assumed to be calibrated such that image points u in pixel coordinates can be mapped into image points x in metric coordinates as if the image had been acquired by a perfect pin-hole.
- the free-moving camera is an arthroscopic camera and that the anatomical part of interest is a bone. It is also assumed that visual markers are planar with a known pattern.
- the surgeon After accessing the anatomical cavity, the surgeon starts by rigidly attaching a marker to the bone surface that is referred as the World Marker (WM). If the marker is planar, then its projection is described by an homography H C , that maps plane points into image points, and encodes the relative rotation R C and translation t C between marker and camera reference frames. Thus, and since H C can be estimated from image information, it is possible to use this homography relation to determine at every frame time instant the 4 ⁇ 4 matrix C that transforms world coordinates into camera coordinates ( FIG. 1 A ).
- the tool or instrument is a calibrated touch-probe such that P T is the vector of 3D coordinates of its tip in the TM reference frame.
- the surgeon can reconstruct a point of interest in the bone surface by touching it with the probe and acquiring a frame where both WM and TM are visible. This enables computation of the pose T of the probe and the obtaining of the point of interest P expressed in world coordinates ( FIG. 1 C ).
- the process above can be applied to successive frames in order to reconstruct a curve in the bone surface.
- the surgeon outlines the contour of interest while keeping both WM and TM in the Field-of-View (FOV) of the free-moving camera.
- FOV Field-of-View
- This enables the obtaining of successive P estimates that define the desired 3D curve. Since 3D reconstruction results are stored in World Marker coordinates, the action of outlining can be stopped and resumed at any time. If the process is interrupted for any reason, it suffices for the camera to see again the WM for all the 3D information to be restored without having to repeat the tedious touching process ( FIG. 1 C ).
- the 3D reconstruction results can be used for the purpose of measuring, estimating shape, or overlying a pre-operative plan in the actual patient anatomy (3D registration).
- This pre-operative plan can be a set of rules using anatomical landmarks, a statistical 3D model of the anatomy of interest, or an actual 3D image of the organ (e.g. CT Scan) augmented with guidance information inserted by the surgeon (surgical plan).
- a suitable 3D registration algorithm is selected for estimating the rigid transformation M that maps points P M in the pre-operative image into corresponding points P in the intra-operative reconstruction obtained with VTIAC ( FIG. 1 D ).
- This enables representing the information of the model, including guidance information, in the system of coordinates of the world marker or, in other words, to overlay the pre-operative plan with the patient's anatomy ( FIG. 1 E ).
- the clinical execution might require, in one embodiment, multiple different instruments—such as guides, drills, shavers, saws, burrs, etc.—that can either be used in sequence or simultaneously.
- Each one of these instruments is assumed to have a Tool Marker (TM) attached that defines a local system of coordinates where the instrument's relevant parts—such as tip, symmetry axis, or even complete CAD model—are represented.
- TM Tool Marker
- the system processes each frame with the objective of detecting, identifying, and estimating the 3D pose of every TM that is in the FOV of the camera.
- the WM is also visible in image, then it is possible to determine the pose of the camera C, locate the instruments in the world coordinate system, relate their poses T with the 3D information stored in the WM reference frame, and ultimately provide real-time assistance to the surgeon ( FIG. 1 E ).
- the last stage of VTIAC consists of assisting the surgeon by performing continuous processing of the video for estimating in real-time the 3D pose of instruments with respect to patient anatomy and/or surgical plan represented in WM coordinates.
- the assistance can take multiple forms depending on a specific task and a preferred user interface.
- Possibilities include overlaying guidance information in video using Augmented Reality (AR), using computer graphics to animate the motion of instruments in a Virtual Reality (VR) environment showing the patient's anatomy and/or surgical plan, or controlling the action of actuators in the case of procedures assisted by robotic systems such as the Mako® or the Navio® robots.
- the apparatus includes:
- the calibration includes determining the vector of parameters k and ⁇ of the back-projection function f ⁇ 1 (the inverse of the projection function f) where k comprises the so-called intrinsic parameters—focal length, principal point, aspect ratio, and skew—and ⁇ stands for the radial distortion parameters.
- x f ⁇ 1 (u; k, ⁇ ) (equation 4)
- the camera can either be pre-calibrated from factory, using any standard method in literature, or calibrated in the Operating Room (OR) just before starting the procedure. The latter is especially recommendable for the case of arthroscopic cameras, or any other camera with exchangeable optics.
- the calibration in the OR can be quickly accomplished by acquiring one image of a known calibration pattern from an arbitrary viewpoint, as described in U.S. Patent Publication No. 2014/0285676, which is incorporated by reference in its entirety. If the camera parameters change during operation because the surgeon rotates the lens scope and/or varies the optical zoom, then the initial calibration may be updated at every frame time using the techniques described in U.S. Patent Publication No.
- the camera calibration must also take into account the medium of operation that, in the case of arthroscopy, is a wet medium. In this situation the initial single image calibration can either be carried in wet medium, or performed in air followed by compensating for the difference in the refractive index of air and water-based medium.
- WM World Marker
- TMs Tool Markers
- the surgeon starts by fixing the World Marker (WM) to the bone surface.
- the WM can be any object comprising at least one planar facet with a known pattern that can be secured (e.g., glued), printed or engraved, and that can be recognized in images; that is small enough to be inserted into the anatomical cavity (e.g., up to 5 mm diameter in the case of arthroscopy); and that can be mechanically attached to the surface such that bone and marker do not move with respect to each other.
- a non-exhaustive list of objects that can be used as WM includes: a screw-like object with a flat head or facet ( FIG. 2 B ); a nail-like object to be fixed by pressure with a flat head or facet; a needle like object with a flat lateral facet for trans-dermic insertion into the joint or cavity; or a flat button-like object that is pulled inside the joint or cavity by a thread or guide.
- the touch-probe in (iii) and the surgical tools in (iv) are instrumented with a visual marker (the Tool Marker or TM), which can either be originally built-in at manufacturing time, or rigidly attached by the user ( FIG. 2 C , FIG. 2 D ).
- TM Tool Marker
- the tip of the tool, the orientation of the tool, or a complete CAD model of the tool may be registered in the TM coordinate frame ( FIG. 2 C ).
- This registration process as described herein is referred to as tool calibration that can either be carried in factory for built-in markers, or performed by the user in case the markers are attached to the tool before starting the procedure.
- the operation can be quickly carried simultaneously with the initial calibration of the camera without requiring the acquisition of additional calibration frames.
- the camera calibration can be accomplished by acquiring a single image of a known grid or checkerboard pattern. This enables recovering the intrinsic parameters k, the radial distortion parameters ⁇ , and the rigid transformation ⁇ that maps coordinates in the grid reference frame into coordinates in the camera reference frame.
- the tool calibration of the surgical instruments can either consist in determining the location of a point, a line or axis, or a CAD model in the coordinate system of the TM attached to the particular instrument. This can be accomplished with the help of the calibrated camera and touch-probe using a method similar to the one used for 3D reconstruction on the bone surface, but where the role of the WM is replaced by the TM of the instrument ( FIG. 1 C ). Thus, for the case of a single point it is enough to pin-point it with the probe while keeping both the TM and the marker of the probe in the camera FOV. For the case of a line or axis the procedure is performed at least two times to reconstruct two points in TM coordinates lying on the line or axis. Finally, if the objective is to register a CAD model of the tool, then the procedure may be performed at least three times to obtain three landmark points in TM coordinates to be used as input in a standard registration method.
- the visual marker used in the WM of (ii) and in the TMs of (iii) and (iv) can comprise a single plane facet with a known pattern as assumed so far, or multiple plane facets with each facet having its own pattern that can be secured (e.g., glued), printed, or engraved, and where the location of each planar pattern is known in a common local coordinate system of the visual marker.
- the advantage of having multiple planar patterns facing different directions is to extend the range of viewing positions and orientations from which the marker can be observed by the camera for estimating the relative 3D pose ( FIG. 1 A , FIG. 1 B ).
- the planar patterns can even be spread across different locations in the tool surface, in which case it suffices for the camera to see one of those patterns to successfully compute the relative pose ⁇ circumflex over (T) ⁇ .
- the visual marker can be non-planar, in which case it should comprise n ⁇ 3 points with known coordinates in the local reference frame of the marker, with these points being such that they can be detected and identified in image in order to allow estimation of the relative pose by applying a Perspective-n-Point (PnP) method.
- PnP Perspective-n-Point
- the small visual markers that are attached to instruments, tools, and anatomy of interest play a fundamental role in VTIAC being key-enablers for using the camera as a measuring device for determining 3D pose.
- the visual marker can have different topological configurations but, for the sake of simplicity and without compromising generality, it will be assumed that the visual marker is a planar surface with a known pattern.
- This planar pattern should be such that it has a local system of coordinates, it is amenable to be detected and uniquely identified from its image projection, and it has fiducial points that can be accurately detected in image for estimating the plane-to-image homography H from point correspondences.
- a point correspondence is the association between a point in the pattern p expressed in local coordinates and its projection x represented in camera coordinates.
- the homography H is a projective transformation that maps the former into the latter, and that can be linearly estimated from N ⁇ 4 point correspondences.
- the homography encodes the rotation and translation between pattern and camera coordinate systems, which means that the factorization of H provides the 3D pose of the pattern in the camera reference frame.
- planar patterns are similar to the CalTag checkerboard patterns, where the quadrilateral shape and high contrast enable fast detection, the sharp corners provide accurate point correspondences, and a bitmap binary code allows visual identification ( FIG. 3 A ).
- These patterns are broadly used as fiducial markers for applications in augmented reality, for which there are several image processing pipelines such as the ARToolKit or the ALVAR. These pipelines implement the steps of detection, identification, and homography estimation in a computationally efficient manner to provide the 3D pose of each planar pattern at every frame time instant.
- this disclosure provides an alternative approach based in photo-geometry.
- the approach includes using standard methods for detection, identification, and initial estimation of pattern rotation r 0 and translation t 0 , followed by refining the 3D pose estimate by minimizing the photo-geometric error in aligning the current pattern image with its template using a warping function that takes into account the non-linear distortion.
- C 0 be the initial 3D pose estimate of the planar pattern in camera coordinates.
- the objective is to determine the pose update ⁇ , that encodes the increments in rotation ⁇ R and in translation ⁇ t , such that the photo-geometric error ⁇ 1 is minimized ( FIG. 3 C , FIG. 3 D )
- T(u) is the pattern template
- I(u) is the current frame
- N i is the image region comprising the pattern
- h the homography map that depends on the relative 3D pose r and t, and f denoting the projection function of the camera that encodes the effect of radial distortion, as described, e.g., in Patent Publication WO/2014054958.
- the template can be understood as a synthetic, fronto-parallel image of the planar pattern ( FIG. 3 D ), that has calibration parameters k′ and ⁇ ′, the homography h depends on the rigid displacement of the views.
- the iterative minimization of the photo-geometric error ⁇ i can be carried using different optimization schemes available in literature such as forward composition, inverse composition, or efficient second order minimization, which requires some changes in formulation and parametrization in SE(3).
- the formulation can also be extended to be resilient to changes in illumination.
- Section 4 describes a method for estimating the 3D pose of a planar visual marker in camera coordinates. Let's consider two of these markers such that one is attached to the anatomy of interest (WM), and the other is attached to a calibrated touch probe (TM).
- WM anatomy of interest
- TM calibrated touch probe
- surgeon places the tip of the probe in the point, positions the camera such that both WM and TM are in the FOV, and commands the system to acquire an image that is processed as follows ( FIG. 4 A ):
- the approach can be extended to obtain a 3D contour or a sparse 3D reconstruction of a surface region, in which case the surgeon uses the touch probe to respectively outline the contour or randomly grasp the surface, while the camera acquires continuous video and steps above are executed for each frame ( FIG. 4 B , FIG. 4 C ).
- the 3D reconstruction results are stored in memory in world coordinates, which means that they can be overlaid in images whenever the WM is in the camera FOV by performing the following steps at each frame time instant ( FIG. 4 A , FIG. 4 B , FIG. 4 C ):
- VTIAC ability of VTIAC to reconstruct and store in memory points, curves, and regions in the anatomy of interest (e.g. bone) has a multitude of purposes and/or possible clinical applications.
- a non-exhaustive list includes:
- the reconstruction results can also be used as input in standard 3D registration methods for aligning or overlying a computational model with the current patient's anatomy.
- Such methods estimate the rigid transformation M that maps points P M in the model into corresponding points P in the intra-operative reconstruction obtained with VTIAC ( FIG. 1 D ).
- surgeon plan is employed in a broad sense and can mean, among other things, a set of rules based on anatomical landmarks, e.g. placing the femoral tunnel of the ACL at 1 ⁇ 3 the length of the notch ceiling measured from its posterior end; the fitting of a statistical model of an anatomy or pathology, e.g. the shape model of CAM femuro-acetabular impingement; or a pre-operative image of the targeted anatomy that can, or cannot, be augmented with guidance information, e.g. a CT scan annotated by the surgeon using a 3D planning software.
- This section describes how VTIAC can combine this 3D data with real-time 3D pose estimation of surgical instruments to provide intra-operative navigation features.
- the surgical instrument that can be a needle, guide, drill, shaver, saw, burr, or any other object required for proper clinical execution—have a TM attached.
- the marker defines a local reference frame where the position of a point, axis, or CAD model of the tool is known (calibrated tool). Navigation is accomplished by executing the following processing steps at every frame time instant:
- VTIAC navigation also works for the case of multiple instruments being used in simultaneous, in which case each instrument has its own TM enabling parallel detection, identification, and estimation of 3D pose T.
- the aiding features can take multiple forms depending on the particular task and/or surgical procedure.
- a non-exhaustive list of these features includes:
- the disclosure has considered that camera and tool or instrument are two entities with independent motions. There are situations for which it might be advantageous to assemble the camera in the tool or instrument such that the two entities become a single rigid body.
- the assembly that is henceforth referred as a Camera Tool or CamT, must be calibrated such that the position of the tool tip, axis of interest, or CAD model of the tool or instrument, is known in the reference frame of the camera.
- the camera can be mounted in a multitude of possible tools ranging from a touch-probe to an impactor for cup placement during hip arthroplasty, passing by burrs and drills. In this setup where camera and tools are physically attached, their relative 3D pose is known, and as long as the camera sees the WM, it is possible to determine the 3D pose of the tool in the global system of coordinates of WM.
- FIG. 7 A shows a prototype embodiment of a Camera Tool (CamT) including a small-camera mounted on a custom made hand-holder that can either be used as a free-hand camera similar ( FIG. 2 A ), or coupled with touch-probe such that camera, holder, and probe become a single rigid body ( FIG. 7 A ).
- the CamT is assumed to be calibrated meaning that the camera is calibrated and the 3D coordinates P C of the tip of the probe are known in the camera reference frame ( FIG. 7 B ).
- the surgeon uses the CamT to touch the point of interest while the WM is kept in the FOV of the camera ( FIG. 7 B ).
- the acquired frame is processed as described in section 3 with the difference that step (ii) is skipped and the formula of step (iii) is replaced by
- the CamT described above can either be pre-calibrated from factory, or calibrated in the OR from a single image of a known grid or checkerboard pattern.
- the surgeon acquires the calibration frame by positioning the camera such that the pattern is visible in image and the tool tip touches a particular point P G whose coordinates are known in the coordinate system of the grid ( FIG. 7 C ).
- the image is used as input in the method that provides the camera intrinsic parameters k, the lens distortion ⁇ , and the rigid transformation ⁇ that maps coordinates in the grid reference frame into coordinates in the camera reference frame.
- the tool calibration is fully accomplished by finding the camera coordinates of the tool tip that are given by
- Section 5 discloses a method for 3D reconstruction where the surgeon uses a calibrated touch-probe to pinpoint points of interest while the camera observes both the WM and the TM of the tool.
- This alternative probe henceforth referred as contactless probe, consists in a laser pointer that emits a collimated beam of visible light.
- the pointer has a visual marker attached—the Tool Marker or TM—and it is assumed to be calibrated such that the position of the line L T defined by the beam is known in TM coordinates.
- the surgeon directs the laser pointer such that the beam becomes incident on the point of interest, and uses the camera to acquire an image where WM, TM, and point of light incidence are visible.
- the point is reconstructed in 3D by intersecting the line L T of the beam with the back-projection line B x of the image point x where the point of light incidence is projected ( FIG. 7 D ).
- the image is processed as stated in Section 5 with the difference that step (iii) is replaced by the two following steps:
- Contactless 3D reconstruction can also be accomplished using an Active Contactless Probe consisting in a Laser Rangefinder (LRF), or other equivalent device or technology relying on Time-of-Flight (ToF) principles, that is able to measure distances ⁇ along the direction of the beam line L T .
- LRF Laser Rangefinder
- ToF Time-of-Flight
- the LRF has a visual marker attached and it is assumed to be calibrated such that the origin and unit direction of measurement, that are respectively S T and d T , are known in the local reference frame of TM.
- the surgeon orients the LRF such that the beam becomes incident with the point of interest in the anatomy, and acquires in a synchronous manner the distance measurement ⁇ and an image where both WM and TM are visible.
- the point of interest can be outside the camera FOV ( FIG. 7 E ).
- the World Marker or WM works as a global reference, which means that it must be viewed by the camera whenever the surgeon wants to use VTIAC for reconstruction or guidance purposes. There might be situations for which keeping the WM in the camera FOV can be difficult to accomplish in practice, either because the camera has a limited FOV, or because the region to cover is simply too broad or wide.
- This problem is solved by using multiple markers as shown in FIG. 7 F .
- the surgeon can fix an auxiliary visual marker (WM′) on a convenient location on the surface of the anatomy, and move the camera such that both WM and WM′ lie in the FOV.
- a frame is acquired, the method of section 4 is applied to determine the 3D pose C and C′ of the two markers ( FIG.
- Section 6 discloses a method for using VTIAC to assist the execution of a clinical procedure where the guidance information is provided by either overlying info in the images or video (AR), or by animating a VR model of anatomy and tools.
- VTIAC can also be used to guide or control the action of a surgical robot ( FIG. 7 G ).
- a surgical system like the Navio® robot relies on conventional OT for determining in real-time the 3D pose between the robotized tool and patient's anatomy and/or surgical plan.
- VTIAC can be used as an alternative to conventional OT for providing the kinematic feedback required to control the robot in closed loop ( FIG. 7 G ).
- the robot end-effector which is the tool or device at the end of the robotic arm or chain, must be instrumented with a Tool Marker and its tip, axis, or CAD model must be registered in TM coordinates.
- the relative 3D pose between end-effector and patient's anatomy/surgical plan is determined as described in section 6 using video acquired by a camera that simultaneously sees the WM and TM.
- This section discloses an embodiment of VTIAC based-navigation for Reconstruction of Anterior Cruciate Ligament (ACL) in the Knee, which can also be generalized for other arthroscopic procedures such as in the shoulder or hip.
- ACL Anterior Cruciate Ligament
- ACL tear is a common pathology for which arthroscopy is the standard treatment (e.g., >300000 cases per year worldwide).
- the procedure includes replacing the torn ACL by a substitution graft that is pulled into the joint through a tunnel opened with a drill. Placing this tunnel in the correct anatomical position is crucial for the knee to fully recover its functionality.
- One technique is the transtibial (TT) approach that opens the tunnel in a single step by drilling from the bottom of the tibia plate till entering into the femur notch. Recent studies show that in about 39% of the cases TT fails in positioning the tunnel at the femoral end, and that much better results can be accomplished using the anteromedial (AM) approach.
- TT transtibial
- AM anteromedial
- VTIAC may be applied to accomplish this intra-operative navigation by indicating the location in the femur notch where to open the tunnel (the ligament footprint) and by guiding the angular orientation of drilling.
- the surgeon starts by calibrating the arthroscopic camera and by attaching the WM in the medial side of the inter-condyle region ( FIG. 5 A ).
- the WM can take the form of a button-like flat surface in the end of a wire guide that is pushed inside-out to go across the femur bone till it surfaces the patient skin. He/she then pulls the wire from the exterior for the marker to be pulled inside the joint and placed against the wall of the femur notch.
- the ligament footprint should be located in the ceiling of the inter-condyle region at 1 ⁇ 3 the length of notch ceiling measured from its posterior end.
- the surgeon uses the touch-probe with the TM to pin-point the two ends of the notch ceiling such that VTIAC can measure the distance and compute the location of the footprint that is overlaid in the video using AR ( FIG. 5 A ).
- the orientation for opening the tunnel may be determined by registering a statistical model of the femur bone.
- the surgeon uses the touch probe to reconstruct the boundary contours of the inter-condyle region ( FIG. 4 B ) or, in alternative, to obtain a sparse 3D reconstruction of the surface of the femur bone ( FIG. 4 C ).
- This 3D data is fed into a suitable 3D registration algorithm that overlays the statistical model with the patient's anatomy.
- the surgeon uses a drill with a TM such that its position can be related in real time with the 3D data stored in memory that includes reconstruction results and the registered statistical model.
- VTIAC indicates the location of the footprint by overlaying in video using AR the point in the anatomy where drill tip should be place (the entry point)
- VTIAC shows in a VR environment the registered model and the current orientation of the drilling tool, where this orientation is computed in real-time from the arthroscopic video where both WM and TM can be seen
- the VR environment shows the drilling direction at each frame time instant such that the surgeon can align it for the exit point to be in the Lateral epicondyle ( FIG. 5 B )
- the tunnel is open along the selected trajectory while VTIAC provides the depth from surface at each frame time instant.
- This section discloses an embodiment of VTIAC based-navigation for Placing Pedicle Screws (PPS) during spine surgery, which can also be generalized to other open procedures where a rigid surface is exposed, such as total hip replacement, total knee replacement, open shoulder surgery and implant placement in dentistry.
- PPS Pedicle Screws
- VTIAC always requires a video input, its use is not limited to arthroscopy.
- the framework can also be applied to open orthopedic procedures, such as knee/hip arthroplasty or spine surgery, as far as a camera is employed to observe incision and relevant anatomy.
- the camera can either be a generic handheld camera ( FIG. 2 A ) or a camera mounted on a tool, such as a touch-probe like in the CamT described in section 7 ( FIG. 7 A ).
- PPS Pedicle Screws
- the surgeon uses a pre-operative model of the vertebra (e.g. CT-Scan or MRI) to specify the 3D line along which the screw must be inserted, as well as the depth of insertion.
- the model and the surgeon specifications are henceforth referred as the pre-operative 3D plan.
- This section describes how VTIAC can be applied to accomplish intra-operative navigation after planning.
- a visual marker plays the role of World Marker (WM) and is placed in an arbitrary position decided by the surgeon.
- the next step is to overlay the pre-operative plan with patient's anatomy in the OR, which passes by reconstructing points and/or curves on the vertebra surface to be used as input in a suitable 3D registration algorithm.
- One possibility is to perform the 3D registration using a set of fiducial points or landmarks in the anatomy.
- the system indicates a succession of landmark points to be reconstructed that are pin-pointed in by the surgeon using the touch-probe ( FIG. 6 A ).
- Another possibility is to use specific contours in the anatomy or a sparse 3D reconstruction of the surface in which case the surgeon randomly grasps the vertebra with the probe ( FIG. 6 B ).
- the reconstruction can either be performed with the CamT, in which case the WM must be in the FOV of the camera ( FIG. 6 A ), or with the standard touch probe with a TM attached, in which case both WM and TM must be visible in images ( FIG. 6 B ).
- the VTIAC is able to overlay the 3D pre-operative plan in the intra-operative video, as well as the tip, axis, or CAD model of the tool, whenever WM and TM are respectively in the FOV of the camera ( FIG. 6 C ).
- the system is able to animate the motion of the tool with respect to the pre-operative model in a VR environment ( FIG. 6 D ).
- the VTIAC can then project the guidance information into the AR view, such as the angle of the tool relatively to the planned direction ( FIG. 6 C ), or to provide a virtual extension of the tool for the surgeon to visualize the expected outcome of the chosen drill location and angle ( FIG. 6 D ).
- the guidance information such as the angle of the tool relatively to the planned direction ( FIG. 6 C )
- the surgeon to visualize the expected outcome of the chosen drill location and angle ( FIG. 6 D ).
- 6 E is as follows: (i) configure the VR such that the optical axis of the virtual camera is aligned with the planned line S of insertion, (ii) move the tool tip along the occluding tissue till the tip overlays with line S that shows as a point, (iii) without moving the tip orient the tool such that its axis L shows as a point coincident with S and finally (iv) insert the tool till desired depth that is indicated by the system.
- VTIAC can be applied for intra-operative navigation in several other clinical procedures.
- a non-exhaustive list of possibilities include:
- PCL Posterior Cruciate Ligament
- FAI Femuro-Acetabular Impingement
- VTIAC can be applied to enforce the pre-planning by overlying the annotated 3D model with the patient's femur in order to safely guide the surgeon.
- the CAM footprint can be overlaid in the arthroscopic video using AR techniques and the system can inform the surgeon about the quantity of the bone tissue to remove at every instant.
- confocal defects are damages in the articular cartilage that can be repaired by filling the holes or craters with a bio-compatible material. This operation often requires placing in the hole or crater an rigid support structure called scaffolder. VTIAC can be used for measuring and determining the shape of confocal defects, as well as to guide the placement of these scaffolds.
- THR Total hip replacement
- the implant consists in a cup, that replaces acetabulum in the pelvic bone, and in a stem with a sphere that replaces the femural head.
- VTIAC can be applied to guide the placement of the cup such that it is inserted with optimal angular orientation, as well as to define the cut plane in the femural neck to remove the head and insert the stem with sphere.
- Knee arthroplasty is an open surgical procedure for replacing total or part of the knee joint by an implant (total or unicompartmental knee replacement).
- VTIAC can be applied to guide the surgeon in cutting the femural condyle and placing the implant.
- Shoulder Joint Replacement This is another open surgical procedure for replacing in total or in part the shoulder joint by an implant.
- VTIAC can be applied in assisting the surgeon in several steps of the execution such as indicating the plane of cut to remove humeral head, or guiding the reaming of humeral shaft and/or glenoid.
- VTIAC can be applied in dental surgery for placing an implant in the maxilar bone as planned in a pre-operative Cone Beam CT (CBCT) of the patient.
- CBCT Cone Beam CT
- the WM is rigidly attached to a tooth
- the CBCT is overlaid with patient's anatomy by using VTIAC features for 3D reconstruction, and the system provides intra-operative guidance for inserting the implant through any of the AR and VR features that have been described in the ACL and PPS examples.
- FIG. 8 is a schematic diagram of an embodiment of an image processing system 500 that may correspond to or may be part of a computer and/or any other computing device, such as a handheld computer, a tablet computer, a laptop computer, a portable device, a workstation, a server, a mainframe, a super computer, and/or a database.
- the image processing system 500 includes a processor 502 , which may also be referenced as a central processor unit (CPU).
- the processor 502 may communicate (e.g., via a system bus) and/or provide instructions to other components within the image processing system 500 , such as the input interface 504 , output interface 506 , and/or memory 508 .
- the processor 502 may include one or more multi-core processors and/or memory (e.g., cache memory) that function as buffers and/or storage for data.
- processor 502 may be part of one or more other processing components, such as application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), and/or digital signal processors (DSPs).
- ASICs application specific integrated circuits
- FPGAs field-programmable gate arrays
- DSPs digital signal processors
- FIG. 8 illustrates that processor 502 may be a single processor, processor 502 is not so limited and instead may represent a plurality of processors.
- the processor 502 may be configured to implement any of the methods described herein.
- FIG. 8 illustrates that memory 508 may be operatively coupled to processor 502 .
- Memory 508 may be a non-transitory computer readable medium configured to store various types of data.
- memory 508 may include one or more memory devices that comprise secondary storage, read-only memory (ROM), and/or random-access memory (RAM).
- the secondary storage is typically comprised of one or more disk drives, optical drives, solid-state drives (SSDs), and/or tape drives and is used for non-volatile storage of data.
- the secondary storage may be used to store overflow data if the allocated RAM is not large enough to hold the working data.
- the secondary storage may also be used to store programs that are loaded into the RAM when such programs are selected for execution.
- the ROM is used to store instructions and perhaps data that are read during program execution.
- the ROM is a non-volatile memory device that typically has a small memory capacity relative to the larger memory capacity of the secondary storage.
- the RAM is used to store volatile data and perhaps to store computer executable instructions.
- the memory 508 may be used to house the instructions for carrying out various embodiments described herein.
- the memory 508 may comprise an image processing module 510 that may be accessed and implemented by processor 502 .
- the image processing module 510 may be stored and accessed within memory embedded in processor 502 (e.g., cache memory).
- the image processing module 510 may estimate the camera response function and the vignetting in case of non-uniform illumination using one or more calibration images.
- memory 508 interfaces with a computer bus so as to communicate and/or transmit information stored in memory 508 to processor 502 during execution of software programs, such as an operating system, application programs, device drivers, and software modules that comprise program code, and/or computer executable process steps, incorporating functionality described herein, e.g., the image processing module 510 .
- Processor 502 first loads computer executable process steps from storage, e.g., memory 510 , storage medium/media, removable media drive, and/or other storage device.
- Processor 502 can then execute the stored process steps in order to execute the loaded computer executable process steps.
- Stored data e.g., data stored by a storage device, can be accessed by processor 502 during the execution of computer executable process steps to instruct one or more components within the image processing system 500 .
- Programming and/or loading executable instructions onto memory 508 and processor 502 in order to transform the image processing system 500 into a non-generic particular machine or apparatus that applies VTIAC to surgical procedures is well-known in the art.
- Implementing instructions, real-time monitoring, and other functions by loading executable software into a computer and/or processor can be converted to a hardware implementation by well-known design rules and/or transform a general-purpose processor to a processor programmed for a specific application. For example, decisions between implementing a concept in software versus hardware may depend on a number of design choices that include stability of the design and numbers of units to be produced and issues involved in translating from the software domain to the hardware domain.
- a design may be developed and tested in a software form and subsequently transformed, by well-known design rules, to an equivalent hardware implementation in an ASIC or application specific hardware that hardwires the instructions of the software.
- a machine controlled by a new ASIC is a particular machine or apparatus, likewise a computer that has been programmed and/or loaded with executable instructions may be viewed as a non-generic particular machine or apparatus.
- FIG. 8 illustrates that the processor 502 may be operatively coupled to an input interface 504 configured to obtain one or more images and output interface 506 configured to output and/or display the images.
- the input interface 504 may be configured to obtain one or more images via electrical, optical, and/or wireless connections using one or more communication protocols.
- the input interface 502 may be a network interface that comprises a plurality of ports configured to receive and/or transmit data via a network.
- the network may transmit image data via wired links, wireless link, and/or logical links.
- the input interface 504 may include but are not limited to a keyboard, universal serial bus (USB) interfaces, CD-ROMs, DVD-ROMs and/or graphical input devices (e.g., onscreen and/or virtual keyboards).
- the output interface 506 may be an interface used to display information in a readable format for a user and/or used to transmit information to a separate apparatus or machine. Examples include, but are not limited to, a graphic display (e.g., monitors and display screens), a user interface, an interface used to connect to a printing device configured to produce hard-copies of the generated results, and output ports used to connect to a network and/or another computing device.
- Image processing system 500 may also include computing components not explicitly shown in FIG. 8 , but well-known in the art, such as one or more power supplies, network interface(s), audio interfaces, displays, and circuitry used to connect the processor 502 , input interfaces 504 , output interface 506 , and memory 508 .
- internal architecture 900 of a computing device(s), computing system, computing platform and the like includes one or more processing units, processors, or processing cores, (also referred to herein as CPUs) 912 , which interface with at least one computer bus 902 .
- Computer-readable medium, or media, 906 , network interface 914 , memory 904 e.g., random access memory (RAM), run-time transient memory, read only memory (ROM), media disk drive interface 920 as an interface for a drive that can read and/or write to media including removable media such as floppy, CD-ROM, DVD, media, display interface 910 as interface for a monitor or other display device, keyboard interface 916 as interface for a keyboard, pointing device interface 918 as an interface for a mouse or other pointing device, and miscellaneous other interfaces 922 not shown individually, such as parallel and serial port interfaces and a universal serial bus (USB) interface.
- RAM random access memory
- ROM read only memory
- media disk drive interface 920 as an interface for a drive that can read and/or write to media including removable media such as floppy, CD-ROM, DVD, media
- display interface 910 as interface for a monitor or other display device
- keyboard interface 916 as interface for a keyboard
- pointing device interface 918 as
- Memory 904 interfaces with computer bus 902 so as to provide information stored in memory 904 to CPU 912 during execution of software programs such as an operating system, application programs, device drivers, and software modules that comprise program code, and/or computer executable process steps, incorporating functionality described herein, e.g., one or more of process flows described herein.
- CPU 912 first loads computer executable process steps from storage, e.g., memory 904 , computer readable storage medium/media 906 , removable media drive, and/or other storage device.
- CPU 912 can then execute the stored process steps in order to execute the loaded computer-executable process steps.
- Stored data e.g., data stored by a storage device, can be accessed by CPU 912 during the execution of computer-executable process steps.
- Persistent storage e.g., medium/media 906
- Persistent storage can be used to store an operating system and one or more application programs.
- Persistent storage can also be used to store device drivers, such as one or more of a digital camera driver, monitor driver, printer driver, scanner driver, or other device drivers, web pages, content files, playlists and other files.
- Persistent storage can further include program modules and data files used to implement one or more embodiments of the present disclosure.
- a network link typically provides information communication using transmission media through one or more networks to other devices that use or process the information.
- the network link may provide a connection through a local network to a host computer or to equipment operated by a Network or Internet Service Provider (ISP).
- ISP equipment in turn provides data communication services through the public, worldwide packet-switching communication network of networks now commonly referred to as the Internet.
- a computer called a server host connected to the Internet hosts a process that provides a service in response to information received over the Internet.
- server host hosts a process that provides information representing video data for presentation at display 910 . It is contemplated that the components of system 900 can be deployed in various configurations within other computer systems, e.g., host and server.
- At least some embodiments of the present disclosure are related to the use of computer system 900 for implementing some or all of the techniques described herein. According to one embodiment, those techniques are performed by computer system 900 in response to processing unit 912 executing one or more sequences of one or more processor instructions contained in memory 904 . Such instructions, also called computer instructions, software and program code, may be read into memory 904 from another computer-readable medium 906 such as storage device or network link. Execution of the sequences of instructions contained in memory 904 causes processing unit 912 to perform one or more of the method steps described herein. In alternative embodiments, hardware, such as ASIC, may be used in place of or in combination with software. Thus, embodiments of the present disclosure are not limited to any specific combination of hardware and software, unless otherwise explicitly stated herein.
- the signals transmitted over network link and other networks through communications interface carry information to and from computer system 900 .
- Computer system 900 can send and receive information, including program code, through the networks, among others, through network link and communications interface.
- a server host transmits program code for a particular application, requested by a message sent from computer, through Internet, ISP equipment, local network and communications interface.
- the received code may be executed by processor 902 as it is received, or may be stored in memory 904 or in storage device or other non-volatile storage for later execution, or both.
- a module is a software, hardware, or firmware (or combinations thereof) system, process or functionality, or component thereof, that performs or facilitates the processes, features, and/or functions described herein (with or without human interaction or augmentation).
- a module can include sub-modules.
- a module, or software components of a module may be stored on a computer readable medium for execution by a processor. Modules may be integral to one or more servers, or be loaded and executed by one or more servers. One or more modules may be grouped into an engine or an application.
Landscapes
- Engineering & Computer Science (AREA)
- Health & Medical Sciences (AREA)
- Medical Informatics (AREA)
- Surgery (AREA)
- Public Health (AREA)
- Life Sciences & Earth Sciences (AREA)
- General Health & Medical Sciences (AREA)
- Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
- Theoretical Computer Science (AREA)
- Biomedical Technology (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Heart & Thoracic Surgery (AREA)
- Molecular Biology (AREA)
- Animal Behavior & Ethology (AREA)
- Veterinary Medicine (AREA)
- Epidemiology (AREA)
- Primary Health Care (AREA)
- Robotics (AREA)
- General Engineering & Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Pathology (AREA)
- Human Computer Interaction (AREA)
- Radiology & Medical Imaging (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Urology & Nephrology (AREA)
- Computer Graphics (AREA)
- Computer Hardware Design (AREA)
- Software Systems (AREA)
- Oral & Maxillofacial Surgery (AREA)
- Geometry (AREA)
- Physical Education & Sports Medicine (AREA)
- Biophysics (AREA)
- Endoscopes (AREA)
- Image Processing (AREA)
- Length Measuring Devices By Optical Means (AREA)
Abstract
Disclosed is a concept for computer-assisted procedures of surgery and diagnosis that target rigid, non-deformable anatomical parts such as bone, tissue, or teeth. The disclosure describes attaching small visual markers to instruments and anatomy of interest (e.g. bone surface), with each marker having a printed known pattern for detection and unique identification in images acquired by a free-moving camera, and a geometry that enables estimating its rotation and translation with respect to the camera using solely image processing techniques.
Description
This patent application is a Reissue Patent Application of U.S. Pat. No. 10,499,996, filed on Sep. 26, 2017 as U.S. patent application Ser. No. 15/561,666, which is a U.S. national phase application of PCT International Patent Application No. PCT/US2016/024262, filed on Mar. 25, 2016, which claims the benefit of U.S. Provisional Patent Application Ser. No. 62/138,529, filed on Mar. 26, 2015 and titled “Methods and Systems for Computer-Aided Navigation in Surgical Procedures”, and U.S. Provisional Patent Application Ser. No. 62/255,513, filed on Nov. 15, 2015 and titled “Methods and Systems for Computer-Aided Navigation in Surgical Procedures”, all of which are hereby incorporated by reference in their entireties.
Not applicable.
The disclosure generally relates to the field of computer-aided surgery, and in particular, but not by way of limitation, the disclosed embodiments refer to computer aided-navigation in camera guided procedures of surgery and diagnosis in anatomical regions with rigid tissues such as bone, which includes arthroscopy of knee, hip, or shoulder, and open surgery in orthopedics and dentistry in which case a camera must be used to observe the operating field. One or more embodiments can also be employed in any other application domain, such as industrial inspection, that uses a camera system to visualize a work space that comprises rigid, non-deformable parts.
Minimally Invasive Surgical (MIS) procedures aim to minimize damage to healthy tissue by accessing targeted organs and anatomical cavities through relatively small size incisions. Since the workspace is not fully exposed, the surgeon typically carries the medical procedure using as guidance video acquired by a camera system that is inserted into the cavity. MIS procedures are being increasingly adopted in different medical specialties, such as orthopedics, abdominal surgery, urology, neurosurgery, and ENT, just to name a few.
Arthroscopy is a MIS procedure for treatment of damaged joints in which instruments and endoscopic camera (the arthroscope) are inserted into the articular cavity through small incisions (the surgical ports). Arthroscopy, as opposed to conventional open surgery, largely preserves the integrity of the articulation, which is beneficial for the patient in terms of reduction of trauma, risk of infection and recovery time. Unfortunately, arthroscopic procedures are relatively difficult to execute because of indirect visualization and limited maneuverability inside the joint, with novices having to undergo a long training period and experts often making mistakes of clinical consequences. This is a scenario where computer-assistive technologies for safely guiding the surgeon throughout the procedure can make a difference, both in terms of improving clinical outcome and in terms of decreasing the surgeon learning curve.
Depending on the particular clinical application, a system for Computer-Aided Surgery (CAS) comprises two distinct stages: (i) an offline step in which the procedure is planned leading to some sort of computational model that can either be a three-dimensional (3D) pre-operative image of the patient's organ (e.g. CT-Scan), a statistical bone model, or a set of guidelines for inferring meaningful locations with respect to anatomical landmarks; and (ii) an intra-operative navigation step in which the computer guides the surgeon throughout the procedure for the execution to be done as defined.
The intra-operative navigation usually passes by overlying the pre-operative computational model with the actual bone, and by localizing in real-time the tools and instruments with respect to each other, and with respect to the targeted organ. Typically, the technology to accomplish this task is Optical-Tracking (OT) that consists in using a stationary stereo head, henceforth called base station, for tracking a set of markers that are rigidly attached to instruments and/or bone. The stereo head comprises two infrared (IR) cameras that track a set of point markers that are rigidly attached to the object of interest. The position of each marker is estimated by simple triangulation and, since their relative arrangement is known ‘a priori’, the 3D pose of the object of interest is computed in the reference frame of the base station. Recently, a technological variant of OT was introduced in which the two IR cameras are replaced by two conventional video cameras operating in the visible spectrum, and the arrangements of IR markers are replaced by planar markers with printed known patterns.
The surgical navigation solutions that are currently available for Orthopedics, Neurosurgery, and ENT invariably rely in OT. In generic terms, the typical workflow passes by the surgeon to rigidly attach a tool marker to patient and/or targeted organ, which is followed by pin pointing anatomical landmarks with a calibrated tracked probe. The 3D position of these landmarks is determined in the coordinate system of the base station and the pre-operative computational model is registered with the patient. From this point on, it is possible to determine in real-time the pose of instruments with respect to patient and plan, which enables the system to safely guide the surgeon throughout the procedure. There are some variants to this scheme that mainly address the difficulties in performing the 3D registration of patient's anatomy with a pre-operative model with a tracked probe that tends to be an error prone, time consuming process. For example, the O-arm from Medtronic® combines OT with a CT-scanner that enables the acquiring of the 3D pre-operative model of patient's anatomy in the Operating Room (OR) before starting the procedure, which avoids the surgeon performing explicit registration. The system that is being developed by 7D Surgical® goes in the same direction with the 3D model being obtained using multi-view reconstruction and structured light to avoid the ionizing radiation of CT-scanning. Nevertheless, these systems still rely in conventional OT to know the relative position between instruments and anatomy after registration has been accomplished.
OT has proved to be an effective way of obtaining real-time 3D information in the OR, which largely explains the fact of being transversally used across different systems and solutions. However, the technology has several drawbacks that preclude a broader dissemination of surgical navigation: (i) it requires a significant investment in capital equipment, namely in acquiring the base station; (ii) it disrupts normal surgical workflow by changing the OR layout to accommodate additional equipment, by forcing the surgeon to work with instruments with bulky tool markers attached, and by constraining the team movements due to the need of preserving lines of sight between base station and tool markers; and (iii) it is not well suited to be used in MIS procedures because organs and tissues are occluded which avoids placing marker tools that can be observed from the outside by the base station. For example, OT based navigation in arthroscopic procedures always requires opening additional incisions such that the marker tool attached to the bone protrudes through patient skin.
In recent years some alternative technologies have emerged in an attempt of obviating the above-mentioned drawbacks. Electromagnetic Tracking (ET) is currently used in some surgical navigation systems with the advantage of not requiring preservation of a line of sight. However, it has the problem of being vulnerable to electromagnetic interference caused by nearby metals and devices, being in practice less reliable and accurate than OT. Moreover, it still requires additional capital equipment, namely a base station, and the need of attaching coil markers with hanging wires to organs makes it non amenable to MIS procedures.
The embodiments in the disclosure provide a new concept for computer-assisted procedures of surgery and diagnosis that target rigid, non-deformable anatomical parts such as bone, tissue, or teeth. The disclosure describes attaching small visual markers to instruments and anatomy of interest (e.g. bone surface), with each marker having a printed known pattern for detection and unique identification in images acquired by a free-moving camera, and a geometry that enables estimating its rotation and translation with respect to the camera using solely image processing techniques.
The concept, henceforth referred as Visual-Tracking Inside the Anatomical Cavity (VTIAC), introduces three main differences from other embodiments of OT/ET in the context of computer-aided surgery: First, the global world reference frame, instead of being the coordinate frame of the external base station, it is the system of coordinates of a marker that is rigidly attached to the anatomy of interest (e.g. bone surface). This marker, referred to herein as World Marker or WM, serves as absolute reference such that all measurements are expressed in its coordinates (world coordinates); Second, the free-moving camera acts as the single sensing modality with all measurements and real-time 3D inferences being carried by processing the acquired video. This feature avoids significant investments in additional capital equipment when compared with OT/ET; and Third, since measurements are performed in high resolution images acquired at close range, the metric accuracy of VTIAC is significantly better than the one accomplished with OT/ET.
The disclosure discloses the apparatus for VTIAC and the required initial calibration procedures, it describes how to use VTIAC to perform very accurate 3D measurements inside the anatomical cavity, and it shows how to use augmented reality, virtual reality, or robotics to provide real-time guidance to the surgeon after registering a pre-operative 3D plan.
In terms of clinical applications VTIAC is specially well suited for arthroscopy where the already existing monocular arthroscope acts as the free-moving camera that provides the video input. VTIAC can be successfully employed in any clinical procedure that targets anatomical regions with rigid parts, such as open orthopaedic surgery or dentistry, in which case the operating field must be observed by a camera that can either be attached to a tool or handheld. The disclosure describes illustrative implementations in knee arthroscopy and spine surgery that by no means limit the range of possible clinical applications.
For a more complete understanding of the present disclosure, reference is made to the following detailed description of exemplary embodiments considered in conjunction with the accompanying drawings.
1. Introduction
It should be understood that, although an illustrative implementation of one or more embodiments are provided below, the various specific embodiments may be implemented using any number of techniques known by persons of ordinary skill in the art. The disclosure should in no way be limited to the illustrative embodiments, drawings, and/or techniques illustrated below, including the exemplary designs and implementations illustrated and described herein.
One or more embodiments disclosed herein applies to camera-guided orthopedic MIS procedures, namely arthroscopy, that is used as illustrative example throughout most of the description. However, the application of the presently disclosed embodiments can include other surgical procedures and clinical specialties where the operating field comprises rigid, non-deformable parts and surfaces. The application of the disclosed embodiments requires a camera system for visualizing the anatomical scene that might already exist (e.g. arthroscopy) or be added (e.g. open orthopedic surgery).
One or more embodiments in the disclosure provide a surgical navigation scheme for arthroscopy and other procedures using a conventional camera and with scenes that comprise rigid surfaces. The surgical navigation scheme will be referred to as Visual-Tracking Inside the Anatomical Cavity (VTIAC). The disclosure relates to attaching small, recognizable visual markers to instruments and rigid anatomy (e.g. bones) and use the free-moving camera, that is the arthroscope in case of arthroscopic procedures, to estimate their relative rotation and translation (the relative 3D pose). For the case of the markers being planar with a printed known pattern, the relative 3D pose is determined by estimating the plane-to-image homography that is factorized to obtain the rotation and translation between plane and camera reference frames. The marker attached to the bone surface, referred to herein as World Marker (WM), serves as absolute reference with all measurements being expressed in its coordinate system (world coordinates). VTIAC can be used to obtain 3D information about the bone surface, register a pre-operative computational model, and ultimately solve the navigation issues by providing guidance using augmented reality, virtual reality, or robotic actuation.
VTIAC introduces many differences relatively to other embodiments of OT/ET in the context of computer-aided surgery in general and arthroscopy in particular. For example, the global world reference frame, instead of being the external stereo head (the base station), is substituted by the system of coordinates of the WM that is inside the articular cavity. This avoids issues related to preserving lines of sight in the OR, as well as the need of having marker tools protruding through patient skin. Second, for example, the approach relies on processing the video acquired by a free-moving camera, which means that in the case of arthroscopy there is no need of investing in additional capital equipment that provides alternative sensing modalities. Third, for example, measurements are performed in the images acquired at close range inside the anatomical cavity, which dramatically increases spatial and/or metric accuracy with respect to OT or ET.
1.1 Prior Art
In embodiments where the visual marker is a planar marker, the plane-to-image homography may be a factor in the VTIAC approach for surgical navigation. The projection of a plane into a perspective image may be described by a 3×3 matrix transformation (the homography) that encodes the plane rotation and translation (the plane 3D pose) in camera coordinates. The homography has been broadly used in the field of Computer Vision for several different purposes, ranging from camera calibration to visual tracking, and passing by 3D motion estimation.
The use of plane homographies in clinical setups has been relatively scarce. For example, an OT system, the MicronTracker® developed by Claronav®, may use planes with recognizable patterns as tool markers. These markers are tracked by a stereo camera system and the pose of the tool is determined through homography factorization. The approach herein described differs from MicronTracker® in that the tracking is performed by a moving monocular camera as opposed to a stationary stereo setup. Moreover, while in MicronTracker® the base station is the external stereo setup, which raises the issues about line of sight inherent to conventional OT, in VTIAC, measurements are carried out with respect to the WM that is rigidly attached to the surface inside the articular joint or anatomical cavity.
Other embodiments may be used to determine the relative pose between a laparoscope and an intra-operative ultrasound (US) probe or laser projector. In particular, the embodiments attach a printed planar pattern to the probe and/or projector that is viewed by the laparoscope. This enables estimation of the plane-to-image homography and determination of the relative pose of the probe and/or projector in camera coordinates. VTIAC provides a much broader range of functionalities that arise from using a World Marker (WM) attached to the bone surface. Thus, VTIAC not only provides the relative pose of tools and devices that are inserted into the anatomical cavity, but it also enables the reconstruction of points and contours on the surface of the organ of interest that are pin-pointed by the surgeon. This information can be used for a multitude of purposes such as metric measurements, registration of pre-operative models, or guidance using augmented reality, that are seamlessly supported by the framework. Moreover, measurements are typically represented in camera coordinates, which means that it is not possible to relate or integrate information across frames because the laparoscope is in constant motion. In VTIAC, all measurements are stored in the coordinate system of the WM that works as an absolute reference across time and space. Thus, the visual tracking process can even be discontinued, and the 3D information obtained till that moment becomes readily available as soon as the WM is redetected in the images acquired by the moving camera.
1.2 Structure and Notation
In order to better illustrate the usefulness of VTIAC, two embodiments that can be applied to design a navigation system for the arthroscopic reconstruction of the Anterior Cruciate Ligament (ACL) and for Placing Pedicle Screws (PPS) in spine surgery are presented (sections 8 and 9). These procedures are mere examples that do not limit in any way the potential applications of VTIAC. As stated in the following sections, the VTIAC can be applied to a multitude of arthroscopic procedures, as well as open procedures and including dentistry surgery.
Notation: If not stated otherwise, points are represented by their vectors of coordinates and vectors are denoted by a bold letter (e.g., P, x). The rigid displacement between coordinate frames is represented by a 4×4 matrix in the Special Euclidean Group (SE(3)) where the left upper 3×3 submatrix is a rotation matrix and 3×1 right upper submatrix is a translation vector. Matrices are typically denoted by plain capital letters (e.g., C, T).
2. Overview of Visual-Tracking Inside the Anatomical Cavity (VTIAC)
The free-moving camera is assumed to be calibrated such that image points u in pixel coordinates can be mapped into image points x in metric coordinates as if the image had been acquired by a perfect pin-hole. For the sake of simplicity, and without lack of generality, it is considered that the free-moving camera is an arthroscopic camera and that the anatomical part of interest is a bone. It is also assumed that visual markers are planar with a known pattern.
After accessing the anatomical cavity, the surgeon starts by rigidly attaching a marker to the bone surface that is referred as the World Marker (WM). If the marker is planar, then its projection is described by an homography HC, that maps plane points into image points, and encodes the relative rotation RC and translation tC between marker and camera reference frames. Thus, and since HC can be estimated from image information, it is possible to use this homography relation to determine at every frame time instant the 4×4 matrix C that transforms world coordinates into camera coordinates (FIG. 1A ).
Consider now an instrument or tool with a similar visual marker attached that is referred as Tool Marker (TM). Repeating the process of the previous paragraph the homography HT can be estimated from image information in order to determine the rigid transformation that maps TM coordinates into camera coordinates. If both WM and TM are simultaneously visible in the image, then it is possible to estimate the 3D poses of world and tool markers in the camera frame and find in a straightforward manner the location T of the tool or instrument in the world coordinate system (FIG. 1B )
T=C−1{circumflex over (T)} (equation 2)
T=C−1{circumflex over (T)} (equation 2)
Let's now assume that the tool or instrument is a calibrated touch-probe such that PT is the vector of 3D coordinates of its tip in the TM reference frame. The surgeon can reconstruct a point of interest in the bone surface by touching it with the probe and acquiring a frame where both WM and TM are visible. This enables computation of the pose T of the probe and the obtaining of the point of interest P expressed in world coordinates (FIG. 1C ).
The process above can be applied to successive frames in order to reconstruct a curve in the bone surface. In this embodiment the surgeon outlines the contour of interest while keeping both WM and TM in the Field-of-View (FOV) of the free-moving camera. This enables the obtaining of successive P estimates that define the desired 3D curve. Since 3D reconstruction results are stored in World Marker coordinates, the action of outlining can be stopped and resumed at any time. If the process is interrupted for any reason, it suffices for the camera to see again the WM for all the 3D information to be restored without having to repeat the tedious touching process (FIG. 1C ).
The 3D reconstruction results, that can either be points, contours, or sparse surface meshes, can be used for the purpose of measuring, estimating shape, or overlying a pre-operative plan in the actual patient anatomy (3D registration). This pre-operative plan can be a set of rules using anatomical landmarks, a statistical 3D model of the anatomy of interest, or an actual 3D image of the organ (e.g. CT Scan) augmented with guidance information inserted by the surgeon (surgical plan). Let's assume the latter for illustrative purposes (FIG. 1D ). In this case a suitable 3D registration algorithm is selected for estimating the rigid transformation M that maps points PM in the pre-operative image into corresponding points P in the intra-operative reconstruction obtained with VTIAC (FIG. 1D ). This enables representing the information of the model, including guidance information, in the system of coordinates of the world marker or, in other words, to overlay the pre-operative plan with the patient's anatomy (FIG. 1E ).
The clinical execution might require, in one embodiment, multiple different instruments—such as guides, drills, shavers, saws, burrs, etc.—that can either be used in sequence or simultaneously. Each one of these instruments is assumed to have a Tool Marker (TM) attached that defines a local system of coordinates where the instrument's relevant parts—such as tip, symmetry axis, or even complete CAD model—are represented. The system processes each frame with the objective of detecting, identifying, and estimating the 3D pose of every TM that is in the FOV of the camera. If the WM is also visible in image, then it is possible to determine the pose of the camera C, locate the instruments in the world coordinate system, relate their poses T with the 3D information stored in the WM reference frame, and ultimately provide real-time assistance to the surgeon (FIG. 1E ).
Thus, the last stage of VTIAC consists of assisting the surgeon by performing continuous processing of the video for estimating in real-time the 3D pose of instruments with respect to patient anatomy and/or surgical plan represented in WM coordinates. The assistance can take multiple forms depending on a specific task and a preferred user interface. Possibilities include overlaying guidance information in video using Augmented Reality (AR), using computer graphics to animate the motion of instruments in a Virtual Reality (VR) environment showing the patient's anatomy and/or surgical plan, or controlling the action of actuators in the case of procedures assisted by robotic systems such as the Mako® or the Navio® robots.
3. Overview of Methods, Apparatus and Initial Calibration Requirements
This section overviews the methods and apparatus that are required to perform computer-aided surgery using VTIAC. The apparatus includes:
-
- (i). A free-moving camera, that can either be an arthroscopic camera or a generic handheld camera, and that is calibrated at all times such that image points in pixel units can be back-projected into directions or image points expressed in the metric coordinate system of the camera (
FIG. 2A ); - (ii). A visual marker, henceforth referred as the World Marker or WM, consisting in an object that is meant to be rigidly attached to a surface, that comprises at least one planar facet with a known pattern that can be secured (e.g., glued), printed or engraved, and where this pattern defines the world system of coordinates (
FIG. 2B ); - (iii). A touch-probe that is a tool or instrument of any material comprising a handgrip and a tip, and that has at least one visual marker, henceforth referred as Tool Marker or TM, that consists in at least one planar facet with a known pattern that can be secured (e.g., glued), printed or engraved, and where the pattern defines a local system of coordinates in which the position of the tip PT is known (
FIG. 2C ); - (iv). The surgical instruments for proper execution of the clinical procedure, where each instrument has at least one visual marker, henceforth referred as Tool Marker or TM, that includes at least one planar facet with a known pattern that can be secured (e.g., glued), printed or engraved, and where the pattern defines a local system of coordinates in which the position of instrument's relevant parts—such as tip, symmetry axis, or even a complete CAD model of the instrument—are known (
FIG. 2D ); - (v). An image processing system, that may correspond to or may be part of a computer and/or any other computing device, that receives as input images and video acquired by the camera (i), as well as commands that may be entered by a button panel, keyboard, camera buttons, foot switch and/or any other input interface, and that outputs the processing results to a display and/or to a robotic actuator (
FIG. 2E );
where this apparatus is used in the following actions or methods: - (vi). Placement of the WM in an arbitrary location in the surface of the rigid anatomical part of interest, with this placement being such that WM and part of interest do not move with respect to each other.
- (vii). 3D reconstruction of points and/or contours in the surface of the rigid anatomical part by using the touch-probe to pin-point those points and/or outline those contours while keeping both WM and TM of the probe in the Field-of-View (FOV) of the camera (
FIG. 1C ). - (viii). Application of 3D reconstruction results for the purpose of measuring, making inference, or overlaying a pre-operative surgical plan with the current patient's anatomy, in which case a suitable 3D registration method may be used for determining the rigid displacement between the coordinate system of the WM, that is attached to the anatomy of interest, and the reference frame of the surgical plan (
FIG. 1D ). - (ix). Assisted clinical execution using the surgical instruments of (iii) where the camera simultaneously observes the TMs and the WM for locating the instruments in world coordinates, and where real-time guidance is accomplished by relating these locations with 3D reconstructions results and/or overlaid surgical plan (
FIG. 1E ).
3.1 Calibration of Free-Moving Camera
- (i). A free-moving camera, that can either be an arthroscopic camera or a generic handheld camera, and that is calibrated at all times such that image points in pixel units can be back-projected into directions or image points expressed in the metric coordinate system of the camera (
Since the VTIAC uses images for measurements and 3D inference, the free-moving camera must be calibrated at all times during the procedure such that 2D image points u, represented in pixel coordinates, can be mapped into 2D points x (or back-projection directions) represented in the metric system of coordinates of the camera. The calibration includes determining the vector of parameters k and ξ of the back-projection function f−1 (the inverse of the projection function f) where k comprises the so-called intrinsic parameters—focal length, principal point, aspect ratio, and skew—and ξ stands for the radial distortion parameters.
x=f−1(u; k, ξ) (equation 4)
x=f−1(u; k, ξ) (equation 4)
The camera can either be pre-calibrated from factory, using any standard method in literature, or calibrated in the Operating Room (OR) just before starting the procedure. The latter is especially recommendable for the case of arthroscopic cameras, or any other camera with exchangeable optics. The calibration in the OR can be quickly accomplished by acquiring one image of a known calibration pattern from an arbitrary viewpoint, as described in U.S. Patent Publication No. 2014/0285676, which is incorporated by reference in its entirety. If the camera parameters change during operation because the surgeon rotates the lens scope and/or varies the optical zoom, then the initial calibration may be updated at every frame time using the techniques described in U.S. Patent Publication No. 2014/0285676 and Patent Publication WO2014054958, both of which are incorporated by reference in their entireties. The camera calibration must also take into account the medium of operation that, in the case of arthroscopy, is a wet medium. In this situation the initial single image calibration can either be carried in wet medium, or performed in air followed by compensating for the difference in the refractive index of air and water-based medium.
3.2 World Marker (WM) and Tool Markers (TMs).
The surgeon starts by fixing the World Marker (WM) to the bone surface. The WM can be any object comprising at least one planar facet with a known pattern that can be secured (e.g., glued), printed or engraved, and that can be recognized in images; that is small enough to be inserted into the anatomical cavity (e.g., up to 5 mm diameter in the case of arthroscopy); and that can be mechanically attached to the surface such that bone and marker do not move with respect to each other.
A non-exhaustive list of objects that can be used as WM includes: a screw-like object with a flat head or facet (FIG. 2B ); a nail-like object to be fixed by pressure with a flat head or facet; a needle like object with a flat lateral facet for trans-dermic insertion into the joint or cavity; or a flat button-like object that is pulled inside the joint or cavity by a thread or guide.
The touch-probe in (iii) and the surgical tools in (iv) are instrumented with a visual marker (the Tool Marker or TM), which can either be originally built-in at manufacturing time, or rigidly attached by the user (FIG. 2C , FIG. 2D ). Depending on the tool purpose, the tip of the tool, the orientation of the tool, or a complete CAD model of the tool may be registered in the TM coordinate frame (FIG. 2C ). This registration process as described herein is referred to as tool calibration that can either be carried in factory for built-in markers, or performed by the user in case the markers are attached to the tool before starting the procedure.
3.3 Tool Calibration in the Operating-Room (OR)
If the tool calibration includes finding the coordinates PT of a particular tool point in the TM reference frame (e.g. the tip of the touch probe (iii)), then the operation can be quickly carried simultaneously with the initial calibration of the camera without requiring the acquisition of additional calibration frames. As described, e.g., in U.S. Patent Publication No. 2014/0285676, the camera calibration can be accomplished by acquiring a single image of a known grid or checkerboard pattern. This enables recovering the intrinsic parameters k, the radial distortion parameters ξ, and the rigid transformation Ĝ that maps coordinates in the grid reference frame into coordinates in the camera reference frame. Thus, if the tool tip is placed in a pre-defined point PG that is known in grid coordinates, and the calibration image is such that TM is visible, then it is possible to estimate the 3D pose {circumflex over (T)} of the tool marker from image information and obtain the TM coordinates of the tool tip by applying the formula below (FIG. 2F ).
The tool calibration of the surgical instruments (iv) can either consist in determining the location of a point, a line or axis, or a CAD model in the coordinate system of the TM attached to the particular instrument. This can be accomplished with the help of the calibrated camera and touch-probe using a method similar to the one used for 3D reconstruction on the bone surface, but where the role of the WM is replaced by the TM of the instrument (FIG. 1C ). Thus, for the case of a single point it is enough to pin-point it with the probe while keeping both the TM and the marker of the probe in the camera FOV. For the case of a line or axis the procedure is performed at least two times to reconstruct two points in TM coordinates lying on the line or axis. Finally, if the objective is to register a CAD model of the tool, then the procedure may be performed at least three times to obtain three landmark points in TM coordinates to be used as input in a standard registration method.
3.4 Alternatives and Extensions in the Physical Configuration of Visual Markers
The visual marker used in the WM of (ii) and in the TMs of (iii) and (iv) can comprise a single plane facet with a known pattern as assumed so far, or multiple plane facets with each facet having its own pattern that can be secured (e.g., glued), printed, or engraved, and where the location of each planar pattern is known in a common local coordinate system of the visual marker. The advantage of having multiple planar patterns facing different directions is to extend the range of viewing positions and orientations from which the marker can be observed by the camera for estimating the relative 3D pose (FIG. 1A , FIG. 1B ). In the case of TMs, the planar patterns can even be spread across different locations in the tool surface, in which case it suffices for the camera to see one of those patterns to successfully compute the relative pose {circumflex over (T)}.
Alternatively, the visual marker can be non-planar, in which case it should comprise n≥3 points with known coordinates in the local reference frame of the marker, with these points being such that they can be detected and identified in image in order to allow estimation of the relative pose by applying a Perspective-n-Point (PnP) method.
4. Estimation of Rotation and Translation (the 3D Pose) of a Known Planar Pattern from Image Information.
The small visual markers that are attached to instruments, tools, and anatomy of interest play a fundamental role in VTIAC being key-enablers for using the camera as a measuring device for determining 3D pose. As discussed, the visual marker can have different topological configurations but, for the sake of simplicity and without compromising generality, it will be assumed that the visual marker is a planar surface with a known pattern.
This planar pattern should be such that it has a local system of coordinates, it is amenable to be detected and uniquely identified from its image projection, and it has fiducial points that can be accurately detected in image for estimating the plane-to-image homography H from point correspondences. A point correspondence is the association between a point in the pattern p expressed in local coordinates and its projection x represented in camera coordinates. The homography H is a projective transformation that maps the former into the latter, and that can be linearly estimated from N≥4 point correspondences. The homography encodes the rotation and translation between pattern and camera coordinate systems, which means that the factorization of H provides the 3D pose of the pattern in the camera reference frame.
There are several pattern designs that meet the above mentioned conditions. It will be assumed, without compromising generality, that the planar patterns are similar to the CalTag checkerboard patterns, where the quadrilateral shape and high contrast enable fast detection, the sharp corners provide accurate point correspondences, and a bitmap binary code allows visual identification (FIG. 3A ). These patterns are broadly used as fiducial markers for applications in augmented reality, for which there are several image processing pipelines such as the ARToolKit or the ALVAR. These pipelines implement the steps of detection, identification, and homography estimation in a computationally efficient manner to provide the 3D pose of each planar pattern at every frame time instant. Unfortunately, the application to computer-aided surgery is not straightforward because the close-range operation requires cameras with a wide FOV that typically introduce very strong radial distortion. The distortion hinders correct detection and point correspondences, which is not compatible with the high-accuracy demands of the medical applications (FIG. 3B ).
One possibility for improving accuracy and robustness of 3D pose estimation is to correct radial distortion via software, before running the processing pipeline for detection, identification, and homography/pose estimation. However, this has several drawbacks, such as the computational effort in warping the entire frame, and the fact that interpolation also introduces artifacts that degrade the accuracy of geometric estimation.
Since radial distortion has a relatively small impact in pattern detection, this disclosure provides an alternative approach based in photo-geometry. The approach includes using standard methods for detection, identification, and initial estimation of pattern rotation r0 and translation t0, followed by refining the 3D pose estimate by minimizing the photo-geometric error in aligning the current pattern image with its template using a warping function that takes into account the non-linear distortion.
Let C0 be the initial 3D pose estimate of the planar pattern in camera coordinates. The objective is to determine the pose update Δ, that encodes the increments in rotation δR and in translation δt, such that the photo-geometric error ε1 is minimized (FIG. 3C , FIG. 3D )
where T(u) is the pattern template, I(u) is the current frame, Ni is the image region comprising the pattern, and w is the image warping function (FIG. 3D ) given by
w(u; r, t)=f(x: k, ξ)o h(x; r, t)o f−1 (u; k′, ξ′) (equation 7)
w(u; r, t)=f(x: k, ξ)o h(x; r, t)o f−1 (u; k′, ξ′) (equation 7)
with h being the homography map that depends on the relative 3D pose r and t, and f denoting the projection function of the camera that encodes the effect of radial distortion, as described, e.g., in Patent Publication WO/2014054958. Since the template can be understood as a synthetic, fronto-parallel image of the planar pattern (FIG. 3D ), that has calibration parameters k′ and ξ′, the homography h depends on the rigid displacement of the views. The final 3D pose estimate is given by:
C=ΔC0 (equation 8)
C=ΔC0 (equation 8)
The iterative minimization of the photo-geometric error εi can be carried using different optimization schemes available in literature such as forward composition, inverse composition, or efficient second order minimization, which requires some changes in formulation and parametrization in SE(3). The formulation can also be extended to be resilient to changes in illumination.
5. 3D Measurement and Reconstruction using VTIAC.
Section 4 describes a method for estimating the 3D pose of a planar visual marker in camera coordinates. Let's consider two of these markers such that one is attached to the anatomy of interest (WM), and the other is attached to a calibrated touch probe (TM).
For reconstructing an arbitrary point P in world coordinates the surgeon places the tip of the probe in the point, positions the camera such that both WM and TM are in the FOV, and commands the system to acquire an image that is processed as follows (FIG. 4A ):
-
- (i). Detect, identify, and estimate the 3D pose C of the WM in camera coordinates using the method in section 4.
- (ii). Detect, identify, and estimate the 3D pose {circumflex over (T)} of the TM in camera coordinates using the method in section 4.
- (iii). Reconstruct point P in world coordinates by applying the formula
-
-
- with PT being the 3D coordinates of the probe tip expressed in the TM reference frame
- (iv). Store the reconstructed point P in memory for future reference.
-
The approach can be extended to obtain a 3D contour or a sparse 3D reconstruction of a surface region, in which case the surgeon uses the touch probe to respectively outline the contour or randomly grasp the surface, while the camera acquires continuous video and steps above are executed for each frame (FIG. 4B , FIG. 4C ).
The 3D reconstruction results are stored in memory in world coordinates, which means that they can be overlaid in images whenever the WM is in the camera FOV by performing the following steps at each frame time instant (FIG. 4A , FIG. 4B , FIG. 4C ):
-
- Detect, identify, and estimate the 3D pose C of the WM in camera coordinates using the method in section 4.
-
Map 3D data from world coordinates into camera coordinates using C. - Project the 3D data into image using function f with camera calibration parameters k, ξ.
The ability of VTIAC to reconstruct and store in memory points, curves, and regions in the anatomy of interest (e.g. bone) has a multitude of purposes and/or possible clinical applications. A non-exhaustive list includes:
-
- measuring the distance between two points (
FIG. 4D ); - measuring thickness (e.g. of cartilage coating);
- measuring depth (e.g. of perforation or insertion);
- measuring volume of a protruding or sunken region (e.g. a confocal cartilage defect);
- finding shape and/or area of a region by determining its boundary contour (
FIG. 4E ); - fitting a parametric curve and/or shape;
- measuring the distance between two points (
As stated, the reconstruction results can also be used as input in standard 3D registration methods for aligning or overlying a computational model with the current patient's anatomy. Such methods estimate the rigid transformation M that maps points PM in the model into corresponding points P in the intra-operative reconstruction obtained with VTIAC (FIG. 1D ).
6. Assisted Execution of the Clinical Procedure using VTIAC.
So far we have shown how to obtain relevant 3D data in the common coordinate system of the WM that may consist in reconstruction results, measurements and other types of 3D inferences, or the registration of surgical plan against patient's anatomy. The term ‘surgical plan’ is employed in a broad sense and can mean, among other things, a set of rules based on anatomical landmarks, e.g. placing the femoral tunnel of the ACL at ⅓ the length of the notch ceiling measured from its posterior end; the fitting of a statistical model of an anatomy or pathology, e.g. the shape model of CAM femuro-acetabular impingement; or a pre-operative image of the targeted anatomy that can, or cannot, be augmented with guidance information, e.g. a CT scan annotated by the surgeon using a 3D planning software. This section describes how VTIAC can combine this 3D data with real-time 3D pose estimation of surgical instruments to provide intra-operative navigation features.
Let the surgical instrument—that can be a needle, guide, drill, shaver, saw, burr, or any other object required for proper clinical execution—have a TM attached. The marker defines a local reference frame where the position of a point, axis, or CAD model of the tool is known (calibrated tool). Navigation is accomplished by executing the following processing steps at every frame time instant:
-
- (i). Detect, identify, and estimate the 3D pose C of the WM in camera coordinates using the method in section 4.
- (ii). If the pose C has been successfully estimated proceed as follows:
- 1. Detect, identify, and estimate the 3D pose {circumflex over (T)} of the TM in camera coordinates using the method in section 4.
- 2. If pose {circumflex over (T)} of the surgical instrument is successfully estimated then
- Compute the 3D pose T of TM in WM
coordinates using equation 2. - Map the tool calibration information, that can be points, axes, or CAD models, into world coordinates using the rigid transformation T.
- Relate the tool calibration information with the 3D data stored in memory to make measurements and inferences for the purpose of real-time guidance (e.g. distances and angles between surgical instrument and guides in surgical plan)
- Compute the 3D pose T of TM in WM
- 3. Communicate guidance to the surgeon by either using Augmented Reality (AR), in which case info is overlaid in image using the camera pose C and projection function f, or by animating a Virtual Reality (VR) 3D model. otherwise inform user that navigation features are not active because WM is outside the camera FOV
VTIAC navigation also works for the case of multiple instruments being used in simultaneous, in which case each instrument has its own TM enabling parallel detection, identification, and estimation of 3D pose T.
The aiding features can take multiple forms depending on the particular task and/or surgical procedure. A non-exhaustive list of these features includes:
-
- Signaling target points in the anatomy for placement or insertion, e.g. indicating the ACL footprint where the femoral tunnel should be open (
FIG. 5A ); - Orienting a tool by providing the angle between its axis and the desired direction of insertion or placement, e.g. overlying the angle between current and desired orientation of a Kirschner wire (
FIG. 6C ); - Orienting a perforation instrument by anticipating the perforation trajectory, e.g. orienting the drill direction for Pedicle Screw Placement (
FIG. 6D ) or orienting the drill direction in ACL reconstruction such that tunnel joins the ACL footprint with the Lateral epicondyle (FIG. 5B ); - Guiding a perforation or insertion instrument by providing a view in VR where the optical axis of the virtual camera is aligned with the desired line of perforation or insertion, e.g. orienting the Kirschner wire during Pedicle Screw Placement (
FIG. 6E ). - Signaling a region or structure of tissue to be resected, as well as quantifying the amount of tissue to be resected, e.g. resection of cam (or pincer) femuroacetabular impingement (
FIG. 4E ) - Real-time measurement of distance or depth of insertion, e.g. insertion of Kirschner wire during Pedicle Screw Placement (
FIG. 6E ).
7. Extensions and Variations
7.1 Free-moving Camera Mounted in a Tool or Instrument:
- Signaling target points in the anatomy for placement or insertion, e.g. indicating the ACL footprint where the femoral tunnel should be open (
The disclosure has considered that camera and tool or instrument are two entities with independent motions. There are situations for which it might be advantageous to assemble the camera in the tool or instrument such that the two entities become a single rigid body. The assembly, that is henceforth referred as a Camera Tool or CamT, must be calibrated such that the position of the tool tip, axis of interest, or CAD model of the tool or instrument, is known in the reference frame of the camera. Depending on the particular clinical application the camera can be mounted in a multitude of possible tools ranging from a touch-probe to an impactor for cup placement during hip arthroplasty, passing by burrs and drills. In this setup where camera and tools are physically attached, their relative 3D pose is known, and as long as the camera sees the WM, it is possible to determine the 3D pose of the tool in the global system of coordinates of WM.
7.2 Single-image Calibration of CamT:
The CamT described above can either be pre-calibrated from factory, or calibrated in the OR from a single image of a known grid or checkerboard pattern. In this case the surgeon acquires the calibration frame by positioning the camera such that the pattern is visible in image and the tool tip touches a particular point PG whose coordinates are known in the coordinate system of the grid (FIG. 7C ). The image is used as input in the method that provides the camera intrinsic parameters k, the lens distortion ξ, and the rigid transformation Ĝ that maps coordinates in the grid reference frame into coordinates in the camera reference frame. The tool calibration is fully accomplished by finding the camera coordinates of the tool tip that are given by
7.3 Contactless Probe using a Laser Pointer:
Section 5 discloses a method for 3D reconstruction where the surgeon uses a calibrated touch-probe to pinpoint points of interest while the camera observes both the WM and the TM of the tool. There might be situations for which touching a particular location in the anatomy is difficult or even unfeasible. Examples include situations of limited access or poor maneuverability where the touch-probe cannot reach a particular location without occluding the WM. It is now disclosed an alternative probe that can replace the conventional touch-probe in the task of performing 3D reconstruction using VTIAC, and that has the advantage of avoiding the need of physical contact.
This alternative probe, henceforth referred as contactless probe, consists in a laser pointer that emits a collimated beam of visible light. The pointer has a visual marker attached—the Tool Marker or TM—and it is assumed to be calibrated such that the position of the line LT defined by the beam is known in TM coordinates.
For reconstruction the surgeon directs the laser pointer such that the beam becomes incident on the point of interest, and uses the camera to acquire an image where WM, TM, and point of light incidence are visible. The point is reconstructed in 3D by intersecting the line LT of the beam with the back-projection line Bx of the image point x where the point of light incidence is projected (FIG. 7D ). Thus, the image is processed as stated in Section 5 with the difference that step (iii) is replaced by the two following steps:
-
- 1. Detect the point of light incidence x in the image and determine the corresponding back-projection line Bx in camera coordinates.
- 2. Determine the 3D coordinates PT of the point of light incidence in the TM reference frame by making:
PT=LTΛ{circumflex over (T)}−1(Bx) - where {circumflex over (T)}−1 (Bx) denotes the back-projection line that is expressed in TM coordinates by the inverse of transformation T and A denotes the operation of intersecting two 3D lines.
7.4 Contactless Probe using a Time-of-Flight (TOF) device:
Contactless 3D reconstruction can also be accomplished using an Active Contactless Probe consisting in a Laser Rangefinder (LRF), or other equivalent device or technology relying on Time-of-Flight (ToF) principles, that is able to measure distances λ along the direction of the beam line LT. The LRF has a visual marker attached and it is assumed to be calibrated such that the origin and unit direction of measurement, that are respectively ST and dT, are known in the local reference frame of TM. For 3D reconstruction the surgeon orients the LRF such that the beam becomes incident with the point of interest in the anatomy, and acquires in a synchronous manner the distance measurement λ and an image where both WM and TM are visible. The point of interest can be outside the camera FOV (FIG. 7E ). The reconstruction is accomplished using the processing steps of Section 5 with the point PT of step (iii) being given by:
PT=ST+λdT
7.5 Using Multiple WMs to Extend the Range of Operation:
PT=ST+λdT
7.5 Using Multiple WMs to Extend the Range of Operation:
The World Marker or WM works as a global reference, which means that it must be viewed by the camera whenever the surgeon wants to use VTIAC for reconstruction or guidance purposes. There might be situations for which keeping the WM in the camera FOV can be difficult to accomplish in practice, either because the camera has a limited FOV, or because the region to cover is simply too broad or wide. This problem is solved by using multiple markers as shown in FIG. 7F . In order to increase the working region the surgeon can fix an auxiliary visual marker (WM′) on a convenient location on the surface of the anatomy, and move the camera such that both WM and WM′ lie in the FOV. A frame is acquired, the method of section 4 is applied to determine the 3D pose C and C′ of the two markers (FIG. 7F ), and the rigid transformation W that maps coordinates in the auxiliary marker into world coordinates is
W′=C−1C′
Since W′ enables to map information from WM into WM′ and vice-versa, it suffices for the camera to see one of the markers for the reconstruction and guidance functionalities of VTIAC to be readily available. The region of operation can be further extended by placing additional markers and repeating the step above to register them in world coordinates.
7.6 Using VTIAC with a Surgical Robot
W′=C−1C′
Since W′ enables to map information from WM into WM′ and vice-versa, it suffices for the camera to see one of the markers for the reconstruction and guidance functionalities of VTIAC to be readily available. The region of operation can be further extended by placing additional markers and repeating the step above to register them in world coordinates.
7.6 Using VTIAC with a Surgical Robot
Section 6 discloses a method for using VTIAC to assist the execution of a clinical procedure where the guidance information is provided by either overlying info in the images or video (AR), or by animating a VR model of anatomy and tools. In addition, VTIAC can also be used to guide or control the action of a surgical robot (FIG. 7G ).
A surgical system like the Navio® robot relies on conventional OT for determining in real-time the 3D pose between the robotized tool and patient's anatomy and/or surgical plan. VTIAC can be used as an alternative to conventional OT for providing the kinematic feedback required to control the robot in closed loop (FIG. 7G ). In this case the robot end-effector, which is the tool or device at the end of the robotic arm or chain, must be instrumented with a Tool Marker and its tip, axis, or CAD model must be registered in TM coordinates. The relative 3D pose between end-effector and patient's anatomy/surgical plan is determined as described in section 6 using video acquired by a camera that simultaneously sees the WM and TM.
8. Example of Application of VTIAC for Arthroscopic Reconstruction of Anterior Cruciate Ligament (ACL) in the Knee
This section discloses an embodiment of VTIAC based-navigation for Reconstruction of Anterior Cruciate Ligament (ACL) in the Knee, which can also be generalized for other arthroscopic procedures such as in the shoulder or hip.
ACL tear is a common pathology for which arthroscopy is the standard treatment (e.g., >300000 cases per year worldwide). The procedure includes replacing the torn ACL by a substitution graft that is pulled into the joint through a tunnel opened with a drill. Placing this tunnel in the correct anatomical position is crucial for the knee to fully recover its functionality. One technique is the transtibial (TT) approach that opens the tunnel in a single step by drilling from the bottom of the tibia plate till entering into the femur notch. Recent studies show that in about 39% of the cases TT fails in positioning the tunnel at the femoral end, and that much better results can be accomplished using the anteromedial (AM) approach. Unfortunately, AM is used in about 15% of the cases because it is more difficult to execute and increases the risk of critically short tunnel or blowout of the posterior femur wall. Intra-operative navigation can help in disseminating the AM approach by dramatically decreasing the execution risk and complexity. VTIAC may be applied to accomplish this intra-operative navigation by indicating the location in the femur notch where to open the tunnel (the ligament footprint) and by guiding the angular orientation of drilling.
In a possible design of the navigated procedure the surgeon starts by calibrating the arthroscopic camera and by attaching the WM in the medial side of the inter-condyle region (FIG. 5A ). The WM can take the form of a button-like flat surface in the end of a wire guide that is pushed inside-out to go across the femur bone till it surfaces the patient skin. He/she then pulls the wire from the exterior for the marker to be pulled inside the joint and placed against the wall of the femur notch. It is well known that the ligament footprint should be located in the ceiling of the inter-condyle region at ⅓ the length of notch ceiling measured from its posterior end. Thus, after placement of the WM, the surgeon uses the touch-probe with the TM to pin-point the two ends of the notch ceiling such that VTIAC can measure the distance and compute the location of the footprint that is overlaid in the video using AR (FIG. 5A ).
The orientation for opening the tunnel may be determined by registering a statistical model of the femur bone. For this purpose, the surgeon uses the touch probe to reconstruct the boundary contours of the inter-condyle region (FIG. 4B ) or, in alternative, to obtain a sparse 3D reconstruction of the surface of the femur bone (FIG. 4C ). This 3D data is fed into a suitable 3D registration algorithm that overlays the statistical model with the patient's anatomy. For opening the tunnel the surgeon uses a drill with a TM such that its position can be related in real time with the 3D data stored in memory that includes reconstruction results and the registered statistical model. One possible strategy for guided opening of the tunnel consists in the following: (i) VTIAC indicates the location of the footprint by overlaying in video using AR the point in the anatomy where drill tip should be place (the entry point), (ii) VTIAC shows in a VR environment the registered model and the current orientation of the drilling tool, where this orientation is computed in real-time from the arthroscopic video where both WM and TM can be seen, (iii) the VR environment shows the drilling direction at each frame time instant such that the surgeon can align it for the exit point to be in the Lateral epicondyle (FIG. 5B ), (iv) the tunnel is open along the selected trajectory while VTIAC provides the depth from surface at each frame time instant.
9. Example of Application of VTIAC for Guiding the Placement of Pedicle Screws (PPS) During Open Surgery of Spine.
This section discloses an embodiment of VTIAC based-navigation for Placing Pedicle Screws (PPS) during spine surgery, which can also be generalized to other open procedures where a rigid surface is exposed, such as total hip replacement, total knee replacement, open shoulder surgery and implant placement in dentistry.
Although VTIAC always requires a video input, its use is not limited to arthroscopy. The framework can also be applied to open orthopedic procedures, such as knee/hip arthroplasty or spine surgery, as far as a camera is employed to observe incision and relevant anatomy. The camera can either be a generic handheld camera (FIG. 2A ) or a camera mounted on a tool, such as a touch-probe like in the CamT described in section 7 (FIG. 7A ).
There are several traumas and pathologies of the spine whose treatment passes by a surgery for vertebra fusion. The procedure includes placing screws in two consecutive vertebras for keeping in position a metallic rod that prevents intervertebral motion. Each screw must be carefully inserted along the vertebra pedicle otherwise it can irremediably damage the spinal medulla or a vital blood vessel. The dominant technique for Placing Pedicle Screws (PPS) is the so-called “free-hand” approach, in which the surgeon relies in his experience and knowledge to insert the screw while occasionally using fluoroscopy to confirm the correct positioning. Since this process is risky and error prone, several manufacturers developed navigation systems for PPS where a pre-operative 3D plan is overlaid with the patient anatomy in the OR using opto-tracking. In this case the surgeon uses a pre-operative model of the vertebra (e.g. CT-Scan or MRI) to specify the 3D line along which the screw must be inserted, as well as the depth of insertion. The model and the surgeon specifications are henceforth referred as the pre-operative 3D plan. This section describes how VTIAC can be applied to accomplish intra-operative navigation after planning.
In the OR, and after opening an incision for partial or total exposition of the vertebra, the surgeon starts by rigidly attaching a visual marker (WM) to the bone surface. This marker plays the role of World Marker (WM) and is placed in an arbitrary position decided by the surgeon. The next step is to overlay the pre-operative plan with patient's anatomy in the OR, which passes by reconstructing points and/or curves on the vertebra surface to be used as input in a suitable 3D registration algorithm.
One possibility is to perform the 3D registration using a set of fiducial points or landmarks in the anatomy. In this case the system indicates a succession of landmark points to be reconstructed that are pin-pointed in by the surgeon using the touch-probe (FIG. 6A ). Another possibility is to use specific contours in the anatomy or a sparse 3D reconstruction of the surface in which case the surgeon randomly grasps the vertebra with the probe (FIG. 6B ). The reconstruction can either be performed with the CamT, in which case the WM must be in the FOV of the camera (FIG. 6A ), or with the standard touch probe with a TM attached, in which case both WM and TM must be visible in images (FIG. 6B ).
After registration, the VTIAC is able to overlay the 3D pre-operative plan in the intra-operative video, as well as the tip, axis, or CAD model of the tool, whenever WM and TM are respectively in the FOV of the camera (FIG. 6C ). As an alternative, and since the position of the WM in the vertebra model becomes known, the system is able to animate the motion of the tool with respect to the pre-operative model in a VR environment (FIG. 6D ).
The VTIAC can then project the guidance information into the AR view, such as the angle of the tool relatively to the planned direction (FIG. 6C ), or to provide a virtual extension of the tool for the surgeon to visualize the expected outcome of the chosen drill location and angle (FIG. 6D ). A strategy that is particularly effective and avoids errors whenever the insertion point on the bone surface is occluded by tissue (FIG. 6E ) is as follows: (i) configure the VR such that the optical axis of the virtual camera is aligned with the planned line S of insertion, (ii) move the tool tip along the occluding tissue till the tip overlays with line S that shows as a point, (iii) without moving the tip orient the tool such that its axis L shows as a point coincident with S and finally (iv) insert the tool till desired depth that is indicated by the system.
10. Application of VTIAC for Intra-operative Guidance in Other Clinical Procedures
VTIAC can be applied for intra-operative navigation in several other clinical procedures. A non-exhaustive list of possibilities include:
Arthroscopic reconstruction of Posterior Cruciate Ligament (PCL): The PCL is a ligament in the knee joint that connects the posterior intercondylar area of the tibia to the medial condyle of the femur. In a similar manner to the ACL, the PCL reconstruction consists in replacing the torn ligament by a substitution graft that is pulled inside the joint through a tunnel opened with a drill. VTIAC can be applied to guide the placement of these tunnels both in tibial and femoral sides.
Arthroscopic Resection of Femuro-Acetabular Impingement (FAI): FAI occurs when the ball shaped femoral head rubs abnormally in the acetabular socket, which in about 91% of the cases is caused by an excess of bone tissue in the femur head-neck that creates a bump known as CAM impingement. The treatment is surgical and consists in removing the CAM to restore the ball shape to the femur-head. To accomplish this objective the surgeon uses a CT-scan of the femur to study the CAM position and plan the extension of resection. This plan is then mentally transposed for the execution in the OR, which is a very error prone process. VTIAC can be applied to enforce the pre-planning by overlying the annotated 3D model with the patient's femur in order to safely guide the surgeon. After model registration the CAM footprint can be overlaid in the arthroscopic video using AR techniques and the system can inform the surgeon about the quantity of the bone tissue to remove at every instant.
Arthroscopic assessment and diagnosis of confocal defects in cartilage: Confocal defects are damages in the articular cartilage that can be repaired by filling the holes or craters with a bio-compatible material. This operation often requires placing in the hole or crater an rigid support structure called scaffolder. VTIAC can be used for measuring and determining the shape of confocal defects, as well as to guide the placement of these scaffolds.
Total hip replacement (THR): THR is an open surgical procedure for replacing the hip joint by an implant. The implant consists in a cup, that replaces acetabulum in the pelvic bone, and in a stem with a sphere that replaces the femural head. VTIAC can be applied to guide the placement of the cup such that it is inserted with optimal angular orientation, as well as to define the cut plane in the femural neck to remove the head and insert the stem with sphere.
Total Knee Replacement and Unicompartmental Knee Replacement: Knee arthroplasty is an open surgical procedure for replacing total or part of the knee joint by an implant (total or unicompartmental knee replacement). VTIAC can be applied to guide the surgeon in cutting the femural condyle and placing the implant.
Shoulder Joint Replacement: This is another open surgical procedure for replacing in total or in part the shoulder joint by an implant. VTIAC can be applied in assisting the surgeon in several steps of the execution such as indicating the plane of cut to remove humeral head, or guiding the reaming of humeral shaft and/or glenoid.
Placement of dental implants in Prosthodontics: VTIAC can be applied in dental surgery for placing an implant in the maxilar bone as planned in a pre-operative Cone Beam CT (CBCT) of the patient. In this case the WM is rigidly attached to a tooth, the CBCT is overlaid with patient's anatomy by using VTIAC features for 3D reconstruction, and the system provides intra-operative guidance for inserting the implant through any of the AR and VR features that have been described in the ACL and PPS examples.
11. Additional Notes and Remarks
As shown in FIG. 8 , the memory 508 may be used to house the instructions for carrying out various embodiments described herein. In an embodiment, the memory 508 may comprise an image processing module 510 that may be accessed and implemented by processor 502. Alternatively, the image processing module 510 may be stored and accessed within memory embedded in processor 502 (e.g., cache memory). Specifically, the image processing module 510 may estimate the camera response function and the vignetting in case of non-uniform illumination using one or more calibration images. In one embodiment, memory 508 interfaces with a computer bus so as to communicate and/or transmit information stored in memory 508 to processor 502 during execution of software programs, such as an operating system, application programs, device drivers, and software modules that comprise program code, and/or computer executable process steps, incorporating functionality described herein, e.g., the image processing module 510. Processor 502 first loads computer executable process steps from storage, e.g., memory 510, storage medium/media, removable media drive, and/or other storage device. Processor 502 can then execute the stored process steps in order to execute the loaded computer executable process steps. Stored data, e.g., data stored by a storage device, can be accessed by processor 502 during the execution of computer executable process steps to instruct one or more components within the image processing system 500.
Programming and/or loading executable instructions onto memory 508 and processor 502 in order to transform the image processing system 500 into a non-generic particular machine or apparatus that applies VTIAC to surgical procedures is well-known in the art. Implementing instructions, real-time monitoring, and other functions by loading executable software into a computer and/or processor can be converted to a hardware implementation by well-known design rules and/or transform a general-purpose processor to a processor programmed for a specific application. For example, decisions between implementing a concept in software versus hardware may depend on a number of design choices that include stability of the design and numbers of units to be produced and issues involved in translating from the software domain to the hardware domain. Often a design may be developed and tested in a software form and subsequently transformed, by well-known design rules, to an equivalent hardware implementation in an ASIC or application specific hardware that hardwires the instructions of the software. In the same manner as a machine controlled by a new ASIC is a particular machine or apparatus, likewise a computer that has been programmed and/or loaded with executable instructions may be viewed as a non-generic particular machine or apparatus.
In addition, FIG. 8 illustrates that the processor 502 may be operatively coupled to an input interface 504 configured to obtain one or more images and output interface 506 configured to output and/or display the images. The input interface 504 may be configured to obtain one or more images via electrical, optical, and/or wireless connections using one or more communication protocols. In one embodiment, the input interface 502 may be a network interface that comprises a plurality of ports configured to receive and/or transmit data via a network. In particular, the network may transmit image data via wired links, wireless link, and/or logical links. Other examples of the input interface 504 may include but are not limited to a keyboard, universal serial bus (USB) interfaces, CD-ROMs, DVD-ROMs and/or graphical input devices (e.g., onscreen and/or virtual keyboards). The output interface 506 may be an interface used to display information in a readable format for a user and/or used to transmit information to a separate apparatus or machine. Examples include, but are not limited to, a graphic display (e.g., monitors and display screens), a user interface, an interface used to connect to a printing device configured to produce hard-copies of the generated results, and output ports used to connect to a network and/or another computing device. Image processing system 500 may also include computing components not explicitly shown in FIG. 8 , but well-known in the art, such as one or more power supplies, network interface(s), audio interfaces, displays, and circuitry used to connect the processor 502, input interfaces 504, output interface 506, and memory 508.
As shown in FIG. 9 , internal architecture 900 of a computing device(s), computing system, computing platform and the like includes one or more processing units, processors, or processing cores, (also referred to herein as CPUs) 912, which interface with at least one computer bus 902. Also interfacing with computer bus 902 are computer-readable medium, or media, 906, network interface 914, memory 904, e.g., random access memory (RAM), run-time transient memory, read only memory (ROM), media disk drive interface 920 as an interface for a drive that can read and/or write to media including removable media such as floppy, CD-ROM, DVD, media, display interface 910 as interface for a monitor or other display device, keyboard interface 916 as interface for a keyboard, pointing device interface 918 as an interface for a mouse or other pointing device, and miscellaneous other interfaces 922 not shown individually, such as parallel and serial port interfaces and a universal serial bus (USB) interface.
Persistent storage, e.g., medium/media 906, can be used to store an operating system and one or more application programs. Persistent storage can also be used to store device drivers, such as one or more of a digital camera driver, monitor driver, printer driver, scanner driver, or other device drivers, web pages, content files, playlists and other files. Persistent storage can further include program modules and data files used to implement one or more embodiments of the present disclosure.
A network link typically provides information communication using transmission media through one or more networks to other devices that use or process the information. For example, the network link may provide a connection through a local network to a host computer or to equipment operated by a Network or Internet Service Provider (ISP). ISP equipment in turn provides data communication services through the public, worldwide packet-switching communication network of networks now commonly referred to as the Internet.
A computer called a server host connected to the Internet hosts a process that provides a service in response to information received over the Internet. For example, server host hosts a process that provides information representing video data for presentation at display 910. It is contemplated that the components of system 900 can be deployed in various configurations within other computer systems, e.g., host and server.
At least some embodiments of the present disclosure are related to the use of computer system 900 for implementing some or all of the techniques described herein. According to one embodiment, those techniques are performed by computer system 900 in response to processing unit 912 executing one or more sequences of one or more processor instructions contained in memory 904. Such instructions, also called computer instructions, software and program code, may be read into memory 904 from another computer-readable medium 906 such as storage device or network link. Execution of the sequences of instructions contained in memory 904 causes processing unit 912 to perform one or more of the method steps described herein. In alternative embodiments, hardware, such as ASIC, may be used in place of or in combination with software. Thus, embodiments of the present disclosure are not limited to any specific combination of hardware and software, unless otherwise explicitly stated herein.
The signals transmitted over network link and other networks through communications interface, carry information to and from computer system 900. Computer system 900 can send and receive information, including program code, through the networks, among others, through network link and communications interface. In an example using the Internet, a server host transmits program code for a particular application, requested by a message sent from computer, through Internet, ISP equipment, local network and communications interface. The received code may be executed by processor 902 as it is received, or may be stored in memory 904 or in storage device or other non-volatile storage for later execution, or both.
For the purposes of this disclosure a module is a software, hardware, or firmware (or combinations thereof) system, process or functionality, or component thereof, that performs or facilitates the processes, features, and/or functions described herein (with or without human interaction or augmentation). A module can include sub-modules. A module, or software components of a module, may be stored on a computer readable medium for execution by a processor. Modules may be integral to one or more servers, or be loaded and executed by one or more servers. One or more modules may be grouped into an engine or an application.
At least one embodiment is disclosed and variations, combinations, and/or modifications of the embodiment(s) and/or features of the embodiment(s) made by a person having ordinary skill in the art are within the scope of the disclosure. Alternative embodiments that result from combining, integrating, and/or omitting features of the embodiment(s) are also within the scope of the disclosure. Where numerical ranges or limitations are expressly stated, such express ranges or limitations may be understood to include iterative ranges or limitations of like magnitude falling within the expressly stated ranges or limitations (e.g., from about 1 to about 10 includes, 2, 3, 4, etc.; greater than 0.10 includes 0.11, 0.12, 0.13, etc.). The use of the term “about” means ±10% of the subsequent number, unless otherwise stated.
Use of the term “optionally” with respect to any element of a claim means that the element is required, or alternatively, the element is not required, both alternatives being within the scope of the claim. Use of broader terms such as comprises, includes, and having may be understood to provide support for narrower terms such as consisting of, consisting essentially of, and comprised substantially of Accordingly, the scope of protection is not limited by the description set out above but is defined by the claims that follow, that scope including all equivalents of the subject matter of the claims. Each and every claim is incorporated as further disclosure into the specification and the claims are embodiment(s) of the present disclosure.
While several embodiments have been provided in the present disclosure, it may be understood that the disclosed embodiments might be embodied in many other specific forms without departing from the spirit or scope of the present disclosure. The present examples are to be considered as illustrative and not restrictive, and the intention is not to be limited to the details given herein. For example, the various elements or components may be combined or integrated in another system or certain features may be omitted, or not implemented.
In addition, the various embodiments described and illustrated in the various embodiments as discrete or separate may be combined or integrated with other systems, modules, techniques, or methods without departing from the scope of the present disclosure. Other items shown or discussed as coupled or directly coupled or communicating with each other may be indirectly coupled or communicating through some interface, device, or intermediate component whether electrically, mechanically, or otherwise. Other examples of changes, substitutions, and alterations are ascertainable by one skilled in the art and may be made without departing from the spirit and scope disclosed herein.
In closing, it should be noted that the discussion of any reference is not an admission that it is prior art to the presently disclosed embodiments, especially any reference that may have a publication date after the priority date of this application. At the same time, each and every claim below is hereby incorporated into this detailed description or specification as additional embodiments of the presently disclosed embodiments.
Claims (102)
1. A system for video based computer-aided surgery and diagnosis in anatomical regions of a patient comprising rigid, non-deformable anatomical parts or tissues comprising:
(i). a free-moving camera that is meant to acquire intra-operative images and video of an operating field, articular joint or anatomical cavity;
(ii). a visual marker, referred to as a World Marker or WM, which is an object that can be rigidly attached to a surface, and that comprises at least one planar facet with a known pattern having a system of coordinates that defines a global or world system of coordinates;
(iii). a touch-probe that is a tool or instrument comprising a handgrip and a tip, and that has at least one visual marker, referred to as a Tool Marker or TM, comprising at least one planar facet with a known pattern that defines a local system of coordinates where a position of the tip PTT is known;
(iv). one or more surgical instruments for surgical execution, where each surgical instrument has at least one visual marker, also referred to as a Tool Marker or TM, comprising at least one planar facet with a known pattern that defines a local system of coordinates in which a position of relevant points and parts in the instrument are known; and
(v). an image processing system, that may correspond to or may be part of a computer and/or any other computing device, that receives as input images and video acquired by the camera (i), as well as commands that may be entered by a button panel, keyboard, camera buttons, foot switch and/or any other input interface, and that outputs processing results and guidance information to a display;
wherein the system is configured to execute steps comprising:
(vi). attachment of the WM of (ii) in an arbitrary location in a surface of a rigid anatomical part of interest, such that the WM of (ii) and the rigid anatomical part of interest do not move one with respect to the other;
(vii). 3D reconstruction of points in the surface of the rigid anatomical part of interest by using the touch-probe of (iii) to pin-point the points in the surface while keeping both the WM and the TM of the probe in a Field-of-View or FOV of the camera of (i) for determining the location of the touch-probe in the world system of coordinates to create 3D reconstruction results;
(viii). use the 3D reconstruction results to perform measurements, make inferences, or overlay a surgical plan with an anatomical region of the patient, in which case a 3D registration method may be used for determining a transformation that maps local coordinates in a surgical plan into world coordinates in the WM attached to the anatomical part of interest; and
(ix). assisted execution of a clinical procedure by using the surgical instruments of (iv) while keeping both WM and TMs of the surgical instruments in the FOV of the camera of (i) such that the surgical instruments can be located in world coordinates and real-time guidance can be provided by relating surgical instrument locations in world coordinates with the 3D reconstruction results and/or overlaid surgical plan;
where the camera in (i) is calibrated such that image points u expressed in pixel coordinates are mapped into image points x expressed in metric units according to
x=f−1(u; k,ξ)
x=f−1(u; k,ξ)
with f−1 being an inverse of a projection function f and where k and ξ stand respectively for camera intrinsic and distortion parameters.
2. The system of claim 1 wherein the camera is pre-calibrated from a factory or is calibrated in an Operating Room (OR) by acquiring a single image of a known calibration grid or checkerboard pattern that allows estimating intrinsic parameters k, a distortion ξ and a rigid transformation Ĝ that relates coordinates of the calibration grid and coordinates of the camera.
3. The system of claim 2 wherein the instrinsic parameters k and the distortion ξ are automatically updated during operation to compensate for one or more of differences in medium, rotation of lens scope with respect to camera-head, and variations in zoom.
4. The system of claim 1 wherein the WM in (ii) comprises, depending on a targeted clinical application and a chosen method (vi) of attachment, one or more of a screw-like object with a flat head or facet, a nail-like object with a flat head or facet to be fixed by pressure, a needle like object with a flat lateral facet for trans-dermic insertion into the joint or cavity, or a flat button-like object that is pulled inside a joint or cavity by a thread or guide.
5. The system of claim 1 wherein each of the (ii) World Marker, (iii) touch probe, and (iv) surgical instruments are manufactured in metal or plastic, and wherein the Tool Markers can be assembled, secured, printed, or engraved.
6. The system of claim 1 wherein the planar pattern of the visual marker in (ii), (iii), and (iv) comprises any pattern that can be detected and uniquely identified using image processing techniques, and that has recognizable fiducial points for estimating a plane-to-image homography that, given a camera calibration, can be factorized in a rotation r and translation t that map points p in the coordinate system of the pattern into points x in the metric coordinate system of the camera.
7. The system of claim 6 , wherein an initial 3D pose estimation r, t is further refined by determining increments in rotation ξR and translation ξt that minimize the following photo-geometric error
where T(u) is the pattern template, I(u) is a current frame, Ni is a pattern region, and w is an image warping function given by the function composition
w(u; r, t)=f(x; k, ξ)o h(x; r, t)o f−1(u; k′, ξ′)
w(u; r, t)=f(x; k, ξ)o h(x; r, t)o f−1(u; k′, ξ′)
with f being the projection function and h denoting a homography map that depends on a relative 3D pose r, t.
8. The system of claim 7 wherein the minimization of the photo-geometric error εI is performed using direct composition, inverse composition, or efficient second order minimization, and wherein formulation of the photo-geometric error takes into account variable illumination conditions.
9. The system of claim 1 wherein a calibration of the touch-probe of (iii) for finding the coordinates PT of its tip is performed in an operating room (OR) in parallel with a camera calibration, in which case the tool tip is placed in a point in a calibration grid with known coordinates PG and a single calibration image shows both the calibration grid and the TM of the touch-probe such that PT can be determined by
with {circumflex over (T)}−1 denoting an inverse of a rigid transformation {circumflex over (T)} that encodes a 3D pose r, t of the TM in camera coordinates that is determined from image information.
10. The system of claim 1 wherein an arbitrary point P is reconstructed in global or world coordinates by acquiring an image as described in (vii) that undergoes the following processing steps:
detect, identify, and estimate a 3D pose C of the WM in camera coordinates;
detect, identify, and estimate a 3D pose {circumflex over (T)} of the TM in camera coordinates;
reconstruct a point P that is in contact with the tip of the touch-probe by making
and
store 3D coordinates of point P in memory.
11. The system of claim 10 , wherein the system is further configured to execute steps comprising:
reconstructing a contour or a sparse mesh of a surface region, in which case the touch-probe is respectively used to outline the contour or randomly grasp the surface region, while the camera acquires a continuous sequence of images as described in (vii), and the processing steps are executed for each frame of the sequence.
12. The system of claim 11 wherein the 3D reconstruction results are used for measuring distances, areas, or volumes, inferring the shape of curves, regions, or surfaces, or overlying a surgical plan with the patient's anatomy in which case a suitable 3D registration method may be employed.
13. The system of claim 12 wherein 3D data is stored in memory, the 3D data including the reconstruction results of (vii) and the measurements, inferences and surgical plan of (vii), wherein the 3D data is overlaid in the video whenever the WM is in the FOV of the camera, in which case each image undergoes the following processing steps in real-time:
detect, identify, and estimate the 3D pose C of the WM in camera coordinates;
map the 3D data from world coordinates into camera coordinates using C; and
project the 3D data into the image using the camera calibration and projection function f (augmented reality).
14. The system of claim 1 wherein the system is configured to, for each of the one or more surgical instruments of (iv), calibrate the surgical instrument by determining in TM coordinates the position of a point, axis, or CAD model of the surgical instrument that can be either performed in factory at manufacturing time, or in the operating room (OR) before starting the clinical procedure, in which case the camera of (i) and the touch-probe of (iii) are used to reconstruct 3D points in the surgical instrument where the role of the WM is replaced by the TM of the instrument for the reconstruction results to be expressed in a local reference frame of the instrument.
15. The system of claim 1 wherein the assisted execution of the clinical procedure is accomplished by acquiring continuous video according to (ix), the video comprising a plurality of images, with each consecutive image undergoing the following processing steps in real-time:
a. detect, identify, and estimate a 3D pose C of the WM in camera coordinates;
b. detect, identify, and estimate a 3D pose {circumflex over (T)} of the TM of each of the one or more surgical instruments in camera coordinates;
c. compute the 3D pose T of TM in WM coordinates using the following equation:
T=C−1{circumflex over (T)}
T=C−1{circumflex over (T)}
d. map the information of surgical instrument calibration, that can be points, axes, or CAD models, into world coordinates using the rigid transformation T;
e. relate the surgical instrument calibration information with 3D data stored in memory to make measurements and inferences for the purpose of real-time guidance; and
f. display aiding features either by using Augmented Reality (AR), in which case guidance information is overlaid in image, or by animating a Virtual Reality (VR) 3D model.
16. The system of claim 15 wherein the aiding features can take multiple forms including highlighting points, regions or structures in anatomy, measuring distance, thickness or depth, measuring angles between axes or between an axis and a plane, anticipating trajectories of insertion, penetration or cut, and delimiting regions or structures in anatomy for resection or dissection.
17. The system of claim 15 wherein multiple surgical instruments are used simultaneously, in which case steps b to e are run in parallel for each instrument whose TM is visible in the image.
18. The system of claim 15 wherein the one or more surgical instruments of (iv) comprises, or is rigidly attached to, a robot end-effector and the guidance or aided execution of (ix) includes sending commands to the robot for the purpose of controlling the motion and/or action of the one or more surgical instruments.
19. The system of claim 1 , wherein WM is a primary marker, wherein one or more secondary markers WM' are attached to the surface of the rigid anatomical part of interest as described in (vi) for the purpose of increasing a working area or region of operation, in which case all systems functionalities of 3D reconstruction and aided execution are readily available whenever one of the secondary markers WM' is visible in image and the transformation W, that maps WM′ coordinates into global or world coordinates in a reference frame of the primary marker WM, is known.
20. The system of claim 19 wherein the transformation W is determined by acquiring an image where both primary and secondary markers are visible, in which case the transformation is given by
W′=C−1C′,
W′=C−1C′,
with C and C′ being respectively the 3D pose of WM and WM' that are computed from image information.
21. The system of claim 1 wherein the camera of (i) is assembled with a tool such that the camera and the tool become a single rigid body with the position of relevant points or parts of the tool being known in camera coordinates, in which case the location of these points or parts can be determined in global or world coordinates whenever the WM of (ii) is in the FOV of the camera and its 3D pose C can be estimated from image information.
22. The system of claim 21 wherein the camera is assembled with a touch-probe to create an ensemble to replace (iii) in the 3D reconstruction step of (vii), in which case the ensemble is used to pin-point an arbitrary point P while keeping the WM in the FOV of the camera to determine its world coordinates by making
with Pc being the position of the probe tip in camera coordinates and C−1 a 3D pose of the camera in WM coordinates that is estimated from image information.
23. The system of claim 22 wherein a calibration of the ensemble for finding coordinates Pc of the touch-probe tip in camera coordinates is performed in an operating room (OR) in parallel with a camera calibration, in which case the tool or instrument tip is placed in a point in a calibration grid with known coordinates PG, and a single calibration image shows the calibration grid such that Pc can be determined by
with Ĝ denoting a rigid transformation that encodes a 3D pose r, t of a pattern of the calibration grid in camera coordinates t.
24. The system of claim 1 wherein the touch probe of (iii) comprises a laser pointer with a visual marker (TM) attached, for which a line LT defined by a laser beam is known in a reference frame of the visual marker (TM), and wherein the 3D reconstruction step of (vii) is performed in a contactless manner by acquiring an image where WM, TM and a point of incidence of the laser beam with the surface are simultaneously visible, and by processing this image with point PT being given by
PT=LTΛ{circumflex over (T)}−1(Bx)
PT=LTΛ{circumflex over (T)}−1(Bx)
with Λ denoting the operation of line intersection in 3D, Bx being the back-projection line of the point of incidence that is detected in the image, and {circumflex over (T)}−1 (Bx) denoting the line Bx expressed in the local system of coordinates of the visual marker (TM).
25. The system of claim 1 wherein the touch probe of (iii) comprises a time-of-flight device that has a visual marker TM attached and that measures distances λ from an origin ST along a beam direction dT that are known in a reference frame of the visual marker TM, and wherein the 3D reconstruction is performed in a contactless manner by acquiring an image as described in (vii) that undergoes the processing steps with the point PT being given by
PT=ST+λdT.
PT=ST+λdT.
26. The system of claim 1 wherein the visual markers of (ii), (iii) and (iv) have multiple planar facets with distinct patterns where the location of each pattern is known in a common coordinate system of the markers for the purpose of extending a range of viewing positions and orientations from which the marker can be observed.
27. The system of claim 1 wherein the visual markers of (ii), (iii) and (iv) are non-planar, in which case each marker should comprise n≥3 recognizable points with known coordinates in its local reference frame for enabling 3D pose estimation from images using a Perspective-n-Point algorithm.
28. The system of claim 1 that is used for computer assisted execution of arthroscopic procedures including anterior and/or posterior cruciate ligament reconstruction, resection of femuro-acetabular impingement, or diagnosis and repair of confocal defects in cartilage, in which case the free-moving camera is the arthroscopic camera used for visualizing the articular joint.
29. The system of claim 1 that is used for computer assisted execution of open surgical procedures in orthopedics, including total hip replacement, total knee replacement, unicompartmental knee replacement, shoulder joint replacement, and pedicle-screw placement, in which case a camera is used to observe the operating field.
30. The system of claim 1 that is used for computer assisted execution of prosthodontic procedures including placement of dental implants.
31. A method for computer-aided execution of surgery and diagnosis in anatomical regions comprising rigid, non-deformable parts or tissues, the method comprising:
(i) attaching a visual marker, referred to as the World Marker or WM, in an arbitrary location in the surface of the rigid anatomical part of interest, the visual marker comprising at least one planar facet with a known pattern whose system of coordinates defines a global or world system of coordinates;
(ii) acquiring, by a free-moving camera, intra-operative images and video of an operating field, articular joint, or anatomical cavity;
(iii) using a touch-probe, which is a tool or instrument comprising a handgrip and a tip, and that has at least one visual marker, referred to as Tool Marker or TM, comprising at least one planar facet with a known pattern that defines a local system of coordinates where a position of the tip PT is known, for pin-pointing points and/or outlining curves in the surface of the rigid anatomical part, while keeping both WM and TM of the probe in the Field-of-View or FOV of the camera to be visible in the acquired images and video;
(iv) receiving, by an image processing system that may correspond to, or may be part of, a computer and/or any other computing device, the images and video acquired in (iii), which are processed for determining the location of the touch-probe and reconstructing in 3D the pin-pointed points and/or outlined curves that are stored in memory in global or world coordinates;
(v) using the 3D reconstruction results to perform measurements, make inferences, or overlay a surgical plan with the patient's anatomy, in which case a 3D registration method may be used for determining the transformation that maps local coordinates in a surgical plan into world coordinates in the WM attached to anatomy of interest;
(vi) performing the surgical procedure with the required instruments, where each instrument has at least one visual marker, also referred to as Tool Marker or TM, comprising at least one planar facet with a known pattern that defines a local system of coordinates in which the position of relevant points and parts in the instrument are known, while keeping both WM and TMs of the instruments in the FOV of the camera to be visible in the acquired images and video;
(vii) receiving, by the image processing system, the images and video acquired in (vi), that are processed such that instruments can be located in world coordinates and real-time guidance can be provided by relating these locations with 3D reconstructions results and/or overlaid surgical plan; and
(viii) outputting, by the image processing system, processing results and guidance information to a display, by either overlaying information in the input video using augmented reality (AR) techniques, or by animating models in a virtual reality (VR) environment;
where the camera is calibrated such that image points u expressed in pixel coordinates are mapped into image points x expressed in metric units according to
x =f−1(u; k, ξ)
x =f−1(u; k, ξ)
with f =1 being the inverse of a projection function f and where k and ξ stand respectively for camera intrinsic and distortion parameters.
32. A system for video based computer-aided surgery and diagnosis in anatomical regions of a patient comprising rigid, non-deformable anatomical parts or tissues comprising:
(i). a free-moving camera that is meant to acquire intra-operative images and video of an operating field, articular joint or anatomical cavity;
(ii). a visual marker, referred to as a World Marker or WM, which is an object that can be rigidly attached to a surface, and that comprises at least one planar facet with a known pattern having a system of coordinates that defines a global or world system of coordinates;
(iii). a touch-probe that is a tool or instrument comprising a handgrip and a tip, and that has at least one visual marker, referred to as a Tool Marker or TM, comprising at least one planar facet with a known pattern that defines a local system of coordinates where a position of the tip PT is known;
(iv). one or more surgical instruments for surgical execution, where each surgical instrument has at least one visual marker, also referred to as a Tool Marker or TM, comprising at least one planar facet with a known pattern that defines a local system of coordinates in which a position of relevant points and parts in the instrument are known; and
(v). an image processing system, that may correspond to or may be part of a computer and/or any other computing device, that receives as input images and video acquired by the camera (i), as well as commands that may be entered by a button panel, keyboard, camera buttons, foot switch and/or any other input interface, and that outputs processing results and guidance information to a display;
wherein the system is configured to execute steps comprising:
(vi). attachment of the WM of (ii) in an arbitrary location in a surface of a rigid anatomical part of interest, such that the WM of (ii) and the rigid anatomical part of interest do not move one with respect to the other;
(vii). 3D reconstruction of points in the surface of the rigid anatomical part of interest by using the touch-probe of (iii) to pin-point the points in the surface while keeping both the WM and the TM of the probe in a Field-of-View or FOV of the camera of (i) for determining the location of the touch-probe in the world system of coordinates to create 3D reconstruction results;
(viii). use the 3D reconstruction results to perform measurements, make inferences, or overlay a surgical plan with an anatomical region of the patient, in which case a 3D registration method may be used for determining a transformation that maps local coordinates in a surgical plan into world coordinates in the WM attached to the anatomical part of interest; and
(ix). assisted execution of a clinical procedure by using the surgical instruments of (iv) while keeping both WM and TMs of the surgical instruments in the FOV of the camera of (i) such that the surgical instruments can be located in world coordinates and real-time guidance can be provided by relating surgical instrument locations in world coordinates;
wherein an arbitrary point P is reconstructed in global or world coordinates by acquiring an image as described in (vii) that undergoes the following processing steps:
detect, identify, and estimate a 3D pose C of the WM in camera coordinates;
detect, identify, and estimate a 3D pose {circumflex over (T)} of the TM in camera coordinates;
reconstruct a point P that is in contact with the tip of the touch-probe by making
and
store 3D coordinates of point P in a memory.
33. A system for video based computer-aided surgery and diagnosis in anatomical regions of a patient comprising rigid, non-deformable anatomical parts or tissues comprising:
(i). a free-moving camera that is meant to acquire intra-operative images and video of an operating field, articular joint or anatomical cavity;
(ii). a visual marker, referred to as a World Marker or WM, which is an object that can be rigidly attached to a surface, and that comprises at least one planar facet with a known pattern having a system of coordinates that defines a global or world system of coordinates;
(iii). a touch-probe that is a tool or instrument comprising a handgrip and a tip, and that has at least one visual marker, referred to as a Tool Marker or TM, comprising at least one planar facet with a known pattern that defines a local system of coordinates where a position of the tip PT is known;
(iv). one or more surgical instruments for surgical execution, where each surgical instrument has at least one visual marker, also referred to as a Tool Marker or TM, comprising at least one planar facet with a known pattern that defines a local system of coordinates in which a position of relevant points and parts in the instrument are known; and
(v). an image processing system, that may correspond to or may be part of a computer and/or any other computing device, that receives as input images and video acquired by the camera (i), as well as commands that may be entered by a button panel, keyboard, camera buttons, foot switch and/or any other input interface, and that outputs processing results and guidance information to a display;
wherein the system is configured to execute steps comprising:
(vi). attachment of the WM of (ii) in an arbitrary location in a surface of a rigid anatomical part of interest, such that the WM of (ii) and the rigid anatomical part of interest do not move one with respect to the other;
(vii). 3D reconstruction of points in the surface of the rigid anatomical part of interest by using the touch-probe of (iii) to pin-point the points in the surface while keeping both the WM and the TM of the probe in a Field-of-View or FOV of the camera of (i) for determining the location of the touch-probe in the world system of coordinates to create 3D reconstruction results;
(viii). use the 3D reconstruction results to perform measurements, make inferences, or overlay a surgical plan with an anatomical region of the patient, in which case a 3D registration method may be used for determining a transformation that maps local coordinates in a surgical plan into world coordinates in the WM attached to the anatomical part of interest; and
(ix).assisted execution of a clinical procedure by using the surgical instruments of (iv) while keeping both WM and TMs of the surgical instruments in the FOV of the camera of (i) such that the surgical instruments can be located in world coordinates and real-time guidance can be provided by relating surgical instrument locations in world coordinates;
wherein the assisted execution of the clinical procedure is accomplished by acquiring continuous video according to (ix), the video comprising a plurality of images, with each consecutive image undergoing the following processing steps in real-time:
a. detect, identify, and estimate a 3D pose C of the WM in camera coordinates;
b. detect, identify, and estimate a 3D pose {circumflex over (T)} of the TM of each of the one or more surgical instruments in camera coordinates;
c. compute the 3D pose T of TM in WM coordinates using the following equation:
T =C−1 {circumflex over (T)}
T =C−1 {circumflex over (T)}
d. map the information of surgical instrument calibration, that can be points, axes, or CAD models, into world coordinates using the rigid transformation T;
e. relate the surgical instrument calibration information with 3D data stored in memory to make measurements and inferences for the purpose of real-time guidance; and
f. display aiding features either by using Augmented Reality (AR), in which case guidance information is overlaid in image or by animating a Virtual Reality (VR) 3D model.
34. A system for computer-aided surgery and diagnosis in an anatomical region of a patient comprising a rigid, non-deformable anatomical part, the system comprising:
a free-moving camera to acquire intra-operative images, video, or images and video of an operating field,
where the free-moving camera is calibrated such that image points u expressed in pixel coordinates are mapped into image points x expressed in metric units according to
x=f−1(u; k, ξ)
x=f−1(u; k, ξ)
with f−1 being the inverse of a projection function f and where k and ξ stand respectively for camera intrinsic and distortion parameters;
a world marker (WM) comprising an object configured to be rigidly coupled to the anatomical part such that the WM and the anatomical part do not move one with respect to each other, wherein the object comprises a first known pattern that defines a first system of coordinates;
one or more surgical instruments, wherein each surgical instrument has at least one tool marker (TM), each TM comprising a respective second known pattern that defines a second system of coordinates in which a position of relevant points and parts of the surgical instrument are known; and
an image processing system configured to:
receive the images, video, or images and video acquired by the camera, wherein the images, video, or images and video include the WM, one or more of the TMs, or the WM and one or more of the TMs in a field of view of the camera;
receive a 3D reconstruction of points of the anatomical part in the first coordinate system; and
output information derived from the 3D reconstruction to a display to assist execution of a clinical procedure using the one or more surgical instruments while keeping the WM, one or more of the TMs, or the WM and one or more of the TMs in the field of view of the camera, wherein the output comprises real-time guidance including a respective location of each surgical instrument in the first coordinate system.
35. The system of claim 34, wherein the WM comprises one or more of a screw-like object with a flat head or facet, a nail-like object with a flat head or facet to be fixed by pressure, a needle-like object with a flat lateral facet for trans-dermic insertion, or a flat button-like object that is pulled by a thread or guide.
36. The system of claim 34, wherein each pattern comprises recognizable fiducial points for estimating a plane-to-image homography that, given a camera calibration, can be factorized in a rotation and translation that map points in the coordinate system of the pattern into points in the coordinate system of the camera.
37. The system of claim 34 wherein at least one of the known patterns is disposed on a planar surface.
38. The system of claim 34, wherein at least one of the known patterns is disposed on a non-planar surface.
39. The system of claim 34, further comprising a further surgical instrument, wherein the camera is rigidly coupled to the further surgical instrument.
40. The system of claim 39, further comprising:
determining a location of a portion of the further surgical instrument in the first coordinate system according to an image that includes the WM and a known positional relationship between the portion of the further surgical instrument and the camera;
wherein the output further comprises the location of the further surgical instrument.
41. The system of claim 34, wherein:
one of the surgical instruments is an end effector of a robotic tool; and
the image processing system is further configured to send a command to the robotic tool to control the end effector.
42. The system of claim 41, wherein the camera is rigidly coupled to the end effector.
43. The system of claim 34, further comprising a tablet computer, the tablet computer comprising the image processing system and the display.
44. The system of claim 34, further comprising:
a point location device for locating points in the rigid anatomical part, wherein a TM is rigidly coupled to the point location device.
45. The system of claim 44, wherein the point location device comprises one or more of:
a touch probe;
a laser pointer; or
a time-of-flight device.
46. The system of claim 34, further comprising a point location device for locating points in the rigid anatomical part, wherein a TM is rigidly coupled to the point location device, wherein the image processing system is configured to reconstruct the known points in the first coordinate system by, for at least one image:
determining a 3D pose of the WM in camera coordinates;
determining a 3D pose of the point location device TM in camera coordinates; and
reconstructing a point captured by the point location device according to the 3D pose of the WM and the 3D pose of the point location device TM.
47. The system of claim 34, wherein the image processing system is configured to determine the locations of the surgical instruments in the first coordinate system by:
determining a 3D pose of the WM in camera coordinates;
determining a 3D pose of the surgical instrument TMs in camera coordinates; and
determining a 3D pose of the surgical instrument TMs in the first coordinate system according to the 3D pose of the WM in camera coordinates and the 3D pose of the surgical instrument TMs in camera coordinates.
48. The system of claim 34, wherein the camera is calibrated such that image points expressed in pixel coordinates are mapped into image points expressed in distance units according to the projection function, the camera intrinsic parameters, and the camera distortion parameters.
49. The system of claim 48, wherein the image processing system is configured to:
calibrate the camera by acquiring, with the camera, a calibration image of a known calibration grid and determining, based on the calibration image, the camera intrinsic parameters and the camera distortion parameters.
50. The system of claim 49, wherein the image processing system is configured to:
update the camera intrinsic parameters and the camera distortion parameters during the surgical procedure to compensate for one or more of differences in medium, rotation of lens scope with respect to camera-head, or variations in zoom.
51. The system of claim 49, further comprising:
a point location device for locating points in the rigid anatomical part, wherein a TM is rigidly coupled to the point location device;
wherein the calibration image comprises the TM of the point location device and a portion of the point location device, wherein the image processing system is configured to:
calibrate the point location device by determining, based on the calibration image, a location of the portion of the point location device according to an inverse of a rigid transformation that encodes a 3D pose of the TM in camera coordinates.
52. The system of claim 34, wherein receiving the 3D reconstruction comprises performing the 3D reconstruction according to one or more known points of the anatomical part and according to a location of the WM in the images, video, or images and video.
53. The system of claim 34, wherein the WM or at least one of the TMs has multiple planar facets with multiple known patterns, where the locations of the multiple known patterns are known in a common coordinate system of the WM or the at least one of the TMs.
54. A method for computer-aided execution of surgery and diagnosis in an anatomical region comprising a rigid anatomical part, the method comprising:
attaching a world marker (WM) to a surface of the rigid anatomical part, the WM comprising a known pattern defining a first system of coordinates;
acquiring, by a free-moving camera, first intra-operative images, video, or images and video, while keeping the WM in a field of view of the camera,
wherein the free-moving camera is calibrated such that image points u expressed in pixel coordinates are mapped into image points x expressed in metric units according to
x=f−1(u; k, ξ)
x=f−1(u; k, ξ)
with f−being the inverse of a projection function f and where k and ξ stand respectively for camera intrinsic and distortion parameters;
receiving a 3D reconstruction of points of the anatomical part in the first coordinate system;
during a surgical procedure performed with one or more surgical instruments, wherein each surgical instrument has a respective tool marker (TM), each TM comprising a respective known pattern that defines a respective second system of coordinates in which positions of one or more points on the instrument are known, acquiring, by the camera, second intra-operative images, video, or images and video, while keeping the WM and the surgical instrument TMs in a field of view of the camera;
processing, by the image processing system, the second images, video, or images and video to determine locations of the surgical instruments in the first coordinate system; and
outputting information derived from the 3D reconstruction to a display to assist execution of a clinical procedure using the one or more surgical instruments while keeping the WM, one or more of the TMs, or the WM and one or more of the TMs in the field of view of the camera, wherein the output comprises real-time guidance including a respective location of each surgical instrument in the first coordinate system.
55. The method of claim 54, wherein the image processing system is configured to determine the locations of the surgical instruments in the first coordinate system by:
determining a 3D pose of the WM in camera coordinates;
determining a 3D pose of the surgical instrument TMs in camera coordinates; and
determining a 3D pose of the surgical instrument TMs in the first coordinate system according to the 3D pose of the WM in camera coordinates and the 3D pose of the surgical instrument TMs in camera coordinates.
56. The method of claim 54, wherein the camera is calibrated such that image points expressed in pixel coordinates are mapped into image points expressed in distance units according to the projection function, the camera intrinsic parameters, and the camera distortion parameters.
57. The method of claim 56, further comprising:
calibrating the camera by acquiring, with the camera, a calibration image of a known calibration grid and determining, based on the calibration image, the camera intrinsic parameters and the camera distortion parameters.
58. The method of claim 57, further comprising:
updating the camera intrinsic parameters and the camera distortion parameters during the surgical procedure to compensate for one or more of differences in medium, rotation of lens scope with respect to camera-head, or variations in zoom.
59. The method of claim 57, further comprising:
while acquiring the first intra-operative images, locating one or more points in the rigid anatomical part using a point location device, wherein a tool marker (TM) is rigidly coupled to the point location device, wherein the TM comprises a known pattern that defines a third system of coordinates in which a position of a portion of the point location device is known;
wherein receiving the 3D reconstruction comprises performing the 3D reconstruction according to the one or more located points and according to a location of the WM in the images, video, or images and video;
wherein the calibration image comprises the TM of the point location device and the portion of the point location device, the method further comprising:
calibrating the point location device by determining, based on the calibration image, a location of the portion of the point location device according to an inverse of a rigid transformation that encodes a 3D pose of the TM in camera coordinates.
60. The method of claim 54, further comprising:
while acquiring the first intra-operative images, locating one or more points in the rigid anatomical part using a point location device, wherein a TM is rigidly coupled to the point location device, wherein the TM comprises a known pattern that defines a third system of coordinates in which a position of a portion of the point location device is known;
wherein receiving the 3D reconstruction comprises performing the 3D reconstruction according to the one or more located points and according to a location of the WM in the images, video, or images and video.
61. The method of claim 60, wherein the point location device comprises one or more of:
a touch probe;
a laser pointer; or
a time-of-flight device.
62. The method of claim 60, wherein the image processing system is configured to reconstruct the located points in the first coordinate system by, for at least one image:
determining a 3D pose of the WM in camera coordinates;
determining a 3D pose of the point location device TM in camera coordinates; and
reconstructing a point P located by the point location device according to the 3D pose of the WM and the 3D pose of the point location device TM.
63. The method of claim 54, wherein the WM comprises one or more of a screw-like object with a flat head or facet, a nail-like object with a flat head or facet to be fixed by pressure, a needle-like object with a flat lateral facet for trans-dermic insertion, or a flat button-like object that is pulled by a thread or guide.
64. The method of claim 54, wherein each pattern comprises recognizable fiducial points for estimating a plane-to-image homography that, given a camera calibration, can be factorized in a rotation and translation that map points in the coordinate system of the pattern into points in the coordinate system of the camera.
65. The method of claim 54, wherein at least one of the known patterns is disposed on a planar surface.
66. The method of claim 54, wherein at least one of the known patterns is disposed on a non-planar surface.
67. The method of claim 54, further comprising:
determining a location of a portion of a further surgical instrument, the camera rigidly coupled to the further surgical instrument, according to an image that includes the WM and a known positional relationship between the portion of the further surgical instrument and the camera.
68. The method of claim 54, wherein:
one of the surgical instruments is an end effector of a robotic tool; and
the image processing system is further configured to send a command to the robotic tool to control the end effector.
69. The method of claim 68, wherein the camera is rigidly coupled to the end effector.
70. The method of claim 54, wherein the image processing system and the display comprise a tablet computer.
71. The method of claim 54, wherein receiving the 3D reconstruction comprises performing the 3D reconstruction according to one or more known points of the anatomical part and according to a location of the WM in the images, video, or images and video.
72. The method of claim 54, wherein the WM or at least one of the TMs has multiple planar facets includes multiple known patterns, where the locations of the multiple known patterns are known in a common coordinate system of the WM or the at least one of the TMs.
73. A system for computer-aided surgery and diagnosis in an anatomical region of a patient comprising a rigid, non-deformable anatomical part, the system comprising:
a camera to acquire intra-operative images, video, or images and video of an operating field,
where the camera is calibrated such that image points u expressed in pixel coordinates are mapped into image points x expressed in metric units according to
x=f−1(u; k, ξ)
x=f−1(u; k, ξ)
with f−1 being the inverse of a projection function f and where k and ξ stand respectively for camera intrinsic and distortion parameters;
a world marker (WM) comprising an object configured to be rigidly coupled to the anatomical part such that the WM and the anatomical part do not move one with respect each other, wherein the object comprises a first known pattern that defines a first system of coordinates;
an instrument comprising a surgical instrument or a point location device, wherein the camera is rigidly coupled to the instrument such that camera has a known positional relationship with a portion of the instrument;
an image processing system configured to:
receive the images, video, or images and video acquired by the camera, wherein the images, video, or images and video include the WM in a field of view of the camera;
receive a 3D reconstruction of points of the anatomical part in the first coordinate system; and
output information derived from the 3D reconstruction to a display to assist execution of a clinical procedure using the instrument while keeping the WM in the field of view of the camera, wherein the output comprises a location of the instrument in the first coordinate system.
74. The system of claim 34, wherein the image processing system is further configured to determine the 3D reconstruction of the points of the anatomical part by determining locations of points of the anatomical part.
75. The system of claim 34, wherein the image processing system is further configured to determine the 3D reconstruction of the points using one or more images captured by the free-moving camera.
76. The system of claim 75, wherein the image processing system is further configured to determine the 3D reconstruction of the points using one or more images each comprising the WM and a point, of the anatomical part, to be located.
77. The system of claim 76, wherein the image processing system is further configured to determine the 3D reconstruction of the points using one or more images each comprising the WM and an illumination by light of a light source of the point to be located.
78. The system of claim 74, wherein the image processing system is further configured to determine the 3D reconstruction of the points using a light source and/or a time-of-flight device.
79. The system of claim 78, further comprising:
the light source to illuminate points of the anatomical part for locating the points for 3D reconstruction.
80. The system of claim 78, wherein:
the system further comprises a light source to emit structured light; and
the image processing system is further configured to determine a location of a point on the anatomical part by analyzing at least one image of a point of the anatomical part illuminated with the structured light.
81. The system of claim 80, wherein the light source is arranged to emit collimated light.
82. The system of claim 81, wherein the light source is configured to emit a collimated beam of visible light.
83. The system of claim 80, wherein the light source is a laser.
84. The system of claim 74, wherein the image processing system is further configured to determine the 3D reconstruction of the points using a touch probe.
85. The system of claim 84, further comprising:
the touch probe, wherein the touch probe has a TM.
86. The system of claim 34, further comprising:
a second surgical instrument, wherein the second surgical instrument is a guide.
87. The system of claim 86, wherein the guide comprises a TM.
88. The system of claim 34, wherein the image processing system is configured to monitor a location of a portion of a surgical instrument in a field of view of the free-moving camera.
89. The system of claim 88, wherein the image processing system is configured to monitor a location of a distal end of the surgical instrument in the field of view of the free-moving camera.
90. The system of claim 89, wherein the image processing system is configured to monitor a location of a tip of the surgical instrument in a field of view of the free-moving camera.
91. The system of claim 90, wherein the surgical instrument is a touch probe and the tip of the surgical instrument is a tip of the touch probe.
92. The system of claim 88, wherein the surgical instrument is rigidly coupled to the free-moving camera.
93. A system for computer-aided surgery and diagnosis in an anatomical region of a patient comprising a rigid, non-deformable anatomical part, the system comprising:
a free-moving camera to acquire intra-operative images, video, or images and video of an operating field,
where the free-moving camera is calibrated such that image points u expressed in pixel coordinates are mapped into image points x expressed in metric units according to
x=f−1(u; k, ξ)
x=f−1(u; k, ξ)
with f−1 being the inverse of a projection function f and where k and ξ stand respectively for camera intrinsic and distortion parameters;
a world marker (WM) comprising an object configured to be rigidly coupled to the anatomical part such that the WM and the anatomical part do not move one with respect each other; and
an image processing system configured to:
receive the images, video, or images and video acquired by the free-moving camera, wherein the images, video, or images and video include the WM in a field of view of the camera;
receive a 3D reconstruction of points of the anatomical part in a first coordinate system defined using the WM; and
output information derived from the 3D reconstruction for display to assist execution of a clinical procedure using a surgical instrument while keeping the WM in the field of view of the camera, wherein the output comprises guidance including a location of a surgical instrument in the first coordinate system.
94. The system of claim 93, wherein the WM is a planar object.
95. The system of claim 93, wherein the WM comprises a known pattern.
96. The system of claim 93, The system of claim 34, wherein the image processing system is further configured to determine the 3D reconstruction of the points of the anatomical part by determining locations of points of the anatomical part.
97. The system of claim 93, wherein the image processing system is further configured to determine the 3D reconstruction of the points using one or more images captured by the free-moving camera.
98. The system of claim 93, wherein the image processing system is configured to monitor a location of a portion of a surgical instrument in a field of view of the free-moving camera.
99. The system of claim 98, wherein the surgical instrument is rigidly coupled to the free-moving camera.
100. The system of claim 99, wherein the surgical instrument is a touch probe.
101. The system of claim 93, further comprising:
a surgical instrument, wherein the surgical instrument is a guide.
102. The system of claim 101, wherein the guide comprises a tool marker (TM).
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US17/548,202 USRE49930E1 (en) | 2015-03-26 | 2016-03-25 | Methods and systems for computer-aided surgery using intra-operative video acquired by a free moving camera |
Applications Claiming Priority (5)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US201562138529P | 2015-03-26 | 2015-03-26 | |
US201562255513P | 2015-11-15 | 2015-11-15 | |
US15/561,666 US10499996B2 (en) | 2015-03-26 | 2016-03-25 | Methods and systems for computer-aided surgery using intra-operative video acquired by a free moving camera |
US17/548,202 USRE49930E1 (en) | 2015-03-26 | 2016-03-25 | Methods and systems for computer-aided surgery using intra-operative video acquired by a free moving camera |
PCT/US2016/024262 WO2016154557A1 (en) | 2015-03-26 | 2016-03-25 | Methods and systems for computer-aided surgery using intra-operative video acquired by a free moving camera |
Publications (1)
Publication Number | Publication Date |
---|---|
USRE49930E1 true USRE49930E1 (en) | 2024-04-23 |
Family
ID=56977750
Family Applications (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US17/548,202 Active 2036-06-06 USRE49930E1 (en) | 2015-03-26 | 2016-03-25 | Methods and systems for computer-aided surgery using intra-operative video acquired by a free moving camera |
US15/561,666 Ceased US10499996B2 (en) | 2015-03-26 | 2016-03-25 | Methods and systems for computer-aided surgery using intra-operative video acquired by a free moving camera |
Family Applications After (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US15/561,666 Ceased US10499996B2 (en) | 2015-03-26 | 2016-03-25 | Methods and systems for computer-aided surgery using intra-operative video acquired by a free moving camera |
Country Status (3)
Country | Link |
---|---|
US (2) | USRE49930E1 (en) |
EP (1) | EP3273854B1 (en) |
WO (1) | WO2016154557A1 (en) |
Families Citing this family (132)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
FR2955481B1 (en) | 2010-01-27 | 2013-06-14 | Tornier Sa | DEVICE AND METHOD FOR GLENOIDAL CHARACTERIZATION OF PROSTHETIC OR RESURFACING OMOPLATE |
US11857266B2 (en) | 2012-06-21 | 2024-01-02 | Globus Medical, Inc. | System for a surveillance marker in robotic-assisted surgery |
US11786324B2 (en) | 2012-06-21 | 2023-10-17 | Globus Medical, Inc. | Surgical robotic automation with tracking markers |
US12004905B2 (en) | 2012-06-21 | 2024-06-11 | Globus Medical, Inc. | Medical imaging systems using robotic actuators and related methods |
US11253327B2 (en) | 2012-06-21 | 2022-02-22 | Globus Medical, Inc. | Systems and methods for automatically changing an end-effector on a surgical robot |
US11963755B2 (en) | 2012-06-21 | 2024-04-23 | Globus Medical Inc. | Apparatus for recording probe movement |
US10624710B2 (en) | 2012-06-21 | 2020-04-21 | Globus Medical, Inc. | System and method for measuring depth of instrumentation |
US11793570B2 (en) | 2012-06-21 | 2023-10-24 | Globus Medical Inc. | Surgical robotic automation with tracking markers |
US11974822B2 (en) | 2012-06-21 | 2024-05-07 | Globus Medical Inc. | Method for a surveillance marker in robotic-assisted surgery |
US11896446B2 (en) | 2012-06-21 | 2024-02-13 | Globus Medical, Inc | Surgical robotic automation with tracking markers |
US11857149B2 (en) | 2012-06-21 | 2024-01-02 | Globus Medical, Inc. | Surgical robotic systems with target trajectory deviation monitoring and related methods |
US10758315B2 (en) | 2012-06-21 | 2020-09-01 | Globus Medical Inc. | Method and system for improving 2D-3D registration convergence |
US11864839B2 (en) | 2012-06-21 | 2024-01-09 | Globus Medical Inc. | Methods of adjusting a virtual implant and related surgical navigation systems |
US11045267B2 (en) | 2012-06-21 | 2021-06-29 | Globus Medical, Inc. | Surgical robotic automation with tracking markers |
US11589771B2 (en) | 2012-06-21 | 2023-02-28 | Globus Medical Inc. | Method for recording probe movement and determining an extent of matter removed |
US11864745B2 (en) | 2012-06-21 | 2024-01-09 | Globus Medical, Inc. | Surgical robotic system with retractor |
US10874466B2 (en) | 2012-06-21 | 2020-12-29 | Globus Medical, Inc. | System and method for surgical tool insertion using multiaxis force and moment feedback |
US10799298B2 (en) | 2012-06-21 | 2020-10-13 | Globus Medical Inc. | Robotic fluoroscopic navigation |
US11399900B2 (en) | 2012-06-21 | 2022-08-02 | Globus Medical, Inc. | Robotic systems providing co-registration using natural fiducials and related methods |
US9993177B2 (en) | 2014-08-28 | 2018-06-12 | DePuy Synthes Products, Inc. | Systems and methods for intraoperatively measuring anatomical orientation |
US10013808B2 (en) | 2015-02-03 | 2018-07-03 | Globus Medical, Inc. | Surgeon head-mounted display apparatuses |
USRE49930E1 (en) | 2015-03-26 | 2024-04-23 | Universidade De Coimbra | Methods and systems for computer-aided surgery using intra-operative video acquired by a free moving camera |
EP3284252B1 (en) | 2015-04-13 | 2021-07-21 | Universidade De Coimbra | Methods and systems for camera characterization in terms of response function, color, and vignetting under non-uniform illumination |
US10058393B2 (en) | 2015-10-21 | 2018-08-28 | P Tech, Llc | Systems and methods for navigation and visualization |
FI20155784A (en) * | 2015-11-02 | 2017-05-03 | Cryotech Nordic Oü | Automated system for laser-assisted dermatological treatment and control procedure |
US9554411B1 (en) | 2015-12-30 | 2017-01-24 | DePuy Synthes Products, Inc. | Systems and methods for wirelessly powering or communicating with sterile-packed devices |
US10335241B2 (en) | 2015-12-30 | 2019-07-02 | DePuy Synthes Products, Inc. | Method and apparatus for intraoperative measurements of anatomical orientation |
US11883217B2 (en) | 2016-02-03 | 2024-01-30 | Globus Medical, Inc. | Portable medical imaging system and method |
WO2017139556A1 (en) * | 2016-02-12 | 2017-08-17 | Medos International Sarl | Systems and methods for intraoperatively measuring anatomical orientation |
WO2017137087A1 (en) * | 2016-02-12 | 2017-08-17 | Brainlab Ag | Method and system for registering a patient with a 3d image using a robot |
US10531926B2 (en) * | 2016-05-23 | 2020-01-14 | Mako Surgical Corp. | Systems and methods for identifying and tracking physical objects during a robotic surgical procedure |
JP2018005091A (en) * | 2016-07-06 | 2018-01-11 | 富士通株式会社 | Display control program, display control method and display controller |
US10716630B2 (en) | 2016-07-15 | 2020-07-21 | Mako Surgical Corp. | Systems and methods for a robotic-assisted revision procedure |
WO2018052966A1 (en) * | 2016-09-16 | 2018-03-22 | Zimmer, Inc. | Augmented reality surgical technique guidance |
EP3533407A4 (en) * | 2016-10-25 | 2020-05-13 | LEXI Co., Ltd. | Surgery assistance system |
US10918398B2 (en) | 2016-11-18 | 2021-02-16 | Stryker Corporation | Method and apparatus for treating a joint, including the treatment of cam-type femoroacetabular impingement in a hip joint and pincer-type femoroacetabular impingement in a hip joint |
EP3582711A1 (en) * | 2017-02-14 | 2019-12-25 | Atracsys Sàrl | High-speed optical tracking with compression and/or cmos windowing |
WO2018169891A1 (en) | 2017-03-13 | 2018-09-20 | Zimmer, Inc. | Augmented reality diagnosis guidance |
WO2018170181A1 (en) * | 2017-03-14 | 2018-09-20 | Universidade De Coimbra | Systems and methods for 3d registration of curves and surfaces using local differential information |
DE102017105421A1 (en) | 2017-03-14 | 2018-09-20 | Olympus Winter & Ibe Gmbh | Medical system and image processing system and method for processing optically detectable information codes containing image data |
US11089975B2 (en) | 2017-03-31 | 2021-08-17 | DePuy Synthes Products, Inc. | Systems, devices and methods for enhancing operative accuracy using inertial measurement units |
CN110603004A (en) * | 2017-04-13 | 2019-12-20 | 虚拟卷尺技术有限公司 | Endoscopic measurement method and tool |
JP7189939B2 (en) * | 2017-05-05 | 2022-12-14 | ストライカー・ユーロピアン・オペレーションズ・リミテッド | surgical navigation system |
US10987190B2 (en) * | 2017-05-09 | 2021-04-27 | Brainlab Ag | Generation of augmented reality image of a medical device |
US10860750B2 (en) * | 2017-06-15 | 2020-12-08 | Sap Se | Model driven layout design for robotics warehouse |
EP3644281A4 (en) * | 2017-06-20 | 2021-04-28 | Sony Interactive Entertainment Inc. | Calibration device, calibration chart, chart pattern generation device, and calibration method |
US11589933B2 (en) * | 2017-06-29 | 2023-02-28 | Ix Innovation Llc | Guiding a robotic surgical system to perform a surgical procedure |
US11432877B2 (en) | 2017-08-02 | 2022-09-06 | Medtech S.A. | Surgical field camera system that only uses images from cameras with an unobstructed sight line for tracking |
CN111712210A (en) * | 2017-10-06 | 2020-09-25 | 智能联合外科公司 | System and method for preoperative planning for total hip arthroplasty |
US11103314B2 (en) * | 2017-11-24 | 2021-08-31 | Synaptive Medical Inc. | Methods and devices for tracking objects by surgical navigation systems |
DE102017129837A1 (en) * | 2017-12-13 | 2019-06-13 | Leibniz-Institut für Photonische Technologien e. V. | Combined examination with imaging and laser measurement |
US11464569B2 (en) | 2018-01-29 | 2022-10-11 | Stryker Corporation | Systems and methods for pre-operative visualization of a joint |
US20190254753A1 (en) | 2018-02-19 | 2019-08-22 | Globus Medical, Inc. | Augmented reality navigation systems for use with robotic surgical systems and methods of their use |
US10213274B1 (en) | 2018-03-12 | 2019-02-26 | King Saud University | Method of tracking and navigation for a dental instrument |
JP6979049B2 (en) * | 2018-06-07 | 2021-12-08 | グローバス メディカル インコーポレイティッド | Robot systems and related methods that provide co-registration using natural standards |
EP3810012A1 (en) | 2018-06-19 | 2021-04-28 | Tornier, Inc. | Mixed reality-aided education using virtual models or virtual representations for orthopedic surgical procedures |
DE102018119343B4 (en) * | 2018-08-08 | 2022-11-03 | Carl Zeiss Meditec Ag | Method for calibrating objects in a reference coordinate system and method for tracking objects |
US20210322148A1 (en) * | 2018-08-28 | 2021-10-21 | Smith & Nephew, Inc. | Robotic assisted ligament graft placement and tensioning |
US20200069372A1 (en) * | 2018-09-05 | 2020-03-05 | Orthosoft Ulc | Method and system for navigating a bone model in computer-assisted surgery |
US11471206B2 (en) | 2018-09-07 | 2022-10-18 | Cilag Gmbh International | Method for controlling a modular energy system user interface |
US12035956B2 (en) | 2018-09-07 | 2024-07-16 | Cilag Gmbh International | Instrument tracking arrangement based on real time clock information |
US11804679B2 (en) | 2018-09-07 | 2023-10-31 | Cilag Gmbh International | Flexible hand-switch circuit |
US11923084B2 (en) | 2018-09-07 | 2024-03-05 | Cilag Gmbh International | First and second communication protocol arrangement for driving primary and secondary devices through a single port |
CN109242914B (en) * | 2018-09-28 | 2021-01-01 | 上海爱观视觉科技有限公司 | Three-dimensional calibration method of movable vision system |
WO2020116085A1 (en) * | 2018-12-05 | 2020-06-11 | ソニー株式会社 | Estimation apparatus, estimation method, and estimation program |
US11730389B2 (en) * | 2019-01-28 | 2023-08-22 | Incremed Ag | Method and system for supporting medical interventions |
US11810473B2 (en) | 2019-01-29 | 2023-11-07 | The Regents Of The University Of California | Optical surface tracking for medical simulation |
US11495142B2 (en) | 2019-01-30 | 2022-11-08 | The Regents Of The University Of California | Ultrasound trainer with internal optical tracking |
CN109903337B (en) * | 2019-02-28 | 2022-06-14 | 北京百度网讯科技有限公司 | Method and apparatus for determining pose of bucket of excavator |
AU2020235624A1 (en) * | 2019-03-12 | 2021-11-04 | Amdt Holdings, Inc. | Monoscopic radiographic image and three-dimensional model registration methods and systems |
US11284957B2 (en) | 2019-03-15 | 2022-03-29 | Cilag Gmbh International | Robotic surgical controls with force feedback |
US11992282B2 (en) | 2019-03-15 | 2024-05-28 | Cilag Gmbh International | Motion capture controls for robotic surgery |
US11490981B2 (en) | 2019-03-15 | 2022-11-08 | Cilag Gmbh International | Robotic surgical controls having feedback capabilities |
US11471229B2 (en) * | 2019-03-15 | 2022-10-18 | Cilag Gmbh International | Robotic surgical systems with selectively lockable end effectors |
US11666401B2 (en) | 2019-03-15 | 2023-06-06 | Cilag Gmbh International | Input controls for robotic surgery |
US11583350B2 (en) | 2019-03-15 | 2023-02-21 | Cilag Gmbh International | Jaw coordination of robotic surgical controls |
US11690690B2 (en) | 2019-03-15 | 2023-07-04 | Cilag Gmbh International | Segmented control inputs for surgical robotic systems |
US11701190B2 (en) | 2019-03-15 | 2023-07-18 | Cilag Gmbh International | Selectable variable response of shaft motion of surgical robotic systems |
WO2020192872A2 (en) * | 2019-03-22 | 2020-10-01 | Brainlab Ag | Method of estimating a position of a medical instrument tip |
US11218822B2 (en) | 2019-03-29 | 2022-01-04 | Cilag Gmbh International | Audio tone construction for an energy module of a modular energy system |
FR3095331A1 (en) * | 2019-04-26 | 2020-10-30 | Ganymed Robotics | Computer-assisted orthopedic surgery procedure |
CN109998682B (en) * | 2019-04-28 | 2020-08-18 | 北京天智航医疗科技股份有限公司 | Probe device, precision detection method, precision detection system and positioning system |
CN112085797B (en) * | 2019-06-12 | 2024-07-19 | 通用电气精准医疗有限责任公司 | 3D camera-medical imaging device coordinate system calibration system and method and application thereof |
TWI741359B (en) * | 2019-08-30 | 2021-10-01 | 國立中央大學 | Mixed reality system integrated with surgical navigation system |
US11574446B2 (en) * | 2019-08-30 | 2023-02-07 | National Central University | Digital image reality aligning kit and method applied to mixed reality system for surgical navigation |
WO2021044522A1 (en) * | 2019-09-03 | 2021-03-11 | 株式会社インキュビット | Surgery assist device |
USD939545S1 (en) | 2019-09-05 | 2021-12-28 | Cilag Gmbh International | Display panel or portion thereof with graphical user interface for energy module |
JP7070802B2 (en) * | 2019-09-27 | 2022-05-18 | 株式会社タダノ | Crane information display system |
AU2020362216A1 (en) | 2019-10-07 | 2022-04-14 | S&N Orion Prime, S.A. | Systems and methods for characterization of an endoscope and automatic calibration of an endoscopic camera system |
EP4042382A1 (en) * | 2019-10-07 | 2022-08-17 | Intuitive Surgical Operations, Inc. | Physical medical element sizing systems and methods |
EP4051160A4 (en) * | 2019-11-01 | 2023-11-29 | True Digital Surgery | Robotic surgical navigation using a proprioceptive digital surgical stereoscopic camera system |
US11992373B2 (en) | 2019-12-10 | 2024-05-28 | Globus Medical, Inc | Augmented reality headset with varied opacity for navigated robotic surgery |
CN111161355B (en) * | 2019-12-11 | 2023-05-09 | 上海交通大学 | Multi-view camera pose and scene pure pose resolving method and system |
US11464581B2 (en) | 2020-01-28 | 2022-10-11 | Globus Medical, Inc. | Pose measurement chaining for extended reality surgical navigation in visible and near infrared spectrums |
US11382699B2 (en) | 2020-02-10 | 2022-07-12 | Globus Medical Inc. | Extended reality visualization of optical tool tracking volume for computer assisted navigation in surgery |
US11207150B2 (en) | 2020-02-19 | 2021-12-28 | Globus Medical, Inc. | Displaying a virtual model of a planned instrument attachment to ensure correct selection of physical instrument attachment |
WO2021168408A1 (en) | 2020-02-20 | 2021-08-26 | Smith & Nephew, Inc. | Methods for arthroscopic video analysis and devices therefor |
CN115210756A (en) * | 2020-04-03 | 2022-10-18 | 史密夫和内修有限公司 | Method for arthroscopic surgical video segmentation and apparatus therefor |
US11607277B2 (en) | 2020-04-29 | 2023-03-21 | Globus Medical, Inc. | Registration of surgical tool with reference array tracked by cameras of an extended reality headset for assisted navigation during surgery |
US11510750B2 (en) | 2020-05-08 | 2022-11-29 | Globus Medical, Inc. | Leveraging two-dimensional digital imaging and communication in medicine imagery in three-dimensional extended reality applications |
US11153555B1 (en) | 2020-05-08 | 2021-10-19 | Globus Medical Inc. | Extended reality headset camera system for computer assisted navigation in surgery |
US11382700B2 (en) | 2020-05-08 | 2022-07-12 | Globus Medical Inc. | Extended reality headset tool tracking and control |
US20210393344A1 (en) * | 2020-06-22 | 2021-12-23 | Auris Health, Inc. | Control scheme calibration for medical instruments |
CN111821025B (en) * | 2020-07-21 | 2022-05-13 | 腾讯科技(深圳)有限公司 | Space positioning method, device, equipment, storage medium and navigation bar |
US12002235B2 (en) | 2020-08-12 | 2024-06-04 | Hong Kong Applied Science and Technology Research Institute Company Limited | Apparatus and method for estimating camera orientation relative to ground surface |
US11737831B2 (en) | 2020-09-02 | 2023-08-29 | Globus Medical Inc. | Surgical object tracking template generation for computer assisted navigation during surgical procedure |
US11813746B2 (en) | 2020-12-30 | 2023-11-14 | Cilag Gmbh International | Dual driving pinion crosscheck |
EP4066749A4 (en) * | 2021-02-05 | 2022-11-30 | Shenzhen Institutes of Advanced Technology Chinese Academy of Sciences | Soft apparatus for opening eyelids and method therefor |
US11963727B2 (en) | 2021-03-30 | 2024-04-23 | Cilag Gmbh International | Method for system architecture for modular energy system |
US11950860B2 (en) | 2021-03-30 | 2024-04-09 | Cilag Gmbh International | User interface mitigation techniques for modular energy systems |
US12004824B2 (en) | 2021-03-30 | 2024-06-11 | Cilag Gmbh International | Architecture for modular energy system |
US11857252B2 (en) | 2021-03-30 | 2024-01-02 | Cilag Gmbh International | Bezel with light blocking features for modular energy system |
US11978554B2 (en) | 2021-03-30 | 2024-05-07 | Cilag Gmbh International | Radio frequency identification token for wireless surgical instruments |
US11968776B2 (en) | 2021-03-30 | 2024-04-23 | Cilag Gmbh International | Method for mechanical packaging for modular energy system |
US11980411B2 (en) | 2021-03-30 | 2024-05-14 | Cilag Gmbh International | Header for modular energy system |
US12040749B2 (en) | 2021-03-30 | 2024-07-16 | Cilag Gmbh International | Modular energy system with dual amplifiers and techniques for updating parameters thereof |
CN113509263B (en) * | 2021-04-01 | 2024-06-14 | 上海复拓知达医疗科技有限公司 | Object space calibration positioning method |
US12004966B2 (en) | 2021-04-12 | 2024-06-11 | Kaliber Labs Inc. | Systems and methods for using image analysis in superior capsule reconstruction |
WO2022219501A1 (en) * | 2021-04-14 | 2022-10-20 | Cilag Gmbh International | System comprising a camera array deployable out of a channel of a tissue penetrating surgical device |
US20220366649A1 (en) * | 2021-05-14 | 2022-11-17 | Edda Technology, Inc. | Method and system of depth determination in model fusion for laparoscopic surgical guidance |
US11928834B2 (en) | 2021-05-24 | 2024-03-12 | Stryker Corporation | Systems and methods for generating three-dimensional measurements using endoscopic video data |
AU2022347455A1 (en) * | 2021-09-20 | 2024-03-28 | Kaliber Labs Inc. | System and method for computer-assisted surgery |
CN113836770B (en) * | 2021-09-28 | 2023-09-12 | 方丽颖 | Method, apparatus, computer device and computer readable storage medium for tool breakpoint resume processing |
TWI826111B (en) * | 2021-11-15 | 2023-12-11 | 美商尼奧西斯股份有限公司 | Methods of planning and executing a robot-implemented dental procedure |
AU2022401872A1 (en) * | 2021-12-01 | 2024-05-16 | Smith & Nephew Asia Pacific Pte. Limited | Bone reamer video based navigation |
CN114767031B (en) * | 2022-03-31 | 2024-03-08 | 常州朗合医疗器械有限公司 | Endoscope apparatus, position guidance apparatus, system, method, and computer-readable storage medium for endoscope |
WO2024033861A1 (en) * | 2022-08-11 | 2024-02-15 | Upsurgeon S.R.L. | Surgical navigation system, surgical navigation method, calibration method of surgical navigation system |
CN115471559B (en) * | 2022-10-31 | 2023-03-17 | 北京石油化工学院 | Head dynamic positioning and tracking method and system |
US20240197410A1 (en) * | 2022-12-19 | 2024-06-20 | Stryker Corporation | Systems and methods for guiding drilled hole placement in endoscopic procedures |
FR3143808A1 (en) * | 2022-12-19 | 2024-06-21 | One Ortho | Stylus for three-dimensional scanning system |
CN116077182B (en) * | 2022-12-23 | 2024-05-28 | 北京纳通医用机器人科技有限公司 | Medical surgical robot control method, device, equipment and medium |
CN116459013B (en) * | 2023-04-24 | 2024-03-22 | 北京微链道爱科技有限公司 | Collaborative robot based on 3D visual recognition |
Citations (74)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5313306A (en) | 1991-05-13 | 1994-05-17 | Telerobotics International, Inc. | Omniview motionless camera endoscopy system |
US5767980A (en) | 1995-06-20 | 1998-06-16 | Goss Graphic Systems, Inc. | Video based color sensing device for a printing press control system |
US5792147A (en) | 1994-03-17 | 1998-08-11 | Roke Manor Research Ltd. | Video-based systems for computer assisted surgery and localisation |
US5836869A (en) | 1994-12-13 | 1998-11-17 | Olympus Optical Co., Ltd. | Image tracking endoscope system |
US6047088A (en) | 1996-12-16 | 2000-04-04 | Sharp Laboratories Of America, Inc. | 2D mesh geometry and motion vector compression |
US6072496A (en) | 1998-06-08 | 2000-06-06 | Microsoft Corporation | Method and system for capturing and representing 3D geometry, color and shading of facial expressions and other animated objects |
US6449103B1 (en) | 1997-04-16 | 2002-09-10 | Jeffrey R. Charles | Solid catadioptric omnidirectional optical system having central coverage means which is associated with a camera, projector, medical instrument, or similar article |
US20040019274A1 (en) | 2001-06-27 | 2004-01-29 | Vanderbilt University | Method and apparatus for collecting and processing physical space data for use while performing image-guided surgery |
US20040070565A1 (en) | 2001-12-05 | 2004-04-15 | Nayar Shree K | Method and apparatus for displaying images |
US20040181149A1 (en) | 2001-02-07 | 2004-09-16 | Ulrich Langlotz | Device and method for intraoperative navigation |
US20050047676A1 (en) | 2003-04-29 | 2005-03-03 | Microsoft Corporation | System and process for generating high dynamic range video |
US20050085720A1 (en) | 2003-10-17 | 2005-04-21 | Jascob Bradley A. | Method and apparatus for surgical navigation |
US20050089199A1 (en) | 2000-08-30 | 2005-04-28 | Microsoft Corporation | Facial image processing methods and systems |
US20050245821A1 (en) * | 2004-03-05 | 2005-11-03 | Assaf Govari | Position sensing system for orthopedic applications |
US20050270375A1 (en) | 2004-05-24 | 2005-12-08 | Pierre Poulin | Camera calibrating apparatus and method |
US20050281465A1 (en) | 2004-02-04 | 2005-12-22 | Joel Marquart | Method and apparatus for computer assistance with total hip replacement procedure |
US20050280709A1 (en) | 2004-06-18 | 2005-12-22 | Japan Aerospace Exploration Agency | Transparent camera calibration tool for camera calibration and calibration method thereof |
US20060082546A1 (en) | 2003-06-23 | 2006-04-20 | Fun Wey | Computer input device tracking six degrees of freedom |
US20060239345A1 (en) | 2002-09-20 | 2006-10-26 | David Taubman | Method of signalling motion information for efficient scalable video compression |
US20060250300A1 (en) * | 2005-05-06 | 2006-11-09 | Jean-Louis Laroche | RF system for tracking objects |
EP1759629A1 (en) | 2005-08-31 | 2007-03-07 | Karl Storz GmbH & Co. KG | Endoscope with variable direction of view |
US20070236514A1 (en) | 2006-03-29 | 2007-10-11 | Bracco Imaging Spa | Methods and Apparatuses for Stereoscopic Image Guided Surgical Navigation |
US20080075324A1 (en) | 2004-07-21 | 2008-03-27 | Japan Science And Technology Agency | Camera Calibration System and Three-Dimensional Measuring System |
US20080097156A1 (en) | 2006-10-23 | 2008-04-24 | Pentax Corporation | Camera calibration for endoscope navigation system |
US20080202509A1 (en) | 2007-02-26 | 2008-08-28 | Microtek Medical, Inc. | Helmets and methods of making and using the same |
US20080239327A1 (en) | 2007-03-30 | 2008-10-02 | Mitutoyo Corporation | Global calibration for stereo vision probe |
US20080269596A1 (en) * | 2004-03-10 | 2008-10-30 | Ian Revie | Orthpaedic Monitoring Systems, Methods, Implants and Instruments |
US20080306490A1 (en) | 2007-05-18 | 2008-12-11 | Ryan Cameron Lakin | Trackable diagnostic scope apparatus and methods of use |
WO2009042644A2 (en) * | 2007-09-25 | 2009-04-02 | Perception Raisonnement Action En Medecine | Methods and apparatus for assisting cartilage diagnostic and therapeutic procedures |
US20100009314A1 (en) * | 2002-02-28 | 2010-01-14 | Materialise Nv | Method and device for placing dental implants |
EP2153794A2 (en) * | 2008-08-15 | 2010-02-17 | Stryker Leibinger GmbH & Co. KG | System for and method of visualizing an interior of a body |
US20100168562A1 (en) | 2008-12-31 | 2010-07-01 | Intuitive Surgical, Inc. | Fiducial marker design and detection for locating surgical instrument in images |
US20100245541A1 (en) | 2009-03-31 | 2010-09-30 | Intuitive Surgical, Inc. | Targets, fixtures, and workflows for calibrating an endoscopic camera |
US20110075922A1 (en) | 2009-09-30 | 2011-03-31 | Disney Enterprises, Inc. | Apparatus and method for removing ink lines and segmentation of color regions of A 2-D image for converting 2-D images into stereoscopic 3-D images |
US20110115798A1 (en) | 2007-05-10 | 2011-05-19 | Nayar Shree K | Methods and systems for creating speech-enabled avatars |
US20110130761A1 (en) * | 2005-04-07 | 2011-06-02 | Perception Raisonnement Action En Medecine | Robotic guide assembly for use in computer-aided surgery |
US7970174B2 (en) | 2006-06-27 | 2011-06-28 | Brainlab Ag | Medical marker tracking with marker property determination |
US20120078049A1 (en) | 2010-09-17 | 2012-03-29 | Henke-Sass, Wolf Gmbh | Endoscope with variable direction of view |
US20120120255A1 (en) | 2009-07-21 | 2012-05-17 | Frederic Cao | Method for estimating a defect in an image-capturing system, and associated systems |
WO2013015699A1 (en) | 2011-07-25 | 2013-01-31 | Universidade De Coimbra | Method and apparatus for automatic camera calibration using one or more images of a checkerboard pattern |
US20130034203A1 (en) | 2011-08-03 | 2013-02-07 | Hongkai Wang | 2d/3d registration of a digital mouse atlas with x-ray projection images and optical camera photos |
WO2013052187A2 (en) * | 2011-06-27 | 2013-04-11 | Board Of Regents Of The University Of Nebraska | On-board tool tracking system and methods of computer assisted surgery |
US20130123801A1 (en) | 2011-11-15 | 2013-05-16 | Macdonald Dettwiler & Associates | Method of real-time tracking of moving/flexible surfaces |
US20130150863A1 (en) | 2011-06-22 | 2013-06-13 | Adrian Baumgartner | Ultrasound ct registration for positioning |
US20130281821A1 (en) * | 2011-01-13 | 2013-10-24 | Koninklijke Philips Electronics N.V. | Intraoperative camera calibration for endoscopic surgery |
US20140022248A1 (en) | 2012-07-20 | 2014-01-23 | Google Inc. | Determining Three-Dimensional (3D) Object Data Models Based On Object Movement |
US20140031668A1 (en) | 2010-09-08 | 2014-01-30 | Disruptive Navigational Technologies, Llc | Surgical and Medical Instrument Tracking Using a Depth-Sensing Device |
US20140035893A1 (en) | 2012-07-31 | 2014-02-06 | Warren Jackson | Color adjustment based on object positioned near display surface |
WO2014054958A2 (en) | 2012-10-05 | 2014-04-10 | Universidade De Coimbra | Method for aligning and tracking point regions in images with radial distortion that outputs motion model parameters, distortion calibration, and variation in zoom |
US20140107471A1 (en) | 2011-06-27 | 2014-04-17 | Hani Haider | On-board tool tracking system and methods of computer assisted surgery |
US8771177B2 (en) | 2008-07-08 | 2014-07-08 | Karl Storz Imaging, Inc. | Wide angle flexible endoscope |
WO2014122301A1 (en) * | 2013-02-11 | 2014-08-14 | Neomedz Sàrl | Tracking apparatus for tracking an object with respect to a body |
US20140327796A1 (en) | 2013-05-02 | 2014-11-06 | National Chung Cheng University | Method for estimating camera response function |
US8902232B2 (en) | 2008-01-31 | 2014-12-02 | University Of Southern California | Facial performance synthesis using deformation driven polynomial displacement maps |
US20150065793A1 (en) | 2008-06-27 | 2015-03-05 | Intuitive Surgical Operations, Inc. | Medical robotic system providing computer generated auxiliary views of a camera instrument for controlling the position and orienting of its tip |
US20150065799A1 (en) | 2008-07-08 | 2015-03-05 | Karl Storz Imaging, Inc. | Solid State Variable Direction Of View Endoscope |
US20150297177A1 (en) * | 2014-04-17 | 2015-10-22 | The Johns Hopkins University | Robot assisted ultrasound system |
US9307892B2 (en) | 2012-11-22 | 2016-04-12 | Karl Storz, Gmbh & Co. Kg | Endoscope with an adjustable viewing direction |
US20160148435A1 (en) | 2014-11-26 | 2016-05-26 | Restoration Robotics, Inc. | Gesture-Based Editing of 3D Models for Hair Transplantation Applications |
US20160161602A1 (en) | 2014-12-09 | 2016-06-09 | Toyota Motor Engineering & Manufacturing North America, Inc. | Sensor calibration for autonomous vehicles |
US20160183841A1 (en) | 2013-08-15 | 2016-06-30 | Intuitive Surgical Operations Inc. | Graphical User Interface For Catheter Positioning And Insertion |
US20160191887A1 (en) | 2014-12-30 | 2016-06-30 | Carlos Quiles Casas | Image-guided surgery with surface reconstruction and augmented reality visualization |
US9398840B2 (en) | 2011-04-20 | 2016-07-26 | Henke-Sass, Wolf Gmbh | Endoscope with variable direction of view |
US20160220099A1 (en) | 2013-10-16 | 2016-08-04 | Olympus Winter & Ibe Gmbh | Endoscope having an adjustable viewing direction |
WO2016154557A1 (en) | 2015-03-26 | 2016-09-29 | Universidade De Coimbra | Methods and systems for computer-aided surgery using intra-operative video acquired by a free moving camera |
WO2016168307A1 (en) | 2015-04-13 | 2016-10-20 | Universidade De Coimbra | Methods and systems for camera characterization in terms of response function, color, and vignetting under non-uniform illumination |
US20160324580A1 (en) | 2015-03-23 | 2016-11-10 | Justin Esterberg | Systems and methods for assisted surgical navigation |
EP3130276A1 (en) | 2015-08-12 | 2017-02-15 | M.S.T. Medical Surgery Technologies Ltd. | Endoscope with wide angle lens and adjustable view |
US20170325669A1 (en) | 2015-05-12 | 2017-11-16 | Avraham Levy | Dynamic field of view endoscope |
US9888831B2 (en) | 2011-04-11 | 2018-02-13 | Olympus Corporation | Imaging device and imaging method |
US20180049622A1 (en) | 2016-08-16 | 2018-02-22 | Insight Medical Systems, Inc. | Systems and methods for sensory augmentation in medical procedures |
US9986183B2 (en) | 2014-06-11 | 2018-05-29 | Samsung Electronics Co., Ltd. | Lens distortion correction device and application processor having the same |
WO2018232322A1 (en) | 2017-06-15 | 2018-12-20 | Children's National Medical Center | System, apparatus and method for calibrating oblique-viewing rigid endoscope |
US11335075B2 (en) | 2017-03-14 | 2022-05-17 | Universidade De Coimbra | Systems and methods for 3D registration of curves and surfaces using local differential information |
-
2016
- 2016-03-25 US US17/548,202 patent/USRE49930E1/en active Active
- 2016-03-25 WO PCT/US2016/024262 patent/WO2016154557A1/en unknown
- 2016-03-25 EP EP16769785.3A patent/EP3273854B1/en active Active
- 2016-03-25 US US15/561,666 patent/US10499996B2/en not_active Ceased
Patent Citations (92)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5313306A (en) | 1991-05-13 | 1994-05-17 | Telerobotics International, Inc. | Omniview motionless camera endoscopy system |
US5792147A (en) | 1994-03-17 | 1998-08-11 | Roke Manor Research Ltd. | Video-based systems for computer assisted surgery and localisation |
US5836869A (en) | 1994-12-13 | 1998-11-17 | Olympus Optical Co., Ltd. | Image tracking endoscope system |
US5767980A (en) | 1995-06-20 | 1998-06-16 | Goss Graphic Systems, Inc. | Video based color sensing device for a printing press control system |
US6047088A (en) | 1996-12-16 | 2000-04-04 | Sharp Laboratories Of America, Inc. | 2D mesh geometry and motion vector compression |
US6449103B1 (en) | 1997-04-16 | 2002-09-10 | Jeffrey R. Charles | Solid catadioptric omnidirectional optical system having central coverage means which is associated with a camera, projector, medical instrument, or similar article |
US6072496A (en) | 1998-06-08 | 2000-06-06 | Microsoft Corporation | Method and system for capturing and representing 3D geometry, color and shading of facial expressions and other animated objects |
US20050089199A1 (en) | 2000-08-30 | 2005-04-28 | Microsoft Corporation | Facial image processing methods and systems |
US20040181149A1 (en) | 2001-02-07 | 2004-09-16 | Ulrich Langlotz | Device and method for intraoperative navigation |
US20040019274A1 (en) | 2001-06-27 | 2004-01-29 | Vanderbilt University | Method and apparatus for collecting and processing physical space data for use while performing image-guided surgery |
US20040070565A1 (en) | 2001-12-05 | 2004-04-15 | Nayar Shree K | Method and apparatus for displaying images |
US20100009314A1 (en) * | 2002-02-28 | 2010-01-14 | Materialise Nv | Method and device for placing dental implants |
US20060239345A1 (en) | 2002-09-20 | 2006-10-26 | David Taubman | Method of signalling motion information for efficient scalable video compression |
US20050047676A1 (en) | 2003-04-29 | 2005-03-03 | Microsoft Corporation | System and process for generating high dynamic range video |
US20060082546A1 (en) | 2003-06-23 | 2006-04-20 | Fun Wey | Computer input device tracking six degrees of freedom |
US7751865B2 (en) | 2003-10-17 | 2010-07-06 | Medtronic Navigation, Inc. | Method and apparatus for surgical navigation |
US20050085720A1 (en) | 2003-10-17 | 2005-04-21 | Jascob Bradley A. | Method and apparatus for surgical navigation |
US20050281465A1 (en) | 2004-02-04 | 2005-12-22 | Joel Marquart | Method and apparatus for computer assistance with total hip replacement procedure |
US20050245821A1 (en) * | 2004-03-05 | 2005-11-03 | Assaf Govari | Position sensing system for orthopedic applications |
US20080269596A1 (en) * | 2004-03-10 | 2008-10-30 | Ian Revie | Orthpaedic Monitoring Systems, Methods, Implants and Instruments |
US20050270375A1 (en) | 2004-05-24 | 2005-12-08 | Pierre Poulin | Camera calibrating apparatus and method |
US7429999B2 (en) | 2004-05-24 | 2008-09-30 | CENTRE DE RECHERCHE INDUSTRIELLE DU QUéBEC | Camera calibrating apparatus and method |
US20050280709A1 (en) | 2004-06-18 | 2005-12-22 | Japan Aerospace Exploration Agency | Transparent camera calibration tool for camera calibration and calibration method thereof |
US7808525B2 (en) | 2004-06-18 | 2010-10-05 | Japan Aerospace Exploration Agency | Transparent camera calibration tool for camera calibration and calibration method thereof |
US20080075324A1 (en) | 2004-07-21 | 2008-03-27 | Japan Science And Technology Agency | Camera Calibration System and Three-Dimensional Measuring System |
US7750969B2 (en) | 2004-07-21 | 2010-07-06 | Japan Science And Technology Agency | Camera calibration system and three-dimensional measuring system |
US20110130761A1 (en) * | 2005-04-07 | 2011-06-02 | Perception Raisonnement Action En Medecine | Robotic guide assembly for use in computer-aided surgery |
US20060250300A1 (en) * | 2005-05-06 | 2006-11-09 | Jean-Louis Laroche | RF system for tracking objects |
EP1759629A1 (en) | 2005-08-31 | 2007-03-07 | Karl Storz GmbH & Co. KG | Endoscope with variable direction of view |
US20070236514A1 (en) | 2006-03-29 | 2007-10-11 | Bracco Imaging Spa | Methods and Apparatuses for Stereoscopic Image Guided Surgical Navigation |
US7970174B2 (en) | 2006-06-27 | 2011-06-28 | Brainlab Ag | Medical marker tracking with marker property determination |
US7892165B2 (en) | 2006-10-23 | 2011-02-22 | Hoya Corporation | Camera calibration for endoscope navigation system |
US20080097156A1 (en) | 2006-10-23 | 2008-04-24 | Pentax Corporation | Camera calibration for endoscope navigation system |
US20080202509A1 (en) | 2007-02-26 | 2008-08-28 | Microtek Medical, Inc. | Helmets and methods of making and using the same |
US20080239327A1 (en) | 2007-03-30 | 2008-10-02 | Mitutoyo Corporation | Global calibration for stereo vision probe |
US20110115798A1 (en) | 2007-05-10 | 2011-05-19 | Nayar Shree K | Methods and systems for creating speech-enabled avatars |
US20080306490A1 (en) | 2007-05-18 | 2008-12-11 | Ryan Cameron Lakin | Trackable diagnostic scope apparatus and methods of use |
US20100256504A1 (en) * | 2007-09-25 | 2010-10-07 | Perception Raisonnement Action En Medecine | Methods and apparatus for assisting cartilage diagnostic and therapeutic procedures |
WO2009042644A2 (en) * | 2007-09-25 | 2009-04-02 | Perception Raisonnement Action En Medecine | Methods and apparatus for assisting cartilage diagnostic and therapeutic procedures |
US8902232B2 (en) | 2008-01-31 | 2014-12-02 | University Of Southern California | Facial performance synthesis using deformation driven polynomial displacement maps |
US20150065793A1 (en) | 2008-06-27 | 2015-03-05 | Intuitive Surgical Operations, Inc. | Medical robotic system providing computer generated auxiliary views of a camera instrument for controlling the position and orienting of its tip |
US8771177B2 (en) | 2008-07-08 | 2014-07-08 | Karl Storz Imaging, Inc. | Wide angle flexible endoscope |
US20150065799A1 (en) | 2008-07-08 | 2015-03-05 | Karl Storz Imaging, Inc. | Solid State Variable Direction Of View Endoscope |
EP2153794A2 (en) * | 2008-08-15 | 2010-02-17 | Stryker Leibinger GmbH & Co. KG | System for and method of visualizing an interior of a body |
US20100039506A1 (en) * | 2008-08-15 | 2010-02-18 | Amir Sarvestani | System for and method of visualizing an interior of body |
US20100168562A1 (en) | 2008-12-31 | 2010-07-01 | Intuitive Surgical, Inc. | Fiducial marker design and detection for locating surgical instrument in images |
US20160048953A1 (en) | 2009-03-31 | 2016-02-18 | Intuitive Surgical Operations, Inc | Targets, fixtures, and workflows for calibrating an endoscopic camera |
US8223193B2 (en) | 2009-03-31 | 2012-07-17 | Intuitive Surgical Operations, Inc. | Targets, fixtures, and workflows for calibrating an endoscopic camera |
US20100245541A1 (en) | 2009-03-31 | 2010-09-30 | Intuitive Surgical, Inc. | Targets, fixtures, and workflows for calibrating an endoscopic camera |
US20120120255A1 (en) | 2009-07-21 | 2012-05-17 | Frederic Cao | Method for estimating a defect in an image-capturing system, and associated systems |
US20110075922A1 (en) | 2009-09-30 | 2011-03-31 | Disney Enterprises, Inc. | Apparatus and method for removing ink lines and segmentation of color regions of A 2-D image for converting 2-D images into stereoscopic 3-D images |
US20140031668A1 (en) | 2010-09-08 | 2014-01-30 | Disruptive Navigational Technologies, Llc | Surgical and Medical Instrument Tracking Using a Depth-Sensing Device |
US20120078049A1 (en) | 2010-09-17 | 2012-03-29 | Henke-Sass, Wolf Gmbh | Endoscope with variable direction of view |
US20130281821A1 (en) * | 2011-01-13 | 2013-10-24 | Koninklijke Philips Electronics N.V. | Intraoperative camera calibration for endoscopic surgery |
US9888831B2 (en) | 2011-04-11 | 2018-02-13 | Olympus Corporation | Imaging device and imaging method |
US9398840B2 (en) | 2011-04-20 | 2016-07-26 | Henke-Sass, Wolf Gmbh | Endoscope with variable direction of view |
US20130150863A1 (en) | 2011-06-22 | 2013-06-13 | Adrian Baumgartner | Ultrasound ct registration for positioning |
US20140236159A1 (en) * | 2011-06-27 | 2014-08-21 | Hani Haider | On-board tool tracking system and methods of computer assisted surgery |
WO2013052187A2 (en) * | 2011-06-27 | 2013-04-11 | Board Of Regents Of The University Of Nebraska | On-board tool tracking system and methods of computer assisted surgery |
US20140107471A1 (en) | 2011-06-27 | 2014-04-17 | Hani Haider | On-board tool tracking system and methods of computer assisted surgery |
US20140285676A1 (en) * | 2011-07-25 | 2014-09-25 | Universidade De Coimbra | Method and apparatus for automatic camera calibration using one or more images of a checkerboard pattern |
US9438897B2 (en) | 2011-07-25 | 2016-09-06 | Universidade De Coimbra | Method and apparatus for automatic camera calibration using one or more images of a checkerboard pattern |
WO2013015699A1 (en) | 2011-07-25 | 2013-01-31 | Universidade De Coimbra | Method and apparatus for automatic camera calibration using one or more images of a checkerboard pattern |
US20130034203A1 (en) | 2011-08-03 | 2013-02-07 | Hongkai Wang | 2d/3d registration of a digital mouse atlas with x-ray projection images and optical camera photos |
US20130123801A1 (en) | 2011-11-15 | 2013-05-16 | Macdonald Dettwiler & Associates | Method of real-time tracking of moving/flexible surfaces |
US20140022248A1 (en) | 2012-07-20 | 2014-01-23 | Google Inc. | Determining Three-Dimensional (3D) Object Data Models Based On Object Movement |
US20140035893A1 (en) | 2012-07-31 | 2014-02-06 | Warren Jackson | Color adjustment based on object positioned near display surface |
WO2014054958A2 (en) | 2012-10-05 | 2014-04-10 | Universidade De Coimbra | Method for aligning and tracking point regions in images with radial distortion that outputs motion model parameters, distortion calibration, and variation in zoom |
US20150254872A1 (en) | 2012-10-05 | 2015-09-10 | Universidade De Coimbra | Method for Aligning and Tracking Point Regions in Images with Radial Distortion that Outputs Motion Model Parameters, Distortion Calibration, and Variation in Zoom |
US9367928B2 (en) | 2012-10-05 | 2016-06-14 | Universidade De Coimbra | Method for aligning and tracking point regions in images with radial distortion that outputs motion model parameters, distortion calibration, and variation in zoom |
US9307892B2 (en) | 2012-11-22 | 2016-04-12 | Karl Storz, Gmbh & Co. Kg | Endoscope with an adjustable viewing direction |
US20160000518A1 (en) * | 2013-02-11 | 2016-01-07 | Neomedz Sàrl | Tracking apparatus for tracking an object with respect to a body |
WO2014122301A1 (en) * | 2013-02-11 | 2014-08-14 | Neomedz Sàrl | Tracking apparatus for tracking an object with respect to a body |
US20140327796A1 (en) | 2013-05-02 | 2014-11-06 | National Chung Cheng University | Method for estimating camera response function |
US20160183841A1 (en) | 2013-08-15 | 2016-06-30 | Intuitive Surgical Operations Inc. | Graphical User Interface For Catheter Positioning And Insertion |
US20160220099A1 (en) | 2013-10-16 | 2016-08-04 | Olympus Winter & Ibe Gmbh | Endoscope having an adjustable viewing direction |
US20150297177A1 (en) * | 2014-04-17 | 2015-10-22 | The Johns Hopkins University | Robot assisted ultrasound system |
US9986183B2 (en) | 2014-06-11 | 2018-05-29 | Samsung Electronics Co., Ltd. | Lens distortion correction device and application processor having the same |
US20160148435A1 (en) | 2014-11-26 | 2016-05-26 | Restoration Robotics, Inc. | Gesture-Based Editing of 3D Models for Hair Transplantation Applications |
US20160161602A1 (en) | 2014-12-09 | 2016-06-09 | Toyota Motor Engineering & Manufacturing North America, Inc. | Sensor calibration for autonomous vehicles |
US20160191887A1 (en) | 2014-12-30 | 2016-06-30 | Carlos Quiles Casas | Image-guided surgery with surface reconstruction and augmented reality visualization |
US20160324580A1 (en) | 2015-03-23 | 2016-11-10 | Justin Esterberg | Systems and methods for assisted surgical navigation |
WO2016154557A1 (en) | 2015-03-26 | 2016-09-29 | Universidade De Coimbra | Methods and systems for computer-aided surgery using intra-operative video acquired by a free moving camera |
US10499996B2 (en) | 2015-03-26 | 2019-12-10 | Universidade De Coimbra | Methods and systems for computer-aided surgery using intra-operative video acquired by a free moving camera |
WO2016168307A1 (en) | 2015-04-13 | 2016-10-20 | Universidade De Coimbra | Methods and systems for camera characterization in terms of response function, color, and vignetting under non-uniform illumination |
US20180089855A1 (en) | 2015-04-13 | 2018-03-29 | Universidade De Coimbra | Methods and Systems for Camera Characterization in Terms of Response Function, Color, and Vignetting Under Non-Uniform Illumination |
US10504239B2 (en) | 2015-04-13 | 2019-12-10 | Universidade De Coimbra | Methods and systems for camera characterization in terms of response function, color, and vignetting under non-uniform illumination |
US20170325669A1 (en) | 2015-05-12 | 2017-11-16 | Avraham Levy | Dynamic field of view endoscope |
EP3130276A1 (en) | 2015-08-12 | 2017-02-15 | M.S.T. Medical Surgery Technologies Ltd. | Endoscope with wide angle lens and adjustable view |
US20180049622A1 (en) | 2016-08-16 | 2018-02-22 | Insight Medical Systems, Inc. | Systems and methods for sensory augmentation in medical procedures |
US11335075B2 (en) | 2017-03-14 | 2022-05-17 | Universidade De Coimbra | Systems and methods for 3D registration of curves and surfaces using local differential information |
WO2018232322A1 (en) | 2017-06-15 | 2018-12-20 | Children's National Medical Center | System, apparatus and method for calibrating oblique-viewing rigid endoscope |
Non-Patent Citations (74)
Title |
---|
A. Agrawal, Y. Taguchi, and S. Ramalingam, "Analytical forward projection for axial non-central diotropic and catadioptric cameras," presented at the ECCV'10: Proceedings of the 11th European Conference on Computer Vision: Part III, Sep. 2010. |
A. Fitzgibbon, M. Pilu, and R. Fisher, "Direct least square fitting of ellipses," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 21, No. 5, pp. 476-480, May 1999. |
A. Rav-Acha and S. Peleg. Lucas-Kanade without Iterative Warping. In IEEE Int. Conf. Image Process., pp. 1097-1100, Oct. 2006. |
A. Salazar-Garibay, E. Malis, and C. Mei. Visual tracking of planes with an uncalibrated central catadioptric camera. In IROS, Mar. 2009. |
Alexander Behrens, Michael Bommes, Thomas Stehle, Sebastian Gross, Steffen Leonhardt, and Til Aach. Real-time image composition of bladder mosaics in fluorescence endoscopy. Computer Science Research and Development, 26:51-64, Feb. 2011. |
Alper Yilmaz, Omar Javed, and Mubarak Shah. Object Tracking: A survey. ACM Comput. Surv., 38, Dec. 2006. |
B. Chapman, G. Jost, and R. Van Der Pass, Using OpenMP: Portable Shared Memory Parallel Programming Scientific Computation and Engineering Series). The MIT Press, 2008. |
Bill Triggs, Philip F. McLauchlan, Richard I. Hartley, and Andrew W. Fitzgibbon. Bundle adjustment—a modern synthesis. In Proceedings of the International Workshop on Vision Algorithms: Theory and Practice, ICCV '99, pp. 298-372, London, UK, Springer-Verlag, 2000. |
Bruce D. Lucas and Takeo Kanade. An Iterative Image Registration Technique with an Application to Stereo Vision; In DARPA Image Understanding Workshop, pp. 121-130, Apr. 1981. |
C. Mei, S. Benhimane, E. Malis, and P. Rives. Constrained multiple planar template tracking for central catadioptric cameras. In British Machine Vision Conference, Sep. 2006. |
C. Mei, S. Benhimane, E. Malis, and P. Rives. Efficient Homography-based Tracking and 3D Reconstruction for Single Viewpoint Sensors. IEEE Trans Robotics, Dec. 2008. |
C. Wengert, M. Reef f, P. Cattin, and G. Siekely, "Fully automatic endoscope calibration for intraoperative use," in Bildverarbeitungfür die Medizin 2006, pp. 419-423, Mar. 2006. |
C. Wu, B. Jaramaz, and S. G. Narasimhan, "A Full Geometric and Photometric Calibration Method for Oblique-viewing Endoscope," International Journal of Computer Aided Surgery, vol. 15, pp. 19-31, Apr. 2010. |
Carr, Peter et al.; "Point-less Calibration: Camera Parameters from Gradient-Based Alignment to Edge Images"; Applications of Computer Vision (WACV), pp. 377-384, Jan. 9, 2012. |
Chen, X. et al., "A Theoretical Analysis of Camera Response Functions in Image Deblurring" European Conference on Computer Vision, 2012. |
D. Claus and A. Fitzgibbon, "A Rational Function lens distortion model for general cameras," Computer Vision and Pattern Recognition IEEE Computer Society Conference on , pp. 213-219, vol. 1, Jun. 2005. |
Darius Burschka, Ming Li, Russell H. Taylor, and Gregory D. Hager. Scale-Invariant Registration of Monocular Endoscopic Images to CT-Scans for Sinus Surgery. In Med. Image Comput. and Computer-Assist. Inter., Sep. 2004. |
David Nistér. An Efficient Solution to the Five-Point Relative Pose Problem. IEEE Trans. Pati Anal. Mach. Intell., Jun. 26, 2004. |
Fitzgibbon, A.W.; "Simultaneous Linear Estimation of Multiple View Geometry and Lens Distortion;" Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition; CVPR 2001, vol. 1, pp. 1-8, Jan. 1, 2001. |
Fu, L. et al., "Camera Response Function Estimation and Application with a Single Image" Informatics in Control, Automation and Robotics, vol. 2, LNEE 133, pp. 149-156, 2011. |
Greg Welch and Gary Bishop. An Introduction to the Kalman Filter. Technical report, University of North Carolina at Chapel Hill, Chapel Hill, N.C., USA, 1995; updated Jul. 2006. |
Harney, L. "Simultaneous Estimation of Camera Response Function, Target Reflectance and Irradiance Values" Department of Computing, Macquarie University, Australia, Conference Dec. 6-8, 2005, Digital Image Computing: Techniques and Applications 2005. |
J. Barreto, J. Santos, P. Menezes, and F. Fonseca, "Ray-based Calibration of Rigid Medical Endoscopes," in OMNIVIS, Sep. 2008. |
J. Helferty, C. Zhang, G. McLennan, and W Higgins, "Videoendoscopic distortion correction and its application to virtual guidance of endoscopy," IEEE Transactions on Medical Imaging, vol. 20, No. 7, pp. 605-617, Jul. 2001. |
J. Mallon and P. F. Whelan, "Which pattern? Biasing aspects of planar calibration patterns and detection methods," Pattern Recognition Letters, vol. 28, No. 8, pp. 921-930, Jan. 2007. |
J.-Y. Bouguet. Camera Calibration Toolbox for Matlab. [Online]. Available: http://www.vision.caltech.edu/bouguetj/calibdoc/index.html#ref; Last Updated Oct. 14, 2015. |
Jalobeanu et al. "Modeling Images of Natural 30 Surfaces: Overview and Potential Applications"; IEEE; Publication [online]. Jun. 2004. [retrieved Jun. 24, 2016]. Retrieved from the Internet: <URL: http://citeseerx.ist.psu.edu/vlewdoc/download?doi=10.1.1.59.1141&rep=rep1 &type=pdf>; pp. 1-9. |
Jean-Yves Bouguet. Pyramidal Implementation of the Lucas Kanade Feature Tracker Description of the algorithm, 2000. |
Jianbo Shi and C. Tomasi. Good features to track. In IEEE Conf. Vis. Patt. Recogn., pp. 593 600, Jun. 1994. |
Joao P. Barreto, Jose Roquette, Peter Sturm, and Fernando Fonseca. Automatic Camera Calibration Applied to Medical Endoscopy. In Brit. Mach. Vis. Conf., Sep. 2009. |
Joao P. Barreto. A Unifying Geometric Representation for Central Projection Systems. Comput. Vis. Imag. Unders., 103(3):208-217, Jun. 2006. |
K. Daniilidis, A. Makadia, and T. Bulow. Image Processing in Catadiop-tric Planes: Spaciotemporal Derivatives and Optical Flow Computation. In Int. Workshop on Omndirectional Vision, Jun. 2002. |
K. Vijayan-Asari, S. Kumar, and D. Radhakrishnan, "A new approach for nonlinear distortion correction in endoscopic images based on least squares estimation," IEEE Transactions on Medical Imaging, vol. 18, No. 4, pp. 345-354, Apr. 1999. |
Kalman, R.E.; A New Approach to Linear Filtering and Prediction Problems; ASME—Journal of Basic Engineering, 82 (Series D): 35-45; Mar. 1960. |
Kevin Koeser, Bogumil Bartczak, and Reinhard Koch. Robust GPU-assisted camera tracking using free-form surface models. Journal of Real-Time Image Processing, 2(2): 133-147, Oct. 2007. |
Kim, Dong Sik et al.; "Joint Optimization of Spatial Registration and Histogram Compensation for Microscopic Images"; Conference Proceedings: Annual International Conference of the IEEE Engineering in Medicine and Biology Society; pp. 3779-3782; Aug. 30, 2006. |
Kim, Seon Joo et al.; "Joint Feature Tracking and Radiometric Calibration from Auto-Exposure Video"; Computer Vision, 2007, pp. 1-8, Oct. 1, 2007. |
L. Matthews, T. Ishikawa, and S. Baker. The Template Update Problem. IEEE Trans. Pall. Anal. Mach. Intell., 26(6):810-815, Jun. 2004. |
Li, X., et al., "Symmetry and Template Guided Completion of Damaged Skulls", Computers & Graphics, vol. 35, pp. 885-893, Jan. 31, 2011. |
Lin, S. et al., "Radiometric Calibration from a Single Image" Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. |
M. A. Fischler and R. C. Bolles, "Random Sample Consensus: A Paradigm for Model Fitting with Applications to Image Analysis and Automated Cartography," Commun. ACM, vol. 24, No. 6, pp. 381-395, Jun. 1981. |
M. Lourenco, J. P. Barreto, and F. Vasconcelos. sRD-SIFT: Keypoint Detection and Matching in Images With Radial Distortion. IEEE Trans Robotics, Jun. 2012. |
M. Menem, "Constraints on perspective images and circular panoramas," BMVC, Sep. 2004. |
Marc Pollefeys, Luc Van Gool, Maarten Vergauwen, Frank Verbiest, Kurt Cornelis, Jan Tops, and Reinhard Koch. Visual Modeling with a Hand-Held Camera. Int. J. Comput. Vis., 59(3):207-232, Sep. 2004. |
Miguel Lourenco and Joao P. Barreto. Tracking features in uncalibrated images with radial distortion. In Eur. Conf. Comput. Vis., pp. 1-14, Oct. 2012. |
Mitra, Niloy J., et al., "Registration of Point Cloud Data from a Geometric Optimization Perspective", Eurographics Symposium on Geometry Processing, pp. 23-32, 2004. |
Myung Hwangbo, Jun-Sik Kim, and Takeo Kanade. Gyro-aided feature tracking for a moving camera: fusion, auto-calibration and GPU implementation. Int. J. of Robot. Res., 30(14):1755-1774, Dec. 2011. |
N. Fukuda, Y. Chen, M. Nakamoto, and T, "A scope cylinder rotation tracking method for oblique-viewing endoscopes without attached sensing device," Software Engineering and Data Mining, No. 1, pp. 684-687, Jun. 2010. |
N. Smith, N. Vakil, and S. Maislin, "Correction of Distortion in Endoscope Images," IEEE Transactions on Medical Imaging, vol. 11, No. 1, pp. 117-122, Mar. 1992. |
Ng, T. et al., "Using Geometry Invariants for Camera Response Function Estimation" IEEE Conference on Computer Vision and Pattern Recognition Jun. 17-22, CVPR 2007. |
P. Baker, C. Fermuller, Y. Aloimonos, and R. Pless. A Spherical Eye from Multiple Cameras (Makes Better Models of the World). In IEEE Conf. Vis. Patt. Recogn., Feb. 2001. |
P. Sturm, S. Ramalingam,J.-P. Tardif, S. Gasparini and J. Barreto, Camera Models and Fundamental Concepts Used in Geometric Computer Vision. Now Publishers, Inc., Jan. 2011. |
PCT International Search Report in International Application No. PCT/PT2016/024262 dated Jun. 24, 2016. |
Peter Hansen, Peter Corke, and Wageeh Boles. Wide-Angle Visual Feature Matching for Outdoor Localization. Int. J. of Robot. Res., 29:267-297, Feb. 2010. |
Qiu, S. et al., "Estimation Method of CCD and CMOS Response Functions Based on a Single Image" International Conference on Optical Instruments and Technology: Optoelectronic Imaging and Process Technology, Proc. SPIE, 751325, Nov. 24, 2009. |
R. Melo, J.P. Barreto, and G. Falcao. A new solution for camera calibration and real-time image distortion correction in medical endoscopy—initial technical evaluation. Biomedical Engineering, IEEE Transactions on, 59(3):634-644, Mar. 2012. |
R. Shahidi, M. Bax, C. Maurer, J. Johnson, E. Wilkinson, B. Wang, J. West, M. Citardi, K. Manwaring, and R. Khadem, "Implementation, calibration and accuracy testing of an image-enhanced endoscopy system," IEEE Transactions on Medical Imaging, vol. 21, No. 12, pp. 1524-1535, Dec. 2002. |
Reg G. Willson and Steven A. Shafer. What is the center of the image? J. Opt. Soc. Am. A, 11(11):2946-2955, Apr. 1993. |
S. D. Buck, F. Maes, A. D'Hoore, and P. Suetens, "Evaluation of a novel calibration technique for optically tracked oblique laparoscopes," Proceedings of the 10th international conference on Medical image computing and computer-assisted intervention—vol. Part I, pp. 467-474, Feb. 2007. |
Simon Baker and Iain Matthews. Equivalence and Efficiency of Image Alignment Algorithms. In IEEE Conf. Vis. Patt. Recogn., vol. 1, pp. 1090-1097, Dec. 2001. |
Simon Baker and Iain Matthews. Lucas-kanade 20 years on: A unifying framework. Int. J. Comput. Vis., 56(3):221-255, Mar. 2004. |
Simon Baker, Daniel Scharstein, J. P. Lewis, Stefan Roth, Michael J. Black, and Richard Szeliski. A database and evaluation methodology for optical flow. Int. J. Comput. Vision, 92(1), Dec. 2011. |
Song, KS et al. "Region Adaptive Correction Method for Radial Distortion of Fish-Eye Image;" Image Processing: Algorithms and Systems X; and Parallel Processing for Imaging Applications II, SPIE, vol. 8295, No. 1., pp. 1-7, Feb. 9, 2012. |
Steffen Gauglitz, Tobias Hollerer, and Matthew Turk. Evaluation of Interest Point Detectors and Feature Descriptors for Visual Tracking. Int. J. Comput. Vis., 94(3):335-360, Mar. 2011. |
Supplementary European Search Report based on EP 16769785 dated Oct. 26, 2018. |
T. Brox and J. Malik. Large displacement optical flow: descriptor matching in variational motion estimation. IEEE Trans. Patt. Anal. Mach. Intell., 33(3):500-513, Mar. 2011. |
T. Stehle, M. Hennes, S. Gross, A. Behrens, J. Wulff, and T. Aach, "Dynamic Distortion Correction for Endoscopy Systems with Exchangeable Optics," in Bildverarbeitungfür die Medizin 2009. Berlin: Springer, pp. 142-146, 2009. |
T. Yamaguchi, M. Nakamoto, Y. Sato, Y. Nakajima, K. Konishi, M. Hashizume, T. Nishii, N. Sugano, H. Yoshikawa, K. Yonenobu, and S. Tamura, "Camera Model and Calibration Procedure for Oblique-Viewing Endoscope," in MICCAI, pp. 373-381, Nov. 2003. |
Toru Tamaki, Tsuyoshi Yamamura, and Noboru Ohnishi. Unified approach to image distortion. In ICPR, pp. 584-587, Aug. 2002. |
USPTO Office Action in U.S. Appl. No. 15/566,536 dated Apr. 4, 2019. |
Y. Chang, "Multi-view 3D Reconstruction for Scenes Under the Refractive plane with known vertical direction," Computer Vision (ICCV), Nov. 2011. |
Yu, W. "Practical Anti-Vignetting Methods for Digital Cameras" IEEE Transactions on Consumer Electronics, vol. 50, Issue 4, pp. 975-983, Nov. 2004. |
Z. Zhang, "Flexible camera calibration by viewing a plane from unknown orientations," in ICCV, pp. 666-673, Sep. 1999. |
Zheng, Y. et al., "Single-Image Vignetting Correction" IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 31, Issue 12, pp. 2243-2256, Oct. 31, 2008. |
Also Published As
Publication number | Publication date |
---|---|
US20180071032A1 (en) | 2018-03-15 |
EP3273854B1 (en) | 2021-09-22 |
EP3273854A1 (en) | 2018-01-31 |
WO2016154557A1 (en) | 2016-09-29 |
EP3273854A4 (en) | 2018-12-05 |
US10499996B2 (en) | 2019-12-10 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
USRE49930E1 (en) | Methods and systems for computer-aided surgery using intra-operative video acquired by a free moving camera | |
US20230355312A1 (en) | Method and system for computer guided surgery | |
US20210307842A1 (en) | Surgical system having assisted navigation | |
CN109069208B (en) | Ultra-wideband positioning for wireless ultrasound tracking and communication | |
US20210121237A1 (en) | Systems and methods for augmented reality display in navigated surgeries | |
US11382698B2 (en) | Surgical navigation system | |
JP2022116157A (en) | Systems for sensory augmentation in medical procedures | |
US8109942B2 (en) | Computer-aided methods, systems, and apparatuses for shoulder arthroplasty | |
US20130211232A1 (en) | Arthroscopic Surgical Planning and Execution with 3D Imaging | |
AU2020201097A1 (en) | Systems for assisting surgery | |
US20070038059A1 (en) | Implant and instrument morphing | |
WO2007115825A1 (en) | Registration-free augmentation device and method | |
JP2009273521A (en) | Navigation system for arthroscopical surgery | |
US20230190136A1 (en) | Systems and methods for computer-assisted shape measurements in video | |
US20230233259A1 (en) | Augmented reality headset systems and methods for surgical planning and guidance for knee surgery | |
Wengert et al. | Endoscopic navigation for minimally invasive suturing | |
JP2009172124A (en) | Surgery navigation system, image display method, computer program and recording medium | |
Bisson et al. | 3D visualization tool for minimally invasive discectomy assistance | |
US20240197410A1 (en) | Systems and methods for guiding drilled hole placement in endoscopic procedures | |
Zhang et al. | Towards an Accurate Augmented-Reality-Assisted Orthopedic Surgical Robotic System Using Bidirectional Generalized Point Set Registration | |
CN117835889A (en) | Ligament repair method and system | |
Jianxi et al. | Design of a computer aided surgical navigation system based on C-arm |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
FEPP | Fee payment procedure |
Free format text: ENTITY STATUS SET TO UNDISCOUNTED (ORIGINAL EVENT CODE: BIG.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY |
|
AS | Assignment |
Owner name: UNIVERSIDADE DE COIMBRA, PORTUGAL Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:DE ALMEIDA BARRETO, JOAO PEDRO;REEL/FRAME:066401/0221 Effective date: 20171215 |