US20110238079A1 - Surgical Cockpit Comprising Multisensory and Multimodal Interfaces for Robotic Surgery and Methods Related Thereto - Google Patents

Surgical Cockpit Comprising Multisensory and Multimodal Interfaces for Robotic Surgery and Methods Related Thereto Download PDF

Info

Publication number
US20110238079A1
US20110238079A1 US13/051,921 US201113051921A US2011238079A1 US 20110238079 A1 US20110238079 A1 US 20110238079A1 US 201113051921 A US201113051921 A US 201113051921A US 2011238079 A1 US2011238079 A1 US 2011238079A1
Authority
US
United States
Prior art keywords
surgical
local
remote
cockpit
operator
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Abandoned
Application number
US13/051,921
Inventor
Blake Hannaford
Louis Kim
Thomas S. Lendvay
Kristen S. Moe
James S. Pridgeon
Jacob Rosen
Laligam Sekhar
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
SPI SURGICAL Inc
Original Assignee
SPI SURGICAL Inc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by SPI SURGICAL Inc filed Critical SPI SURGICAL Inc
Priority to US13/051,921 priority Critical patent/US20110238079A1/en
Publication of US20110238079A1 publication Critical patent/US20110238079A1/en
Assigned to SPI Surgical, Inc. reassignment SPI Surgical, Inc. ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: PRIDGEON, JAMES S., KIM, LOUIS, SEKHAR, LALIGAM, LENDVAY, THOMAS S., HANNAFORD, BLAKE, MOE, KRISTEN S., ROSEN, JACOB
Priority to US14/181,388 priority patent/US9474580B2/en
Abandoned legal-status Critical Current

Links

Images

Classifications

    • AHUMAN NECESSITIES
    • A61MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61BDIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B34/00Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
    • A61B34/30Surgical robots
    • A61B34/35Surgical robots for telesurgery
    • AHUMAN NECESSITIES
    • A61MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61BDIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B17/00Surgical instruments, devices or methods, e.g. tourniquets
    • A61B17/00234Surgical instruments, devices or methods, e.g. tourniquets for minimally invasive surgery
    • AHUMAN NECESSITIES
    • A61MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61BDIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B17/00Surgical instruments, devices or methods, e.g. tourniquets
    • A61B17/34Trocars; Puncturing needles
    • A61B17/3417Details of tips or shafts, e.g. grooves, expandable, bendable; Multiple coaxial sliding cannulas, e.g. for dilating
    • A61B17/3421Cannulas
    • AHUMAN NECESSITIES
    • A61MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61BDIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B17/00Surgical instruments, devices or methods, e.g. tourniquets
    • A61B17/34Trocars; Puncturing needles
    • A61B17/3417Details of tips or shafts, e.g. grooves, expandable, bendable; Multiple coaxial sliding cannulas, e.g. for dilating
    • A61B17/3421Cannulas
    • A61B17/3423Access ports, e.g. toroid shape introducers for instruments or hands
    • AHUMAN NECESSITIES
    • A61MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61BDIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B34/00Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
    • A61B34/30Surgical robots
    • AHUMAN NECESSITIES
    • A61MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61BDIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B34/00Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
    • A61B34/70Manipulators specially adapted for use in surgery
    • A61B34/74Manipulators with manual electric input means
    • AHUMAN NECESSITIES
    • A61MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61BDIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B34/00Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
    • A61B34/70Manipulators specially adapted for use in surgery
    • A61B34/76Manipulators having means for providing feel, e.g. force or tactile feedback
    • AHUMAN NECESSITIES
    • A61MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61BDIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B90/00Instruments, implements or accessories specially adapted for surgery or diagnosis and not covered by any of the groups A61B1/00 - A61B50/00, e.g. for luxation treatment or for protecting wound edges
    • A61B90/60Supports for surgeons, e.g. chairs or hand supports
    • AHUMAN NECESSITIES
    • A61MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61MDEVICES FOR INTRODUCING MEDIA INTO, OR ONTO, THE BODY; DEVICES FOR TRANSDUCING BODY MEDIA OR FOR TAKING MEDIA FROM THE BODY; DEVICES FOR PRODUCING OR ENDING SLEEP OR STUPOR
    • A61M29/00Dilators with or without means for introducing media, e.g. remedies
    • BPERFORMING OPERATIONS; TRANSPORTING
    • B25HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
    • B25JMANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
    • B25J13/00Controls for manipulators
    • B25J13/02Hand grip control means
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G06F3/014Hand-worn input/output arrangements, e.g. data gloves
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/016Input arrangements with force or tactile feedback as computer generated output to the user
    • GPHYSICS
    • G16INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
    • G16HHEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
    • G16H20/00ICT specially adapted for therapies or health-improving plans, e.g. for handling prescriptions, for steering therapy or for monitoring patient compliance
    • G16H20/40ICT specially adapted for therapies or health-improving plans, e.g. for handling prescriptions, for steering therapy or for monitoring patient compliance relating to mechanical, radiation or invasive therapies, e.g. surgery, laser therapy, dialysis or acupuncture
    • GPHYSICS
    • G16INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
    • G16HHEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
    • G16H40/00ICT specially adapted for the management or administration of healthcare resources or facilities; ICT specially adapted for the management or operation of medical equipment or devices
    • G16H40/60ICT specially adapted for the management or administration of healthcare resources or facilities; ICT specially adapted for the management or operation of medical equipment or devices for the operation of medical equipment or devices
    • G16H40/67ICT specially adapted for the management or administration of healthcare resources or facilities; ICT specially adapted for the management or operation of medical equipment or devices for the operation of medical equipment or devices for remote operation
    • GPHYSICS
    • G16INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
    • G16HHEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
    • G16H80/00ICT specially adapted for facilitating communication between medical practitioners or patients, e.g. for collaborative diagnosis, therapy or health monitoring
    • AHUMAN NECESSITIES
    • A61MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61BDIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B17/00Surgical instruments, devices or methods, e.g. tourniquets
    • A61B2017/00973Surgical instruments, devices or methods, e.g. tourniquets pedal-operated
    • AHUMAN NECESSITIES
    • A61MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61BDIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B17/00Surgical instruments, devices or methods, e.g. tourniquets
    • A61B17/34Trocars; Puncturing needles
    • A61B17/3417Details of tips or shafts, e.g. grooves, expandable, bendable; Multiple coaxial sliding cannulas, e.g. for dilating
    • A61B17/3421Cannulas
    • A61B2017/3433Cannulas with different outer diameters of the cannula
    • AHUMAN NECESSITIES
    • A61MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61BDIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B17/00Surgical instruments, devices or methods, e.g. tourniquets
    • A61B17/34Trocars; Puncturing needles
    • A61B17/3417Details of tips or shafts, e.g. grooves, expandable, bendable; Multiple coaxial sliding cannulas, e.g. for dilating
    • A61B2017/3454Details of tips

Definitions

  • war fighters remain vulnerable to blast wounds, burns, and multiple penetrating injuries not usually encountered in civilian settings.
  • war surgery is aimed to treat combat casualties at far forward locations and under austere conditions continues to save lives, the nature of battlefield injuries shortens the “golden hour” in which highly skilled medical attention is required to stabilize the soldier.
  • present systems and methods, etc. provide these and/or other advantages.
  • Present systems and methods, etc. extend the spectrum of surgical expertise to a level appropriate for the type and nature of battlefield wounds.
  • the present systems and methods, etc. also extend to any desired area of surgery including those well beyond the military arena.
  • the present systems and methods, etc. comprise local surgical cockpits comprising local surgical consoles that can communicate with any desired remote surgical module (surgical robot), for example via a shared Transmission Control Protocol/Internet Protocol (TCP/IP) or other unified open source communication protocol or other suitable communication system.
  • the systems and methods, etc., herein can also comprise a modular approach wherein multiple surgical consoles can network supporting collaborative surgery regardless of the physical location of the surgeons or other operators relative to each other and/or relative to the surgical site.
  • a surgeon operating a local surgical cockpit can teleoperate using a remote surgical module on a patient in the same room as the surgeon, or surgeons located in multiple safe locations can telemanipulate remote multiple surgical robots on a patient in or close to a war zone or any remote location.
  • One aspect of the systems, methods, etc., herein is to provide a multisensory, multi-modal surgical workstation (surgical cockpit).
  • This workstation can provide a command post allowing the surgeon(s) to visually immerse themselves into the remote surgical space. It provides peripheral information such as vital signs, as well as visual feeds from the operating room (OR) surrounding the actual surgical site on or in the patient.
  • the surgical console provides high situational awareness as well as the capability to dynamically interact with the other functions of the OR. This is accomplished by multiple streams of visual, audio, and kinesthetic inputs. Special attention can also paid to avoiding information overload of the surgeon.
  • the present methods, devices, systems, etc. are related to a local surgical cockpit comprising a base, a frame disposed on the base, a seat for an operator disposed on the frame, and a remote surgical console configured such that the operator can operate the console for remote surgery while in the seat, wherein the seat can be ergonomic and can be operably connected to the frame such that the seat can be retainably tilted from a substantially upright position to a substantially supine position.
  • the seat can comprise an independently movable headrest, backrest, seating plate and footrest, the seat further comprising positioning elements operably connected to the independently movable headrest, backrest, seating plate and footrest and providing at least three axes of retainable positioning movement for each of the independently movable headrest, backrest, seating plate and footrest.
  • the seat can comprise a lumbar support comprising retainable positioning movement for support of the lower back.
  • the reference body posture of the seat can correspond to a human body posture that can be fully relaxed in micro gravity.
  • the cockpit further can comprise at least one peripheral device operably connected to move with the seat when the seat is moved so that the location of the peripheral device relative to the operator in the seat is substantially unchanged.
  • the peripheral device can be at least one of a monitor facing an operator in the seat and operably linked to display a remote surgical site, a heads-up display disposed in front of the local surgeon's eyes, and an input device disposed at a hand of the operator and operably linked to provide input to a corresponding device located at the remote surgical site.
  • a local surgical cockpit comprising a local surgical console configured for transmitting surgical movements of local surgeon operating the local surgical console to a remote surgery site, and a head-mounted display disposed in front of the local surgeon's eyes in surgical position in the cockpit to operate the console for surgery, wherein the head-mounted display can be configured to depict at least images of a remote surgical site under remote operation by the operator.
  • the local surgical cockpit can be part of a system and the system further can comprise remote image sensors operably connected to the head-mounted display to transmit the image of the remote surgical site.
  • the head-mounted display can extend to the local surgeon's eyes from an articulated boom or other retention structure disposed in front of the local surgeon's eyes, which retention structure can be actuated by at least one hand control located on the cockpit, or by voice control or otherwise as desired.
  • the head-mounted display can be disposed on a head-mounted frame configured to rest on an operator's head and to maintain the images in front of the local surgeon's eyes when the operator's head moves.
  • the head-mounted display can comprise two separate streams of video displayed to each eye of the local surgeon's eyes, each stream comprising corresponding right and left eye views of a remote surgical site to provide a 3-D image of the site.
  • the cockpit further can comprise at least one monitor operably held to the base of the cockpit, and the head-mounted display can comprise two separate streams of video displayed to each eye of the local surgeon's eyes, each stream comprising corresponding right and left eye views of a remote surgical site to provide a 3-D image of the site.
  • the monitor(s) can also be 3-D.
  • the systems, etc. are directed to a local surgical cockpit comprising a local surgical console configured for transmitting surgical movements of an operator operating the local surgical console to a remote surgery site, and comprising at least one image display device configured to depict at least one image of the remote surgical site, the display device further depicting augmented reality for the operator comprising augmented information shown on the display and superimposed over the image of the remote surgical site.
  • the local surgical cockpit can be part of a system and the system further can comprise remote image sensors operably connected to the head-mounted display to transmit the image of the remote surgical site.
  • the augmented information can comprise at least one of preselected margins to dissect during the surgery and a mask of vital structures in the remote surgical site overlaid over the images of the remote surgical site.
  • the display device can further display further augmented information either to a side of or superimposed over the image of the remote surgical site and the further augmented information can comprise at least one of blood pressure, temperature, O 2 level, CO 2 level, intracranial pressure, a preplanned trajectory for a surgical tool, tool type, suction on/off, a bottom task bar, recording capabilities, current time, and elapsed time.
  • the image of the remote surgical site and the augmented information can comprise blending graphical images with real-world views of the remote surgical sit, and can be provided by at least one of an endoscopic camera, a remote surgical site camera, or a camera showing an operating room.
  • the systems, etc. are directed to a local surgical cockpit comprising a local surgical console configured for transmitting surgical movements of an operator operating the local surgical console to a remote surgery site, and comprising a local surgical instrument comprising local input surgical fingers configured to provide input to corresponding remote surgical fingers configured to manipulate a remote surgical instrument at a remote operation site, wherein the local fingers can be high frequency haptic fingers configured to provide tactile feedback to the operator based on acceleration of the remote surgical instrument manipulated by the remote surgical fingers.
  • the local surgical cockpit can be part of a system and the system further can comprise the remote surgical fingers, and wherein the remote surgical fingers can be haptic fingers configured to provide tactile feedback to the operator based on acceleration of the remote surgical instrument manipulated by the remote surgical fingers.
  • the local surgical cockpits can be configured such that operators in different locales can operate simultaneously on a single surgical site; such that operators can relieve each other in a single surgery at a single surgical site; or to provide a teaching surgical cockpit and a student surgical cockpit providing haptic feedback to a student operator generated by a teaching operator.
  • the haptic feedback to the student can comprise movements of a remote surgical instrument controlled by the teaching operator or tactile feedback from a surgical site being operated on by the teaching operator.
  • the systems, etc. are directed to a local surgical cockpit comprising a local surgical console configured for transmitting surgical movements of an operator operating the local surgical console to a remote surgery site, and comprising at least seven degrees of freedom for a local surgical instrument manipulated by a robotic arm manipulated by the operator, wherein the console can be configured such that the seven degrees of freedom can be transmissible to a remote surgical instrument located at a remote surgical site and manipulated by the operator operating the console.
  • the local surgical cockpit can be part of a system and the system further can comprise the remote surgical instrument operably connected to the local surgical instrument such that the remote surgical instrument precisely responds in at least seven corresponding degrees of freedom to movements of the local surgical instrument.
  • the degrees of freedom can comprise at least nine degrees of freedom for the local surgical instrument manipulated by the operator and a corresponding nine degrees of freedom for the remote surgical instrument.
  • the degrees of freedom can comprise at least twelve degrees of freedom for the local surgical instrument manipulated by the operator and a corresponding twelve degrees of freedom for the remote surgical instrument, wherein the local robotic arm can comprise a shoulder joint, an elbow joint, a wrist joint and the three fingers, each comprising at least the following degrees of freedom: shoulder can comprise 2 degrees of freedom; elbow can comprise 1 degree of freedom; wrist can comprise 3 degrees of freedom; the three fingers can comprise 2 degrees of freedom each.
  • the local surgical instrument can comprise at least three input fingers configured to provide input to a corresponding at least three remote surgical fingers configured to manipulate a remote surgical instrument at a remote operation site, wherein the at least three input fingers can be configured to be manipulated by a single hand of an operator operating the local surgical instrument, and wherein the at least seven degrees of freedom can comprise at least two degrees of freedom for two of the three remote surgical fingers and at least three degrees of freedom for a third of the three remote surgical fingers, or the degrees of freedom can comprise at least nine degrees of freedom comprising at least three degrees of freedom for each of the three remote surgical fingers.
  • the local robotic arm can comprise a shoulder joint, an elbow joint, a wrist joint and the three fingers, each comprising at least the following degrees of freedom: shoulder can comprise 2 degrees of freedom; elbow can comprise 1 degree of freedom; wrist can comprise 3 degrees of freedom; the three fingers can comprise 2 degrees of freedom each. Or the three fingers can comprise 3 degrees of freedom each.
  • the degrees of freedom provide for variable desired positioning and orientation of a tip of the remote surgical instrument in space in 6 parameters including Cartesian position (x,y,z), and angular orientation (x y z ⁇ , ⁇ , ⁇ ).
  • Control of the remote surgical instrument further can comprise scaling factors configured such that motion input by the operator can be attenuated or amplified with respect to the remote surgical instrument.
  • Control further can comprise indexing configured to allow the operator to disengage the input device from the remote surgical instrument to reposition his/her arms and engage again.
  • the systems, etc. are directed to a local surgical cockpit comprising a local surgical console configured for transmitting surgical movements of an operator operating the local surgical console to a remote surgery site, and comprising a local surgical instrument comprising at least three input fingers configured to provide input to a corresponding at least three remote surgical fingers configured to manipulate a remote surgical instrument at a remote operation site.
  • the at least three input fingers can be configured to be manipulated by a single hand of an operator operating the local surgical instrument.
  • the local surgical cockpit can be part of a system and the system further can comprise the three remote surgical fingers operably connected to the three input fingers such that the three remote surgical fingers precisely respond to movements of the three input fingers.
  • the at least three input fingers can be configured to correspond respectively to a) an operator's thumb, b) an operator's index and middle fingers, and c) an operator's ring and little fingers; or to a) an operator's thumb, b) an operator's index finger, and c) an operator's middle, ring and little fingers.
  • the at least three input fingers can be haptic fingers configured to provide tactile feedback to the operator based on acceleration of a remote surgical instrument manipulated by the remote surgical fingers.
  • the three input fingers can be operably connected so that two fingers control remote surgical fingers and the remaining third finger controls an external device, which can be a one or more of an electrocautery device, a laser photocoagulator, a staple applier or other device as desired.
  • the external device can also be an optical aspect of the camera system such as focus, zoom, rotation, or field-of-view.
  • the systems, etc. are directed to a local surgical cockpit comprising a local surgical console configured for transmitting surgical movements of an operator operating the local surgical console to a remote surgery site, and comprising at least one haptic foot pedal configured to be operably connected to at least one remote device at a remote operation site, wherein the at least one haptic foot pedal can be configured to be manipulated by a foot of the operator operating the local surgical console to cause a movement or control change in the remote device.
  • the local surgical cockpit can be part of a system and the system further can comprise the at least one remote device operably connected to the at least one haptic foot pedal.
  • the local surgical cockpit further can comprise at least two haptic foot pedals configured to be operably connected to the at least one remote device at the remote operation site.
  • the local surgical cockpit further can comprise at least eight haptic foot pedals configured to be operably connected to at least two remote devices at the remote operation site, the at least eight haptic foot pedals divided to provide at least a first foot pedal set and second foot pedal set, wherein a first foot pedal set and second foot pedal set can be each disposed to be manipulated by a right foot of the operator and by a left foot of the operator, respectively.
  • the sets can each contain four pedals with each of the four pedals in one of four quadrants of a circle.
  • the pedals can also be set in an array. Opposed or otherwise set off pairs of pedals can be assigned opposed functions at the remote surgical site. The opposed functions can be suction and irrigation.
  • the four pedals can also be assigned complementary functions for a remote instrument at the remote surgical site. The four pedals can control the viewing angles of an endoscopic camera.
  • the local surgical cockpit further can comprise a dead zone that prevents two opposing functions being implemented simultaneously.
  • the at least one haptic foot pedal can also control at least one of camera angle, camera zoom, camera focus, irrigation, suction, robot brakes, electric coagulation, laser photocoagulation.
  • the systems, etc. are directed to a local surgical cockpit comprising a local surgical console configured for transmitting surgical movements of an operator operating the local surgical console to a remote surgery site, and comprising at least one virtual console control knob presented virtually to the operator and configured to be manipulated by the operator to generate control signals for a corresponding remote device at a remote operation site.
  • the local surgical cockpit can be part of a system and the system further can comprise the corresponding remote device.
  • the at least one virtual console control knob can be a binary switch configured to provide on/off signals to the corresponding remote device.
  • the at least one virtual console control knob can be a gradual control knob configured to provide gradual control signals to the corresponding remote device.
  • the virtual control knob can be operably connected to one of the three fingers of the haptic device. In some embodiments, the virtual control knob must be virtually gripped by two or more fingers of the haptic device before it may be rotated.
  • the systems, etc. are directed to at least two local surgical cockpits each comprising a surgical console configured for transmitting surgical movements of an operator operating the local surgical console to a distant remote surgery site located outside at least one building containing at least one of the surgical cockpits, wherein each cockpit can comprise a respective first and second set of at least two local robotic input arms configured to provide input to corresponding first and second remote sets of at least two corresponding remote robotic arms each configured to manipulate a remote surgical instrument at a single remote operation site, wherein the respective first and second set of local robotic input arms can be configured to be manipulated by respective first and second surgeons working in concert on the remote surgical site.
  • the local surgical cockpit system can be part of a further system and the further system further can comprise the first and second remote sets of at least two corresponding remote robotic arms.
  • the distant remote surgery site can be located outside any building containing any of the local surgical cockpits.
  • the system can be configured such that operators in different locales can operate simultaneously on a single surgical site; such that operators can relieve each other in a single surgery at a single surgical site; and/or to provide a teaching surgical cockpit and a student surgical cockpit providing haptic feedback to a student operator generated by a teaching operator.
  • the haptic feedback to the student can comprise movements of a remote surgical instrument controlled by the teaching operator or tactile feedback from a surgical site being operated on by the teaching operator.
  • the systems, etc. are directed to a local surgical cockpit comprising a local surgical console configured for transmitting surgical movements of an operator operating the local surgical console to a remote surgery site, and comprising at least four local robotic input arms configured to provide input to a corresponding at least four remote robotic arms each configured to manipulate a remote surgical instrument at a remote operation site, wherein the at least four local robotic input arms can be configured to be manipulated by at least one surgeon operating the local surgical console.
  • the local surgical cockpit can be part of a system and the system further can comprise the four remote robotic arms operably connected to the four local robotic input arms such that the four remote robotic arms precisely respond to movements of the four local robotic input arms.
  • the four remote robotic arms can be held in a sole arm-retention structure, which can be configured to hold the four remote robotic arms such that the arms cannot collide with each other.
  • the cockpit can be part of a system comprising at least two local surgical cockpits each configured for an operator, and wherein the system can be configured such that each operator can simultaneously hold a single remote robotic arm, or such that the operators can switch control of a remote robotic arm between each other.
  • the systems, etc. are directed to a local surgical cockpit comprising a local surgical console configured for transmitting surgical movements of an operator operating the local surgical console to a remote surgery site, and comprising local 3-dimensional audio configured to obtain remote 3-dimensional audio input from a remote operation site and provide corresponding local 3-dimensional audio to an operator operating the console.
  • the local surgical cockpit can be part of a system and the system further can comprise remote 3-dimensional audio sensors operably connected to the local 3-dimensional audio such that the local 3-dimensional audio precisely transmit 3-dimensional audio signals from the remote 3-dimensional audio sensors.
  • the 3-dimensional audio signals can be correlated with tactile feedback to provide correlated response to haptic input devices at the local surgical cockpit.
  • FIG. 1 depicts an isometric view of an exemplary cockpit and console according to various aspects and features discussed herein.
  • FIG. 2 is a view of a fully relaxed body posture as observed in astronauts who are subjected to microgravity.
  • FIG. 3 is a second view of a fully relaxed body posture as observed in astronauts who are subjected to microgravity.
  • FIG. 4 is a third view of a fully relaxed body posture as observed in astronauts who are subjected to microgravity.
  • FIG. 5 depicts a side view of exemplary components of a seat as discussed herein.
  • FIG. 6 depicts a head-borne HMD comprising a headset.
  • FIG. 7 depicts a side view of a further exemplary cockpit and console according to various aspects and features discussed herein.
  • FIG. 8 depicts an isometric view of another exemplary cockpit and console according to various aspects and features discussed herein.
  • FIG. 9 depicts an isometric view of still a further exemplary cockpit and console according to various aspects and features discussed herein comprising an array of eight intuitive haptic foot pedals (four for each foot) that allows the surgeon to control multiple devices and to switch the control between them.
  • FIG. 10 depicts an isometric view of a three fingers haptic hand wherein the middle and index fingers are lumped into a first port, the ring and the fifth finger are lumped into a second port and the thumb is in third port.
  • FIG. 11 shows some of the degrees of freedom obtainable with a three-fingered hand.
  • FIG. 12 depicts a top view of a three fingers haptic hand wherein the index finger is lumped into a first port, and the middle, ring and fifth finger are lumped into a second port, and the thumb is in third port.
  • FIG. 13 depicts an isometric view of an articulated haptic arm coupled to a three fingers haptic hand.
  • FIG. 14 depicts an exemplary conceptual layout of the visual information displayed by the cockpit to the surgeon
  • FIG. 15 depicts an isometric view of four robotics arms that can be teleoperated by one or two surgeons in multiple modes of operation
  • FIG. 16 is a block diagram depicting certain aspects of an exemplary software architecture that can be used with the local surgical cockpits and remote surgical procedures herein.
  • FIG. 1 depicts an exemplary cockpit and console according to various aspects and features discussed herein.
  • Cockpit 2 comprises a surgical console 25 comprising screens, input devices and the like, and a structural frame 4 disposed on a base 6 .
  • the frame 4 provides physical support, directly or indirectly through other components, to the console components of the cockpit 2 such as robotic arms 20 and head mounted display 22 (HMD), and provides adjustable mounting capabilities for every desired element.
  • various console elements discussed herein can be disposed on other support structures instead of the frame 4 , such as nearby walls, desks, tripod stands, etc.
  • the cockpits 2 herein can also comprise two or more different surgical consoles in one cockpit or two or more different surgical consoles in two or more different surgical cockpits that are operably connected to each other either locally or at the remote surgical site (or otherwise as desired). This allows, e.g., surgeons or other operators in completely different locales to operate or otherwise interact simultaneously on a single surgical site.
  • the seat 8 is an adjustable ergonomic seat that positions the body of the surgeon in a desired position such as an optimal, selectable posture to reduce fatigue or other discomfort.
  • the seat 8 allows for positioning and orientation of its components (e.g., headrest 16 , backrest 14 , seat plate 10 , footrest 12 , armrests 18 ) in any possible configuration from sitting fully upright to a completely supine position and to accommodate different body types.
  • positioning elements of the cockpit 2 are operably connected to the independently movable headrest 16 , backrest 14 , seating plate 10 , footrest 12 and armrest 18 to provide at least three axes of retainable positioning movement for each of those independently movable elements relative to each other.
  • all such seating elements can be movable relative to the frame, or one can be securely retained on the frame 2 and the other movable seating elements can be movable.
  • the reference body posture can be the one adopted by the human body in microgravity.
  • the joint 26 between the stationary base 6 and the frame 4 can be actuated to allow titling motion of the frame 4 .
  • the tilting motion accommodates the body position of the surgeon through different angles with respect to gravity.
  • the tilting angle can range from a supine posture where the gravity vector is perpendicular to the spine, to a full upright posture where the gravity vector is parallel to the spine.
  • the backrest 14 of seat 8 can also comprise a variable lumbar support 96 that provides retainable positioning movement for support of the lower back.
  • the frame 4 is articulatable and provides the ability to removably attach and adjust peripheral devices such as monitors 24 , HMDs 22 and forearmrests 18 and input devices 34 and for such peripheral devices to articulate in unison with respect to a stationary base 6 and/or seat 8 .
  • peripheral devices such as monitors 24 , HMDs 22 and forearmrests 18 and input devices 34
  • the location of the display and input device 34 move accordingly so that the relative position of the peripheral devices to the surgeon stays substantially the same.
  • the adjustable elements of the seat allow body posture adjustments automatically or on demand. In the automatic mode, the chair positioning can be under the control of a high-level software module.
  • the seat and other positioning elements can be moved electronically or mechanically via motors, manually, or otherwise as desired.
  • Articulated mechanical linkages and interfaces can be provided in the frame 4 for desired subsystems such as (1) two sets of articulated linkages 28 , 30 for attaching the displays 50 such as an array of screens or monitors (one such monitor 24 is shown in FIG. 1 and FIG. 2 connected to articulated linkage 28 ) and the HMD 22 , which is connected to HMD articulated linkage 30 ; (2) two interfaces 32 to support the arms and the input device 34 ; (3) a footrest support interface 36 to support the foot pedals 38 and the footrest 12 ; and, (4) a seat interface 40 to support the seat 8 . All of these interfaces can be fully adjustable. Additional interfaces include headrest support interface 42 and backrest support interface 44 . Friction based mechanical joints, or other suitable connectors, lock the cockpit 2 in the desired position. Interfaces that change their position frequently, such as interface 28 for the HMD 22 , can be locked in place by electro-mechanical brakes.
  • a surgeon can be required to perform high dexterity manipulation during the course of surgery that may last for several hours.
  • the chair can be designed such that it can be adjusted to emulate the same body posture as observed in astronauts who are subjected to microgravity (see FIGS. 2-4 ). In this body position the muscles reach their rest length and thus reduce potential fatigue.
  • the capability to adjust the body posture can be needed to avoid postural fixity, promote blood circulation, reduce joint pressure and muscle tension, and increase situational awareness
  • arm position affects the ability of the surgeon to control the input device 34 to the surgical robot.
  • Arm manipulability can be a term that can define mathematically how joint angles (shoulder and elbow) are mapped into the hand position. It can be shown mathematically and proved experimentally that in order to maximize the manipulability, the elbow joint angle should be about 90°. Interestingly, this elbow joint angle can be also adopted by the human body in microgravity (see FIGS. 2 - 4 —the elbow angle can be 92°+/ ⁇ 15°). Following this rationale, the arm of the surgeon can be positioned using adjustable armrests 18 with the same angles as indicated in FIGS. 2-4 to maximize the manipulability of the arm.
  • Footrest 12 supports the feet in a similar fashion to FIGS. 2-4 given the same rationale. This can enhance the ergonomic interactions between the surgeon's feet and hands with the controls and peripheral equipment. Further, the surgeon's foot can be fixed in space and ankle movements can be used to activate the pedals surrounding it. Functions such as controlling the cameras can be implemented by linking the ankle flexion/extension to the camera pitch movements and ankle rotation to the camera yaw movements. Haptics can be added to the pedals. For example, force feedback applied through the pedals can be correlated with camera position, or with irrigation or suction pressure.
  • the seat 8 can be adjusted to adapt to different bodies and changing body posture, automatically or on demand. This can avoid postural fixity, promote blood circulation, reduce joint pressure and muscle tension and increase situational awareness.
  • FIG. 1 shows a frame-mounted HMD 22 ;
  • FIG. 6 depicts a head-borne HMD 22 comprising a headset 90 .
  • the embodiment in FIG. 6 comprises a head strap 46 and a vertical strap 48 and 3-D audio input device 80 .
  • the display 50 is disposed in front of the eyes of a user.
  • the display can be divided into two components: (1) the hardware and (2) the information layout. From the hardware perspective, one or more, even several, 2-D and/or 3-D modalities can be used, for example two screens with projected mirrors, a Head Mounted Display with two separate streams of video displayed to each eye or a single 3-D screen, e.g., 120 Hz, with active and synchronized shutter glasses at 60 Hz (Nvidia 3-D Vision).
  • 2-D and/or 3-D modalities can be used, for example two screens with projected mirrors, a Head Mounted Display with two separate streams of video displayed to each eye or a single 3-D screen, e.g., 120 Hz, with active and synchronized shutter glasses at 60 Hz (Nvidia 3-D Vision).
  • Table 1 For analyzing the graphical information available to the surgeon in the OR, an exemplary layout for a display 50 is summarized in Table 1 and depicted in FIG. 14 :
  • Augmented Information (on/off): (e.g., textbook anatomy, model or video Preselected Margins to dissect Blood Pressure clip of an expert performing the procedure, Masks of vital structures Temp remote collaborator) O2 SAT [Bottom Left] CO2 [Bottom Right] Intracranial pressure 2-D display of MRI CT scans (can Preplanned trajectory 2-D display of OR overview be browsed) Tool Type Suction On/Off Bottom Task bar Recording capabilities Time Elapsed time
  • the elements shown can include the remote surgical site 52 , remote operating room 54 , remote target 56 and remote surgical device 82 as well as other desired information. Such information can be presented in segments of a single screen or on multiple screens.
  • FIG. 14 depicts an exemplary conceptual layout of the visual information displayed by the cockpit 2 to the surgeon.
  • This layout translates the verbal specification and description listed in Table 1.
  • High level software of cockpit 2 can manage the content of the display 50 .
  • the central view of the remote surgical site 52 point can be kept clear of overlays such as augmented reality by default if desired, with display of specific components under control of the surgeon.
  • Information will flow to this display 50 from the endoscope/camera(s) targeting the remote surgical site 52 and remote target 56 within such site.
  • the display in FIG. 14 as well as haptic feedback and other information from the remote surgical site and/or operating room are transmitted to multiple local surgical cockpits.
  • the dissection plane can be shown as a deformable blue line; this direction line can be defined during the preoperative stage and can be tracked during surgery by the high level software.
  • Force reflection signatures can be acquired as the position error between the input device 34 and the remote surgical instrument 58 by the high level software module from a low level software module, discussed further below.
  • This information can be presented as colored dots attached to the surgical instruments (as overlays) to indicate safe (green) and unsafe (red) contacts for tissue resection.
  • Patient vital signs can be optionally shown in the central view or docked in the display margins. This peripheral information can be acquired by the high level software module and present visually to the surgeon.
  • Recording capabilities can be embedded into the main view and controlled by the high level software module allowing the surgeon to record the entire operation or individual segments (with chapter markers set by the surgeon if desired) to form a detailed medical record of the procedure as well as broadcast for teaching and remote conferencing.
  • the additional monitors 24 present information acquired by imaging modalities along with an overview of the operating room or other information as desired. The nature of the information can be procedure specific.
  • the display 50 can be a modular display integrated into the surgical cockpit 2 with the following capabilities:
  • Desirable elements in the display design can include (1) reduced cognitive load for the surgeon, (2) support for surgical “flow”, and (3) ultimately, increased patient safety.
  • the displays, and other elements herein, can be used for both normal surgeries and for microsurgeries.
  • the microscope traditionally used for microsurgery may be replaced, such as with a 1-2 mm scope. This change provides a wider field of view.
  • the multiple (e.g., five) monitor layout and the HMD 22 present information in 3-D and provide support for various visualization, communication, and surgical performance functions, including (1) surgical instrument and the surgical scene—sensory information can be displayed for smart tools with embedded sensors; (2) two-handed haptic clinical information e.g. compression tension; (3) “augmented reality” blending graphical images with real-world views and real robot slaves e.g. go/no-go zones; (4) case archiving and videoconferencing for guiding and collaborative purposes.
  • every pieces of information presented on the display 50 can be called up or suppressed by the surgeon. This approach allows a custom display of information that can be dynamically changed during the surgical procedure.
  • Additional display 50 modalities include large field-of-view dome projection displays, which afford significantly larger display “real estate,” and high-resolution autostereoscopic displays.
  • the augmented reality can also provide numerical models to indicate current tissue stresses propagating from instrument manipulations.
  • the interface display, robot control commands, and/or audio can be recorded (with chapter markers set by the surgeon if desired) to form a detailed medical record of the procedure as well as broadcast for teaching and remote conferencing.
  • the cockpit 2 can include at least two functional interfaces for the surgeon's hands (master robotic arm) and the feet (foot pedal array).
  • the interfaces can include two robotic arms 20 and eight foot pedals 28 in two groups or sets of four.
  • the surgeon controls all the specific functions of the surgical robot through these controls.
  • the local controls also transmit force feedback to the surgeon as the remote surgical tools interact with the target tissue.
  • Separate interfaces control peripheral devices such as positioning/zooming the camera, camera angle, camera focus, suctions/irrigation, robot brakes, and cautery including electric coagulation, laser photocoagulation, stapling, etc.
  • the first and second foot pedal sets can each contain four foot pedals 38 .
  • the four pedals in can be arrayed in an arc in front of the user's foot.
  • the four pedals can each be in one of four quadrants of a circle, typically 90° apart. Opposed pairs can be assigned opposed functions at the remote surgical site, such as suction and irrigation.
  • Each of the four pedals can also or alternatively be assigned complementary functions for a remote instrument(s) at the remote surgical site such as the viewing angles, focus, zoom, etc., of an endoscopic or operating room camera robot brakes, cautery such as electric coagulation or laser photocoagulation, etc.
  • the following discussion is directed to an exemplary individual input device 34 , namely a robotic arm 20 .
  • the master input device 34 is a multi degree of freedom (DOF) haptic device including two subsystems: (a) an articulated haptic arm 62 and (b) a three fingers haptic hand 64 .
  • DOF multi degree of freedom
  • Three capabilities that input device 34 typically includes to facilitate the fundamental control of surgical tools by the surgeon through the cockpit 2 comprise: (1) positioning and orientation of the tool tip in space requires 6 parameters—Cartesian position (x,y,z), and angular orientation (x y z ⁇ , ⁇ , ⁇ ); (2) scaling factors introduced such that the motion of the surgeon hands controlling the input device 34 can be scaled down (attenuate) or up (amplify) with respect to the robot; and, (3) indexing (“clutching”), which allows the surgeon to disengage the input device 34 from the robot to reposition his/her arms and engage again.
  • the arm 62 of the input device 34 can include six or seven DOF or more.
  • the arm can be constructed as a cable actuated SCARA-based machine (SCARA—Selective Compliant Articulated Robot Arm), or otherwise as desired (as with all components of the systems discussed herein, unless specifically stated otherwise, the specific materials, manufacturing methods, etc., for the components can be selected to optimize particular features and characteristics.
  • SCARA Selective Compliant Articulated Robot Arm
  • the components can be selected to optimize particular features and characteristics.
  • the components can be made of steel, carbon fiber, etc., so long as the composition is acceptable for the desired purpose).
  • a cable actuated system can be common practice in designing haptic devices. It allows for location of the actuators on a stationary base 6 to transfer torques to each one of the joints through a system of pulleys and cables. This configuration leads to a lightweight, low inertia, low friction, and back drivable haptic device that can reflect back to the user the forces generated when the surgical robot interacts with tissues. Back drivability can be an important characteristic of any force feedback haptic device. It generates a negligible effect of resistance as the operator moves the input device 34 in free space which can be the desired response as the surgical robot does not interact with any tissue.
  • the actuation and position sensors can be supported by a system of amplifiers, along with low level software modules incorporating servo feedback loops for tele-operation and force feedback control algorithms.
  • the SCARA-based mechanism can be a classical robotic arm configuration which includes three consecutive axes with a rotation axis perpendicular to the ground or parallel to the gravitational vector.
  • actuators are still typically incorporated into the tactile input device 34 .
  • Their secondary can be to preserve the registration between the input device 34 and the remote surgical tool by locking the orientation of the tip of the input device 34 (arm) once indexing is taking place. Indexing should typically only allowed for repositioning the end effector of the input device 34 in the Cartesian space (x,y,z) through translation or repositioning. The orientation of the input device 34 during the indexing process must typically be preserved. Reorientation is not typically allowed during the indexing process in order to preserve the registration between the input device 34 of the cockpit 2 and the surgical tool attached to the surgical robot.
  • Force feedback capabilities can be achieved by a cable driven master mechanism with a set of actuators (e.g., brushed DC motors) and position sensors (encoders and potentiometers) attached to its base.
  • Brushless DC motors have a minor advantage compared with brushed motors as far as torque to weight ratio. However the high numbers of electrical wires for motor commutation may cancel out their minor advantage.
  • the actuators can be selected to generate the following peak forces and torques: (1) translational forces 67 N, grasping force 42 N, torques 2.4 Nm.
  • the actuation and position sensors can be supported by a system of amplifiers, along with newly developed low-level software modules incorporating servo feedback loops for teleoperation and force feedback control algorithms.
  • the input device 34 comprises: (1) the direct kinematics of the haptic arm defined by mapping the joint angle to the end effector (hand interface), (2) The Jacobian matrix can be derived by mapping the joint angular velocity to the end effector velocities, (3) the manipulability as a performance measure can be defined, (4) a cost function can be defined taking into account the manipulability measure and the link length of the mechanism.
  • the cost function can be calculated across an entire workspace of different combinations of link lengths, for example ones that have maximal dynamic manipulability with the minimal link lengths within a workspace of 10 ⁇ 10 ⁇ 10 cm.
  • the surgeon's hands interact with the master devices through a three-finger mechanical interface such as the three fingers haptic hand 64 .
  • the middle and index fingers are lumped into the first port 72
  • the ring and the fifth finger are lumped into the second port 74 and the thumb is in third port 76 .
  • the index finger is put into the first port 72
  • the middle, ring and the fifth finger are lumped into the second port 74 and the thumb is in third port 76 .
  • FIG. 11 shows some of the degrees of freedom obtainable with a three-fingered hand.
  • the three fingers interface allows a wide spectrum of control capabilities by the surgeon's hand over a remote surgical instrument or other remote device.
  • the haptic hands herein can switch between various user-finger configurations to maximize dexterity as desired.
  • Each of the tactile devices such as the robotic arm 62 and three fingers haptic hand 64 can comprise 12 DOF or more.
  • Cutaneous information can be provided via the mechano-receptive nerve endings in the glabrous skin of the human hand. It can be primarily a means of relaying information regarding smallscale details in the form of skin stretch, compression and vibration. Kinesthetic sensing encompasses larger scale details, such as basic object shape and mechanical properties, for example, compliance. This can be achieved via feedback from the muscular and skeletal system.
  • 3 out of the 6 DOF that can be sensed for the three fingers mechanism may include force feedback.
  • the remaining 3 non-actuated DOF will can be position information that can be provided by the low frequency motion of the hand and the arm, cutaneous information can be provided to the surgeon regarding tissue texture via high frequency actuators incorporated into the finger pad interface of the 3 fingers hand.
  • Low frequency indicates forces provided in the range of 0-100 Hz; high frequency indicates forces provided in the range of 100 Hz and higher.
  • buttons can be implemented: (1) binary button (On/Off); (b) gradual knob (Volume/Magnitude).
  • each finger has force feedback on 2 out of the 3 DOF allowing flexion/extension movements and feedback.
  • the fingers' adduction/abduction movements can be supported by a passive DOF with no force feedback.
  • the three-finger design is particularly useful for use with (1) virtual knobs and switches 78 (such as shown in FIG. 14 ).
  • virtual knobs and switches 78 such as shown in FIG. 14 .
  • the virtual control knob can be operably connected to one or more of the three fingers of the haptic input device(s).
  • the virtual control knob can also be configured so that it must be virtually gripped by two or more fingers of the haptic device before it may be rotated or otherwise manipulated.
  • FIG. 9 depicts a cockpit 2 comprising an array of 8 intuitive haptic foot pedals (4 for each foot) that allows the surgeon to control multiple devices and to switch the control between them.
  • Two desirable pedal configurations (1) serial arrangement (similar to a car) and (2) spatial arrangement in which 4 pedals surround each foot such that by flexing/extending the ankle and rotating the feet left/right all 4 pedals are accessible.
  • the serial arrangement can be limited to 3 pedals for each foot. Increasing the number of pedals beyond three makes it difficult for the surgeon to locate the pedals while being immersed visually in the surgical site.
  • the spatial arrangement of 4 pedals sounding the feet from 4 different orthogonal directions provides an easier registration between the feet and the pedals and a richer medium as an input device 34 .
  • the feet tactile interface can include a passive gimbal mechanism 84 to support the heel of the foot.
  • the gimbal mechanism 84 allows the surgeon to move freely in any direction while avoiding gamble lock (the rotation in any direction will never exceed 90°).
  • the four pedals can be arranged around the distal end of the feet in four orthogonal planes. Flexing the ankle joint will press the top pedal while extending the joint will press the bottom pedal. Moving the distal part of the foot left and right will press the left and right pedals.
  • a dead zone can be implemented in the design preventing a situation in which the feet activate two opposing functions simultaneously (e.g., suction and irrigation).
  • Each pedal can be controlled by a single DC servo motor.
  • the force displacement characteristics can be defined. Displacement characteristics of each pedal provide the opportunity to change the function of this interface and assign functions to each pedal based on a specific operation (similar to the third finger of the hand interface).
  • the software can allow the surgeon to change the nature of the pedal from an on/off switch to a gradual control switch.
  • the distribution of functions between the hand's third finger and the foot can be assigned as desired.
  • Functions such as controlling the cameras can be implemented by linking the ankle flexion/extension to the camera pitch movements and ankle rotation to the camera yaw movements.
  • Haptics can be added to the pedals. For example, force feedback applied through the pedals can be correlated with irrigation or suction pressure.
  • the cockpits 2 herein can also comprise two or more different surgical consoles in one cockpit or two or more different surgical consoles in two or more different surgical cockpits that are operably connected to each other either locally or at the remote surgical site (or otherwise as desired).
  • This allows, e.g., surgeons in completely different locales to operate simultaneously on a single surgical site.
  • This also allows, e.g., surgeons in the same or different locales to relieve each other in a single surgery at a single surgical site.
  • FIG. 15 depicts an isometric view of a system comprising four remote robotics arms 86 a - 86 d holding remote surgical instruments 88 a - 88 d in a remote operating room, which arms 86 can be teleoperated by one or two (or more) surgeons in multiple modes of operation, such as (1) solo by a single surgeon from a local or a remote location (2) by two surgeons in which one or both are located locally with the robot or in one or two remote sites.
  • the four robotic arms can contain a variety of peripheral devices and/or functions, such as an endoscope 100 comprising an endoscopic camera 102 in remote surgical instrument 88 a coupled with positioning and/or zooming the camera, camera angle, camera focus.
  • remote surgical instrument 88 b can contain a suction device 104 and irrigation device 106
  • remote surgical instrument 88 d comprises a cautery device 108 for, for example, electric coagulation or laser photocoagulation.
  • the robotic arms can also be under the control of robot brakes 110 and can provide a staple applier 114 .
  • the four arms 86 system duplicates two surgeons collaborating and simultaneously interacting with the remote surgical site.
  • the new system provides a new opportunity to explore collaborative surgery which to some extent was not possible before due to the limited number of available remote surgical arms 86 .
  • the control over one or two of the arms can be assigned to an artificial agent (software) and to facilitate new methods of automation.
  • any robotic arm can be assigned at any point by the primary surgeon to the other surgeon(s) or the artificial agent regardless of its location (local or remote).
  • the four remote robotic arms are held in a sole arm-retention structure 116 .
  • the sole arm-retention structure can be configured to hold the four remote robotic arms such that the arms cannot collide with each other.
  • the cockpit can be part of a system comprising at least two local surgical cockpits each configured for an operator, and the system can be configured such that each operator can simultaneously hold a single remote robotic arm, and/or so that the operators can switch control of a remote robotic arm between each other.
  • FIG. 15 also depicts an external, remote operating room camera 98 .
  • the functions of this camera such as position, zooming, angle, focus, etc., and other remote operating room devices and peripherals can also be controlled by the surgeon or other operator in the surgical cockpit 2 .
  • FIG. 6 depicts an exemplary headset 90 comprising a 3-D auditory input device 80 .
  • the Auditory Interface can be any suitable 3-D auditory system such as a system comprising two sonic beam focused speakers.
  • 3-D auditory technologies include Hypersonic Sound (HSS) by American Technology Corp and Audio Spotlight by Holosonics. These technologies allow a focal beam of sound to be transmitted into a specific point in space (surgeon's ears) and avoid sound “pollution” in the operating room (OR). Both can still provide desired sound input from the operating room.
  • HSS Hypersonic Sound
  • 3-D auditory input devices 80 permit local surgical areas/cockpits to become consistent virtual listening areas without the pervasive omnidirectionality of conventional loudspeakers.
  • the high-precision targeting of directional beam of sound significantly minimizes the levels of noise pollution in the local operating rooms while still allowing the surgeon to respond to other sources of auditory inputs from the remote operating room.
  • the 3-D auditory input devices 80 can be used, for example, to convey the following information via either natural or synthetic sound cues: (1) collisions between the surgical tools, (2) contact between the tool tip and some types of tissue, (3) stress levels applied to the tissue, (4) vital signs and emergency limits, (5) local pulse and vascular blood flow to denote vessels that may need to be preserved or ligated.
  • Audio may play an important role in perceiving the important information generated by the surgical site or the OR by the surgeon located in a remote site.
  • FIG. 16 is a block diagram depicting certain aspects of an exemplary software architecture that can be used with the local surgical cockpits 2 and remote surgical procedures herein.
  • the software architecture includes two layers: high level software 92 and low level software 94 .
  • the software comprises multiple interfaces to peripheral hardware and software components such as seat 8 , display 50 and 3-D audio input devices 80 .
  • the software can also be used for the surgical cockpit 2 to communicate with any desired surgical module (surgical robot) that shares a proper communication channel such as a common and universally accepted Transmission Control Protocol/Internet Protocol (TCP/IP).
  • TCP/IP Transmission Control Protocol/Internet Protocol
  • the surgical cockpit can not only be used with nearby remote surgical sites but also with distant remote surgery sites located outside at least one, and possibly any, building containing at least one of the surgical cockpit(s).
  • the high level software 92 can comprise a software layer that smoothly interfaces the teleoperation capability with image-guided software modules residing either at the remote or local surgeon site, and a software module that facilitates collaborative communication among multiple surgical consoles.
  • the high-level software module 92 can merge the information provided by the hand, foot, and peripheral input device 34 . Image data and vital signs are presented continuously and/or through computer generated audio cues.
  • the high level module can be divided into processes, and if desired each process can be dedicated to one of the peripheral elements depicted in FIG. 8 .
  • the low level software 94 can comprise a robust software layer that leverages control techniques to support haptic feedback via Internet connections at local, national, and global scales.
  • the low-level software module can be responsible for the interpretation of the low-level input signals acquired from the hand and feet interfaces. Signals are translated to and from the surgical robot over a network using TCP/IP.
  • Network communications can update the software modules, for example the high level module that manages visual and audio display.
  • the low level software can be primarily a real-time segment of the software running on a RT Linux operating system.
  • the high level module can be non real-time software.
  • the interaction between the high level and low level modules can be primarily unidirectional, in which position and orientation of the surgical tool location can be sent from the low level to the high level, with the only exception in which information can be sent back to the low level module being when a tool reaches a position that exceeds a predefined safe zone.
  • the present invention provides an improved surgical device that permits surgical instruments to reach remote portions of the body with reduced trauma.
  • the device sheath may be steered to a surgical site around sensitive or critical tissue.
  • the surgical tool components may be removed for replacement or cleaning without the device having to be straightened or removed from the body.
  • the tool deflection assemblies and methodology renders precise control of the surgical tool components in all required degrees and directions of movement.
  • the present invention is thus well suited for use in many different applications, including robotic surgical systems.

Abstract

Local surgical cockpits comprising local surgical consoles that can communicate with any desired remote surgical module (surgical robot), for example via a shared Transmission Control Protocol/Internet Protocol (TCP/IP) or other unified open source communication protocol or other suitable communication system. The systems and methods, etc., herein can also comprise a modular approach wherein multiple surgical consoles can network supporting collaborative surgery regardless of the physical location of the surgeons relative to each other and/or relative to the surgical site. Thus, for example, an operator operating a local surgical cockpit can teleoperate using a remote surgical module on a patient in the same room as the surgeon, or surgeons located in multiple safe locations can telemanipulate remote multiple surgical robots on a patient in or close to a war zone.

Description

    PRIORITY CLAIM
  • The present application claims the benefit of copending U.S. Provisional Application Ser. No. 61/315,018, filed Mar. 18, 2010, which application is incorporated herein by reference in its entirety.
  • BACKGROUND OF THE INVENTION
  • Despite advances in personnel, technology, and force protection, war fighters remain vulnerable to blast wounds, burns, and multiple penetrating injuries not usually encountered in civilian settings. There is a fundamental need to deploy skilled personnel equipped with advanced technology to provide medical and surgical attention as close as possible to the point of injury for a soldier in the battlefield and civilians in a remote location or during natural or manmade disasters. Although war surgery is aimed to treat combat casualties at far forward locations and under austere conditions continues to save lives, the nature of battlefield injuries shortens the “golden hour” in which highly skilled medical attention is required to stabilize the soldier.
  • One of the major problems in the military and its integrated health services support system to triage, treat, evacuate, and return soldiers to duty is the occasional mismatch between the type of injury and the type of surgeon available to treat it. For example, a heart surgeon may be required to perform a craniotomy. Telemedicine in general and telerobotic surgery in particular are a means to mediate the narrow spectrum of available front line surgical expertise.
  • There has gone unmet a need for improved methods that provide one or more of the needs indicated above, for example robust telerobotic capabilities so an expert surgeon could perform critical steps of an operation from the continental U.S. or other desired location while the soldier or other patient is located anywhere around the globe.
  • The present systems and methods, etc., provide these and/or other advantages. Present systems and methods, etc., extend the spectrum of surgical expertise to a level appropriate for the type and nature of battlefield wounds. The present systems and methods, etc., also extend to any desired area of surgery including those well beyond the military arena.
  • SUMMARY
  • The present systems and methods, etc., comprise local surgical cockpits comprising local surgical consoles that can communicate with any desired remote surgical module (surgical robot), for example via a shared Transmission Control Protocol/Internet Protocol (TCP/IP) or other unified open source communication protocol or other suitable communication system. The systems and methods, etc., herein can also comprise a modular approach wherein multiple surgical consoles can network supporting collaborative surgery regardless of the physical location of the surgeons or other operators relative to each other and/or relative to the surgical site. Thus, for example, a surgeon operating a local surgical cockpit can teleoperate using a remote surgical module on a patient in the same room as the surgeon, or surgeons located in multiple safe locations can telemanipulate remote multiple surgical robots on a patient in or close to a war zone or any remote location.
  • One aspect of the systems, methods, etc., herein is to provide a multisensory, multi-modal surgical workstation (surgical cockpit). This workstation can provide a command post allowing the surgeon(s) to visually immerse themselves into the remote surgical space. It provides peripheral information such as vital signs, as well as visual feeds from the operating room (OR) surrounding the actual surgical site on or in the patient. In this way, the surgical console provides high situational awareness as well as the capability to dynamically interact with the other functions of the OR. This is accomplished by multiple streams of visual, audio, and kinesthetic inputs. Special attention can also paid to avoiding information overload of the surgeon.
  • In a further aspect, the present methods, devices, systems, etc., are related to a local surgical cockpit comprising a base, a frame disposed on the base, a seat for an operator disposed on the frame, and a remote surgical console configured such that the operator can operate the console for remote surgery while in the seat, wherein the seat can be ergonomic and can be operably connected to the frame such that the seat can be retainably tilted from a substantially upright position to a substantially supine position.
  • In certain embodiments, the seat can comprise an independently movable headrest, backrest, seating plate and footrest, the seat further comprising positioning elements operably connected to the independently movable headrest, backrest, seating plate and footrest and providing at least three axes of retainable positioning movement for each of the independently movable headrest, backrest, seating plate and footrest. The seat can comprise a lumbar support comprising retainable positioning movement for support of the lower back. The reference body posture of the seat can correspond to a human body posture that can be fully relaxed in micro gravity.
  • The cockpit further can comprise at least one peripheral device operably connected to move with the seat when the seat is moved so that the location of the peripheral device relative to the operator in the seat is substantially unchanged. The peripheral device can be at least one of a monitor facing an operator in the seat and operably linked to display a remote surgical site, a heads-up display disposed in front of the local surgeon's eyes, and an input device disposed at a hand of the operator and operably linked to provide input to a corresponding device located at the remote surgical site.
  • Another aspect comprises a local surgical cockpit comprising a local surgical console configured for transmitting surgical movements of local surgeon operating the local surgical console to a remote surgery site, and a head-mounted display disposed in front of the local surgeon's eyes in surgical position in the cockpit to operate the console for surgery, wherein the head-mounted display can be configured to depict at least images of a remote surgical site under remote operation by the operator. The local surgical cockpit can be part of a system and the system further can comprise remote image sensors operably connected to the head-mounted display to transmit the image of the remote surgical site. The head-mounted display can extend to the local surgeon's eyes from an articulated boom or other retention structure disposed in front of the local surgeon's eyes, which retention structure can be actuated by at least one hand control located on the cockpit, or by voice control or otherwise as desired. The head-mounted display can be disposed on a head-mounted frame configured to rest on an operator's head and to maintain the images in front of the local surgeon's eyes when the operator's head moves. The head-mounted display can comprise two separate streams of video displayed to each eye of the local surgeon's eyes, each stream comprising corresponding right and left eye views of a remote surgical site to provide a 3-D image of the site.
  • The cockpit further can comprise at least one monitor operably held to the base of the cockpit, and the head-mounted display can comprise two separate streams of video displayed to each eye of the local surgeon's eyes, each stream comprising corresponding right and left eye views of a remote surgical site to provide a 3-D image of the site. The monitor(s) can also be 3-D.
  • In another aspect, the systems, etc., are directed to a local surgical cockpit comprising a local surgical console configured for transmitting surgical movements of an operator operating the local surgical console to a remote surgery site, and comprising at least one image display device configured to depict at least one image of the remote surgical site, the display device further depicting augmented reality for the operator comprising augmented information shown on the display and superimposed over the image of the remote surgical site.
  • The local surgical cockpit can be part of a system and the system further can comprise remote image sensors operably connected to the head-mounted display to transmit the image of the remote surgical site. The augmented information can comprise at least one of preselected margins to dissect during the surgery and a mask of vital structures in the remote surgical site overlaid over the images of the remote surgical site. The display device can further display further augmented information either to a side of or superimposed over the image of the remote surgical site and the further augmented information can comprise at least one of blood pressure, temperature, O2 level, CO2 level, intracranial pressure, a preplanned trajectory for a surgical tool, tool type, suction on/off, a bottom task bar, recording capabilities, current time, and elapsed time. The image of the remote surgical site and the augmented information can comprise blending graphical images with real-world views of the remote surgical sit, and can be provided by at least one of an endoscopic camera, a remote surgical site camera, or a camera showing an operating room.
  • In still another aspect, the systems, etc., are directed to a local surgical cockpit comprising a local surgical console configured for transmitting surgical movements of an operator operating the local surgical console to a remote surgery site, and comprising a local surgical instrument comprising local input surgical fingers configured to provide input to corresponding remote surgical fingers configured to manipulate a remote surgical instrument at a remote operation site, wherein the local fingers can be high frequency haptic fingers configured to provide tactile feedback to the operator based on acceleration of the remote surgical instrument manipulated by the remote surgical fingers.
  • The local surgical cockpit can be part of a system and the system further can comprise the remote surgical fingers, and wherein the remote surgical fingers can be haptic fingers configured to provide tactile feedback to the operator based on acceleration of the remote surgical instrument manipulated by the remote surgical fingers.
  • The local surgical cockpits can be configured such that operators in different locales can operate simultaneously on a single surgical site; such that operators can relieve each other in a single surgery at a single surgical site; or to provide a teaching surgical cockpit and a student surgical cockpit providing haptic feedback to a student operator generated by a teaching operator. The haptic feedback to the student can comprise movements of a remote surgical instrument controlled by the teaching operator or tactile feedback from a surgical site being operated on by the teaching operator.
  • In a further aspect, the systems, etc., are directed to a local surgical cockpit comprising a local surgical console configured for transmitting surgical movements of an operator operating the local surgical console to a remote surgery site, and comprising at least seven degrees of freedom for a local surgical instrument manipulated by a robotic arm manipulated by the operator, wherein the console can be configured such that the seven degrees of freedom can be transmissible to a remote surgical instrument located at a remote surgical site and manipulated by the operator operating the console.
  • The local surgical cockpit can be part of a system and the system further can comprise the remote surgical instrument operably connected to the local surgical instrument such that the remote surgical instrument precisely responds in at least seven corresponding degrees of freedom to movements of the local surgical instrument. The degrees of freedom can comprise at least nine degrees of freedom for the local surgical instrument manipulated by the operator and a corresponding nine degrees of freedom for the remote surgical instrument. The degrees of freedom can comprise at least twelve degrees of freedom for the local surgical instrument manipulated by the operator and a corresponding twelve degrees of freedom for the remote surgical instrument, wherein the local robotic arm can comprise a shoulder joint, an elbow joint, a wrist joint and the three fingers, each comprising at least the following degrees of freedom: shoulder can comprise 2 degrees of freedom; elbow can comprise 1 degree of freedom; wrist can comprise 3 degrees of freedom; the three fingers can comprise 2 degrees of freedom each.
  • The local surgical instrument can comprise at least three input fingers configured to provide input to a corresponding at least three remote surgical fingers configured to manipulate a remote surgical instrument at a remote operation site, wherein the at least three input fingers can be configured to be manipulated by a single hand of an operator operating the local surgical instrument, and wherein the at least seven degrees of freedom can comprise at least two degrees of freedom for two of the three remote surgical fingers and at least three degrees of freedom for a third of the three remote surgical fingers, or the degrees of freedom can comprise at least nine degrees of freedom comprising at least three degrees of freedom for each of the three remote surgical fingers.
  • The local robotic arm can comprise a shoulder joint, an elbow joint, a wrist joint and the three fingers, each comprising at least the following degrees of freedom: shoulder can comprise 2 degrees of freedom; elbow can comprise 1 degree of freedom; wrist can comprise 3 degrees of freedom; the three fingers can comprise 2 degrees of freedom each. Or the three fingers can comprise 3 degrees of freedom each.
  • The degrees of freedom provide for variable desired positioning and orientation of a tip of the remote surgical instrument in space in 6 parameters including Cartesian position (x,y,z), and angular orientation (x y z θ, θ, θ). Control of the remote surgical instrument further can comprise scaling factors configured such that motion input by the operator can be attenuated or amplified with respect to the remote surgical instrument. Control further can comprise indexing configured to allow the operator to disengage the input device from the remote surgical instrument to reposition his/her arms and engage again.
  • In still yet another aspect, the systems, etc., are directed to a local surgical cockpit comprising a local surgical console configured for transmitting surgical movements of an operator operating the local surgical console to a remote surgery site, and comprising a local surgical instrument comprising at least three input fingers configured to provide input to a corresponding at least three remote surgical fingers configured to manipulate a remote surgical instrument at a remote operation site. The at least three input fingers can be configured to be manipulated by a single hand of an operator operating the local surgical instrument.
  • The local surgical cockpit can be part of a system and the system further can comprise the three remote surgical fingers operably connected to the three input fingers such that the three remote surgical fingers precisely respond to movements of the three input fingers.
  • The at least three input fingers can be configured to correspond respectively to a) an operator's thumb, b) an operator's index and middle fingers, and c) an operator's ring and little fingers; or to a) an operator's thumb, b) an operator's index finger, and c) an operator's middle, ring and little fingers. The at least three input fingers can be haptic fingers configured to provide tactile feedback to the operator based on acceleration of a remote surgical instrument manipulated by the remote surgical fingers. The three input fingers can be operably connected so that two fingers control remote surgical fingers and the remaining third finger controls an external device, which can be a one or more of an electrocautery device, a laser photocoagulator, a staple applier or other device as desired. The external device can also be an optical aspect of the camera system such as focus, zoom, rotation, or field-of-view.
  • In another aspect, the systems, etc., are directed to a local surgical cockpit comprising a local surgical console configured for transmitting surgical movements of an operator operating the local surgical console to a remote surgery site, and comprising at least one haptic foot pedal configured to be operably connected to at least one remote device at a remote operation site, wherein the at least one haptic foot pedal can be configured to be manipulated by a foot of the operator operating the local surgical console to cause a movement or control change in the remote device.
  • The local surgical cockpit can be part of a system and the system further can comprise the at least one remote device operably connected to the at least one haptic foot pedal. The local surgical cockpit further can comprise at least two haptic foot pedals configured to be operably connected to the at least one remote device at the remote operation site. The local surgical cockpit further can comprise at least eight haptic foot pedals configured to be operably connected to at least two remote devices at the remote operation site, the at least eight haptic foot pedals divided to provide at least a first foot pedal set and second foot pedal set, wherein a first foot pedal set and second foot pedal set can be each disposed to be manipulated by a right foot of the operator and by a left foot of the operator, respectively.
  • If desired, for each of the first foot pedal set and second foot pedal set, the sets can each contain four pedals with each of the four pedals in one of four quadrants of a circle. The pedals can also be set in an array. Opposed or otherwise set off pairs of pedals can be assigned opposed functions at the remote surgical site. The opposed functions can be suction and irrigation. The four pedals can also be assigned complementary functions for a remote instrument at the remote surgical site. The four pedals can control the viewing angles of an endoscopic camera. The local surgical cockpit further can comprise a dead zone that prevents two opposing functions being implemented simultaneously.
  • The at least one haptic foot pedal can also control at least one of camera angle, camera zoom, camera focus, irrigation, suction, robot brakes, electric coagulation, laser photocoagulation.
  • In a further aspect, the systems, etc., are directed to a local surgical cockpit comprising a local surgical console configured for transmitting surgical movements of an operator operating the local surgical console to a remote surgery site, and comprising at least one virtual console control knob presented virtually to the operator and configured to be manipulated by the operator to generate control signals for a corresponding remote device at a remote operation site.
  • The local surgical cockpit can be part of a system and the system further can comprise the corresponding remote device. The at least one virtual console control knob can be a binary switch configured to provide on/off signals to the corresponding remote device. The at least one virtual console control knob can be a gradual control knob configured to provide gradual control signals to the corresponding remote device. The virtual control knob can be operably connected to one of the three fingers of the haptic device. In some embodiments, the virtual control knob must be virtually gripped by two or more fingers of the haptic device before it may be rotated.
  • In still yet a further aspect, the systems, etc., are directed to at least two local surgical cockpits each comprising a surgical console configured for transmitting surgical movements of an operator operating the local surgical console to a distant remote surgery site located outside at least one building containing at least one of the surgical cockpits, wherein each cockpit can comprise a respective first and second set of at least two local robotic input arms configured to provide input to corresponding first and second remote sets of at least two corresponding remote robotic arms each configured to manipulate a remote surgical instrument at a single remote operation site, wherein the respective first and second set of local robotic input arms can be configured to be manipulated by respective first and second surgeons working in concert on the remote surgical site.
  • The local surgical cockpit system can be part of a further system and the further system further can comprise the first and second remote sets of at least two corresponding remote robotic arms. The distant remote surgery site can be located outside any building containing any of the local surgical cockpits. The system can be configured such that operators in different locales can operate simultaneously on a single surgical site; such that operators can relieve each other in a single surgery at a single surgical site; and/or to provide a teaching surgical cockpit and a student surgical cockpit providing haptic feedback to a student operator generated by a teaching operator. The haptic feedback to the student can comprise movements of a remote surgical instrument controlled by the teaching operator or tactile feedback from a surgical site being operated on by the teaching operator.
  • In still yet a further aspect, the systems, etc., are directed to a local surgical cockpit comprising a local surgical console configured for transmitting surgical movements of an operator operating the local surgical console to a remote surgery site, and comprising at least four local robotic input arms configured to provide input to a corresponding at least four remote robotic arms each configured to manipulate a remote surgical instrument at a remote operation site, wherein the at least four local robotic input arms can be configured to be manipulated by at least one surgeon operating the local surgical console.
  • The local surgical cockpit can be part of a system and the system further can comprise the four remote robotic arms operably connected to the four local robotic input arms such that the four remote robotic arms precisely respond to movements of the four local robotic input arms. The four remote robotic arms can be held in a sole arm-retention structure, which can be configured to hold the four remote robotic arms such that the arms cannot collide with each other. The cockpit can be part of a system comprising at least two local surgical cockpits each configured for an operator, and wherein the system can be configured such that each operator can simultaneously hold a single remote robotic arm, or such that the operators can switch control of a remote robotic arm between each other.
  • In another aspect, the systems, etc., are directed to a local surgical cockpit comprising a local surgical console configured for transmitting surgical movements of an operator operating the local surgical console to a remote surgery site, and comprising local 3-dimensional audio configured to obtain remote 3-dimensional audio input from a remote operation site and provide corresponding local 3-dimensional audio to an operator operating the console.
  • The local surgical cockpit can be part of a system and the system further can comprise remote 3-dimensional audio sensors operably connected to the local 3-dimensional audio such that the local 3-dimensional audio precisely transmit 3-dimensional audio signals from the remote 3-dimensional audio sensors. The 3-dimensional audio signals can be correlated with tactile feedback to provide correlated response to haptic input devices at the local surgical cockpit.
  • These and other aspects, features and embodiments are set forth within this application, including the following Detailed Description and attached drawings. Unless expressly stated otherwise, all embodiments, aspects, features, etc., can be mixed and matched, combined and permuted in any desired manner.
  • BRIEF DESCRIPTION OF THE DRAWINGS
  • FIG. 1 depicts an isometric view of an exemplary cockpit and console according to various aspects and features discussed herein.
  • FIG. 2 is a view of a fully relaxed body posture as observed in astronauts who are subjected to microgravity.
  • FIG. 3 is a second view of a fully relaxed body posture as observed in astronauts who are subjected to microgravity.
  • FIG. 4 is a third view of a fully relaxed body posture as observed in astronauts who are subjected to microgravity.
  • FIG. 5 depicts a side view of exemplary components of a seat as discussed herein.
  • FIG. 6 depicts a head-borne HMD comprising a headset.
  • FIG. 7 depicts a side view of a further exemplary cockpit and console according to various aspects and features discussed herein.
  • FIG. 8 depicts an isometric view of another exemplary cockpit and console according to various aspects and features discussed herein.
  • FIG. 9 depicts an isometric view of still a further exemplary cockpit and console according to various aspects and features discussed herein comprising an array of eight intuitive haptic foot pedals (four for each foot) that allows the surgeon to control multiple devices and to switch the control between them.
  • FIG. 10 depicts an isometric view of a three fingers haptic hand wherein the middle and index fingers are lumped into a first port, the ring and the fifth finger are lumped into a second port and the thumb is in third port.
  • FIG. 11 shows some of the degrees of freedom obtainable with a three-fingered hand.
  • FIG. 12 depicts a top view of a three fingers haptic hand wherein the index finger is lumped into a first port, and the middle, ring and fifth finger are lumped into a second port, and the thumb is in third port.
  • FIG. 13 depicts an isometric view of an articulated haptic arm coupled to a three fingers haptic hand.
  • FIG. 14 depicts an exemplary conceptual layout of the visual information displayed by the cockpit to the surgeon
  • FIG. 15 depicts an isometric view of four robotics arms that can be teleoperated by one or two surgeons in multiple modes of operation
  • FIG. 16 is a block diagram depicting certain aspects of an exemplary software architecture that can be used with the local surgical cockpits and remote surgical procedures herein.
  • DETAILED DESCRIPTION
  • FIG. 1 depicts an exemplary cockpit and console according to various aspects and features discussed herein. Cockpit 2 comprises a surgical console 25 comprising screens, input devices and the like, and a structural frame 4 disposed on a base 6. The frame 4 provides physical support, directly or indirectly through other components, to the console components of the cockpit 2 such as robotic arms 20 and head mounted display 22 (HMD), and provides adjustable mounting capabilities for every desired element. In other embodiments, various console elements discussed herein can be disposed on other support structures instead of the frame 4, such as nearby walls, desks, tripod stands, etc.
  • The cockpits 2 herein can also comprise two or more different surgical consoles in one cockpit or two or more different surgical consoles in two or more different surgical cockpits that are operably connected to each other either locally or at the remote surgical site (or otherwise as desired). This allows, e.g., surgeons or other operators in completely different locales to operate or otherwise interact simultaneously on a single surgical site.
  • The seat 8 is an adjustable ergonomic seat that positions the body of the surgeon in a desired position such as an optimal, selectable posture to reduce fatigue or other discomfort. The seat 8 allows for positioning and orientation of its components (e.g., headrest 16, backrest 14, seat plate 10, footrest 12, armrests 18) in any possible configuration from sitting fully upright to a completely supine position and to accommodate different body types. Thus, positioning elements of the cockpit 2 are operably connected to the independently movable headrest 16, backrest 14, seating plate 10, footrest 12 and armrest 18 to provide at least three axes of retainable positioning movement for each of those independently movable elements relative to each other. For example, all such seating elements can be movable relative to the frame, or one can be securely retained on the frame 2 and the other movable seating elements can be movable. The reference body posture can be the one adopted by the human body in microgravity.
  • As shown in FIG. 5 as well as FIG. 1 and other figures depicting the full cockpit, the joint 26 between the stationary base 6 and the frame 4 can be actuated to allow titling motion of the frame 4. The tilting motion accommodates the body position of the surgeon through different angles with respect to gravity. The tilting angle can range from a supine posture where the gravity vector is perpendicular to the spine, to a full upright posture where the gravity vector is parallel to the spine. During the tilting of the entire frame 4 as a whole, the relative position and orientation of the surgeon's body segments as well as the relative position and orientation with respect to the peripheral elements can be maintained. The backrest 14 of seat 8 can also comprise a variable lumbar support 96 that provides retainable positioning movement for support of the lower back.
  • As shown in this embodiment, the frame 4 is articulatable and provides the ability to removably attach and adjust peripheral devices such as monitors 24, HMDs 22 and forearmrests 18 and input devices 34 and for such peripheral devices to articulate in unison with respect to a stationary base 6 and/or seat 8. For example, when the seat position is modified, the location of the display and input device 34 move accordingly so that the relative position of the peripheral devices to the surgeon stays substantially the same. The adjustable elements of the seat allow body posture adjustments automatically or on demand. In the automatic mode, the chair positioning can be under the control of a high-level software module. The seat and other positioning elements can be moved electronically or mechanically via motors, manually, or otherwise as desired.
  • Articulated mechanical linkages and interfaces can be provided in the frame 4 for desired subsystems such as (1) two sets of articulated linkages 28, 30 for attaching the displays 50 such as an array of screens or monitors (one such monitor 24 is shown in FIG. 1 and FIG. 2 connected to articulated linkage 28) and the HMD 22, which is connected to HMD articulated linkage 30; (2) two interfaces 32 to support the arms and the input device 34; (3) a footrest support interface 36 to support the foot pedals 38 and the footrest 12; and, (4) a seat interface 40 to support the seat 8. All of these interfaces can be fully adjustable. Additional interfaces include headrest support interface 42 and backrest support interface 44. Friction based mechanical joints, or other suitable connectors, lock the cockpit 2 in the desired position. Interfaces that change their position frequently, such as interface 28 for the HMD 22, can be locked in place by electro-mechanical brakes.
  • A surgeon can be required to perform high dexterity manipulation during the course of surgery that may last for several hours. The chair can be designed such that it can be adjusted to emulate the same body posture as observed in astronauts who are subjected to microgravity (see FIGS. 2-4). In this body position the muscles reach their rest length and thus reduce potential fatigue. Moreover, the capability to adjust the body posture can be needed to avoid postural fixity, promote blood circulation, reduce joint pressure and muscle tension, and increase situational awareness
  • In addition to the body posture, arm position affects the ability of the surgeon to control the input device 34 to the surgical robot. Arm manipulability can be a term that can define mathematically how joint angles (shoulder and elbow) are mapped into the hand position. It can be shown mathematically and proved experimentally that in order to maximize the manipulability, the elbow joint angle should be about 90°. Interestingly, this elbow joint angle can be also adopted by the human body in microgravity (see FIGS. 2-4—the elbow angle can be 92°+/−15°). Following this rationale, the arm of the surgeon can be positioned using adjustable armrests 18 with the same angles as indicated in FIGS. 2-4 to maximize the manipulability of the arm.
  • Footrest 12 supports the feet in a similar fashion to FIGS. 2-4 given the same rationale. This can enhance the ergonomic interactions between the surgeon's feet and hands with the controls and peripheral equipment. Further, the surgeon's foot can be fixed in space and ankle movements can be used to activate the pedals surrounding it. Functions such as controlling the cameras can be implemented by linking the ankle flexion/extension to the camera pitch movements and ankle rotation to the camera yaw movements. Haptics can be added to the pedals. For example, force feedback applied through the pedals can be correlated with camera position, or with irrigation or suction pressure.
  • Thus, the seat 8 can be adjusted to adapt to different bodies and changing body posture, automatically or on demand. This can avoid postural fixity, promote blood circulation, reduce joint pressure and muscle tension and increase situational awareness.
  • FIG. 1 shows a frame-mounted HMD 22; FIG. 6 depicts a head-borne HMD 22 comprising a headset 90. The embodiment in FIG. 6 comprises a head strap 46 and a vertical strap 48 and 3-D audio input device 80. The display 50 is disposed in front of the eyes of a user.
  • The display (HMD or otherwise such as a frame-borne monitor) can be divided into two components: (1) the hardware and (2) the information layout. From the hardware perspective, one or more, even several, 2-D and/or 3-D modalities can be used, for example two screens with projected mirrors, a Head Mounted Display with two separate streams of video displayed to each eye or a single 3-D screen, e.g., 120 Hz, with active and synchronized shutter glasses at 60 Hz (Nvidia 3-D Vision). For analyzing the graphical information available to the surgeon in the OR, an exemplary layout for a display 50 is summarized in Table 1 and depicted in FIG. 14:
  • [Top Left] [Main Display] [Top Right]
    3-D remonstration of the target 3-D display of Surgical Site 2-D display of Mentoring Information
    Augmented Information: Augmented Information (on/off): (e.g., textbook anatomy, model or video
     Preselected Margins to dissect  Blood Pressure clip of an expert performing the procedure,
     Masks of vital structures  Temp remote collaborator)
     O2 SAT
    [Bottom Left]  CO2 [Bottom Right]
     Intracranial pressure
    2-D display of MRI CT scans (can Preplanned trajectory 2-D display of OR overview
    be browsed)  Tool Type
     Suction On/Off
    Bottom Task bar
     Recording capabilities
     Time
     Elapsed time
  • As can be seen in the Table and FIG. 14, the elements shown can include the remote surgical site 52, remote operating room 54, remote target 56 and remote surgical device 82 as well as other desired information. Such information can be presented in segments of a single screen or on multiple screens.
  • More specifically, FIG. 14 depicts an exemplary conceptual layout of the visual information displayed by the cockpit 2 to the surgeon. This layout translates the verbal specification and description listed in Table 1. High level software of cockpit 2 can manage the content of the display 50. The central view of the remote surgical site 52 point can be kept clear of overlays such as augmented reality by default if desired, with display of specific components under control of the surgeon. Information will flow to this display 50 from the endoscope/camera(s) targeting the remote surgical site 52 and remote target 56 within such site. In some embodiments, the display in FIG. 14 as well as haptic feedback and other information from the remote surgical site and/or operating room are transmitted to multiple local surgical cockpits. This allows, e.g., surgeons in completely different locales to operate simultaneously on a single surgical site with the same visual display, as depicted in FIG. 14, or with visual displays containing identical core information as well as additional custom information as desired by the surgeon. This also allows, e.g., surgeons in the same or different locales to relieve each other during a single surgery and for training surgeons with common tactile and visual feedback through the visual displays and haptic input devices, etc.
  • In this example the dissection plane can be shown as a deformable blue line; this direction line can be defined during the preoperative stage and can be tracked during surgery by the high level software. Force reflection signatures can be acquired as the position error between the input device 34 and the remote surgical instrument 58 by the high level software module from a low level software module, discussed further below. This information can be presented as colored dots attached to the surgical instruments (as overlays) to indicate safe (green) and unsafe (red) contacts for tissue resection. Patient vital signs can be optionally shown in the central view or docked in the display margins. This peripheral information can be acquired by the high level software module and present visually to the surgeon. Recording capabilities can be embedded into the main view and controlled by the high level software module allowing the surgeon to record the entire operation or individual segments (with chapter markers set by the surgeon if desired) to form a detailed medical record of the procedure as well as broadcast for teaching and remote conferencing. The additional monitors 24 present information acquired by imaging modalities along with an overview of the operating room or other information as desired. The nature of the information can be procedure specific.
  • Returning to FIG. 1 and also referring to FIGS. 7-9, which depict various alternative cockpit embodiments and arrangements, the display 50 can be a modular display integrated into the surgical cockpit 2 with the following capabilities:
      • 2-D and 3-D high definition visual displays (e.g., 1080 progressive lines) arranged in an array of 2-5 monitors 24. FIG. 9 shows an example with multiple monitors 24.
      • Head mounted display 22. The HMD can include two miniature screens located 2 cm away from each eye and fed by two separate streams of video signals to generate a stereoscopic image of the surgical scene. The HMD 22 can be mounted on an articulated boom 112 as in FIGS. 1 and 7-9, and can provide a redundant display to the flat panel array. In some embodiments, the HMD 22 displays only the remote surgical site 52, for situations where the surgeons wish to focus on the surgical site exclusively. The HMD can be mounted on any other suitable system such as a headset 90 as in FIG. 6 or other mechanisms such as several cables or parallel arms.
      • Synthesized display of multiple video sources fed from the endoscopic camera or surgical site camera, a camera of the OR, and along with applications presenting imaging information.
  • Desirable elements in the display design can include (1) reduced cognitive load for the surgeon, (2) support for surgical “flow”, and (3) ultimately, increased patient safety. The displays, and other elements herein, can be used for both normal surgeries and for microsurgeries. Moreover, the microscope traditionally used for microsurgery may be replaced, such as with a 1-2 mm scope. This change provides a wider field of view.
  • In certain embodiments, the multiple (e.g., five) monitor layout and the HMD 22 present information in 3-D and provide support for various visualization, communication, and surgical performance functions, including (1) surgical instrument and the surgical scene—sensory information can be displayed for smart tools with embedded sensors; (2) two-handed haptic clinical information e.g. compression tension; (3) “augmented reality” blending graphical images with real-world views and real robot slaves e.g. go/no-go zones; (4) case archiving and videoconferencing for guiding and collaborative purposes. In order to provide a clean and informative interface while managing the cognitive load of the surgeon, every pieces of information presented on the display 50 can be called up or suppressed by the surgeon. This approach allows a custom display of information that can be dynamically changed during the surgical procedure.
  • Additional display 50 modalities include large field-of-view dome projection displays, which afford significantly larger display “real estate,” and high-resolution autostereoscopic displays.
  • The augmented reality can also provide numerical models to indicate current tissue stresses propagating from instrument manipulations. The interface display, robot control commands, and/or audio can be recorded (with chapter markers set by the surgeon if desired) to form a detailed medical record of the procedure as well as broadcast for teaching and remote conferencing.
  • For example as shown in FIG. 9, the cockpit 2 can include at least two functional interfaces for the surgeon's hands (master robotic arm) and the feet (foot pedal array). As shown in FIG. 9 and other figures, the interfaces can include two robotic arms 20 and eight foot pedals 28 in two groups or sets of four. The surgeon controls all the specific functions of the surgical robot through these controls. The local controls also transmit force feedback to the surgeon as the remote surgical tools interact with the target tissue. Separate interfaces control peripheral devices such as positioning/zooming the camera, camera angle, camera focus, suctions/irrigation, robot brakes, and cautery including electric coagulation, laser photocoagulation, stapling, etc.
  • In some embodiments, as shown in FIG. 9, the first and second foot pedal sets can each contain four foot pedals 38. In one arrangement, as shown in front of the left foot of the seated figure in FIG. 9, the four pedals in can be arrayed in an arc in front of the user's foot. In another arrangement, as shown in front of the right foot of the seated figure in FIG. 9, the four pedals can each be in one of four quadrants of a circle, typically 90° apart. Opposed pairs can be assigned opposed functions at the remote surgical site, such as suction and irrigation. Each of the four pedals can also or alternatively be assigned complementary functions for a remote instrument(s) at the remote surgical site such as the viewing angles, focus, zoom, etc., of an endoscopic or operating room camera robot brakes, cautery such as electric coagulation or laser photocoagulation, etc.
  • The following discussion is directed to an exemplary individual input device 34, namely a robotic arm 20.
  • Turning to FIGS. 10-13, the master input device 34 is a multi degree of freedom (DOF) haptic device including two subsystems: (a) an articulated haptic arm 62 and (b) a three fingers haptic hand 64. Three capabilities that input device 34 typically includes to facilitate the fundamental control of surgical tools by the surgeon through the cockpit 2 comprise: (1) positioning and orientation of the tool tip in space requires 6 parameters—Cartesian position (x,y,z), and angular orientation (x y z θ, θ, θ); (2) scaling factors introduced such that the motion of the surgeon hands controlling the input device 34 can be scaled down (attenuate) or up (amplify) with respect to the robot; and, (3) indexing (“clutching”), which allows the surgeon to disengage the input device 34 from the robot to reposition his/her arms and engage again.
  • Two input devices 34 are typically fabricated and integrated into each cockpit 2—one for each hand of the surgeon. The arm 62 of the input device 34 can include six or seven DOF or more. The arm can be constructed as a cable actuated SCARA-based machine (SCARA—Selective Compliant Articulated Robot Arm), or otherwise as desired (as with all components of the systems discussed herein, unless specifically stated otherwise, the specific materials, manufacturing methods, etc., for the components can be selected to optimize particular features and characteristics. Thus, the components can be made of steel, carbon fiber, etc., so long as the composition is acceptable for the desired purpose).
  • A cable actuated system can be common practice in designing haptic devices. It allows for location of the actuators on a stationary base 6 to transfer torques to each one of the joints through a system of pulleys and cables. This configuration leads to a lightweight, low inertia, low friction, and back drivable haptic device that can reflect back to the user the forces generated when the surgical robot interacts with tissues. Back drivability can be an important characteristic of any force feedback haptic device. It generates a negligible effect of resistance as the operator moves the input device 34 in free space which can be the desired response as the surgical robot does not interact with any tissue.
  • The actuation and position sensors can be supported by a system of amplifiers, along with low level software modules incorporating servo feedback loops for tele-operation and force feedback control algorithms.
  • If actuators with high gear ratios are introduced to the system, the user feels the reflected inertia along with the friction in the gearbox—forces that mask the smaller effect of force feedback generated as a result of the interaction of the surgical robotic tool interacting with the tissue. The SCARA-based mechanism can be a classical robotic arm configuration which includes three consecutive axes with a rotation axis perpendicular to the ground or parallel to the gravitational vector. As a result, gravitational loads are not fully reflected into the actuated joint and most of the load can be supported by the structure elements of the haptic device and not by the actuator associated with the joint—a situation that leads to use of smaller actuators for the joint.
  • Force feedback can be eliminated from the operational mode of the system if so desired. In such a situation, actuators are still typically incorporated into the tactile input device 34. Their secondary can be to preserve the registration between the input device 34 and the remote surgical tool by locking the orientation of the tip of the input device 34 (arm) once indexing is taking place. Indexing should typically only allowed for repositioning the end effector of the input device 34 in the Cartesian space (x,y,z) through translation or repositioning. The orientation of the input device 34 during the indexing process must typically be preserved. Reorientation is not typically allowed during the indexing process in order to preserve the registration between the input device 34 of the cockpit 2 and the surgical tool attached to the surgical robot.
  • Force feedback capabilities can be achieved by a cable driven master mechanism with a set of actuators (e.g., brushed DC motors) and position sensors (encoders and potentiometers) attached to its base. Brushless DC motors have a minor advantage compared with brushed motors as far as torque to weight ratio. However the high numbers of electrical wires for motor commutation may cancel out their minor advantage.
  • For example, the actuators can be selected to generate the following peak forces and torques: (1) translational forces 67 N, grasping force 42 N, torques 2.4 Nm. The actuation and position sensors can be supported by a system of amplifiers, along with newly developed low-level software modules incorporating servo feedback loops for teleoperation and force feedback control algorithms.
  • In another example, the input device 34 comprises: (1) the direct kinematics of the haptic arm defined by mapping the joint angle to the end effector (hand interface), (2) The Jacobian matrix can be derived by mapping the joint angular velocity to the end effector velocities, (3) the manipulability as a performance measure can be defined, (4) a cost function can be defined taking into account the manipulability measure and the link length of the mechanism. Using a brute force numerical solution, for example, the cost function can be calculated across an entire workspace of different combinations of link lengths, for example ones that have maximal dynamic manipulability with the minimal link lengths within a workspace of 10×10×10 cm.
  • The surgeon's hands interact with the master devices through a three-finger mechanical interface such as the three fingers haptic hand 64. In FIG. 10, the middle and index fingers are lumped into the first port 72, the ring and the fifth finger are lumped into the second port 74 and the thumb is in third port 76. In FIG. 12, the index finger is put into the first port 72, the middle, ring and the fifth finger are lumped into the second port 74 and the thumb is in third port 76.
  • The 3 fingers interface mediates the significant gap between the 5 fingers (5 fingers×4 DOF per finger=20 DOF) of the human hand and 2 fingers systems having only 1 DOF. FIG. 11 shows some of the degrees of freedom obtainable with a three-fingered hand. Thus, the three fingers interface allows a wide spectrum of control capabilities by the surgeon's hand over a remote surgical instrument or other remote device.
  • The haptic hands herein can switch between various user-finger configurations to maximize dexterity as desired. Each of the tactile devices such as the robotic arm 62 and three fingers haptic hand 64 can comprise 12 DOF or more.
  • Information perceived through the human sense of touch (haptics) can be classified into two categories, cutaneous and kinesthetic. Cutaneous information can be provided via the mechano-receptive nerve endings in the glabrous skin of the human hand. It can be primarily a means of relaying information regarding smallscale details in the form of skin stretch, compression and vibration. Kinesthetic sensing encompasses larger scale details, such as basic object shape and mechanical properties, for example, compliance. This can be achieved via feedback from the muscular and skeletal system.
  • If desired, not all DOF need have tactile (haptic) feedback. For example, only 3 out of the 6 DOF that can be sensed for the three fingers mechanism may include force feedback. Thus, in some embodiments, there will be a reduction from 12 potential actuated DOF in the arm-hand combination to only 9 fully actuated DOF with force feedback at the hand (fingers). For example, the remaining 3 non-actuated DOF will can be position information that can be provided by the low frequency motion of the hand and the arm, cutaneous information can be provided to the surgeon regarding tissue texture via high frequency actuators incorporated into the finger pad interface of the 3 fingers hand. Low frequency indicates forces provided in the range of 0-100 Hz; high frequency indicates forces provided in the range of 100 Hz and higher.
  • The following functions were identified for the third finger: microscope control, suction, irrigation, drilling, clutching. In order to provide full control of these functions, two types of buttons can be implemented: (1) binary button (On/Off); (b) gradual knob (Volume/Magnitude).
  • Master Device DOF with Force
    Joint DOF Feedback
    Shoulder
    2 2
    Elbow 1 1
    Wrist 3 3
    3 Fingers 6 3
    Total 12 9
  • In certain embodiments, each finger has force feedback on 2 out of the 3 DOF allowing flexion/extension movements and feedback. The fingers' adduction/abduction movements can be supported by a passive DOF with no force feedback.
  • The three-finger design is particularly useful for use with (1) virtual knobs and switches 78 (such as shown in FIG. 14). For example, to control retractors, electrocoagulators, or view of the scene and (2) anticipating new tool designs in the future, it allows the surgeon to regain the level of dexterity and manipulability of the human hand used in open surgery but lost in current robotic systems. The virtual control knob can be operably connected to one or more of the three fingers of the haptic input device(s). The virtual control knob can also be configured so that it must be virtually gripped by two or more fingers of the haptic device before it may be rotated or otherwise manipulated.
  • FIG. 9 depicts a cockpit 2 comprising an array of 8 intuitive haptic foot pedals (4 for each foot) that allows the surgeon to control multiple devices and to switch the control between them. Two desirable pedal configurations: (1) serial arrangement (similar to a car) and (2) spatial arrangement in which 4 pedals surround each foot such that by flexing/extending the ankle and rotating the feet left/right all 4 pedals are accessible. The serial arrangement can be limited to 3 pedals for each foot. Increasing the number of pedals beyond three makes it difficult for the surgeon to locate the pedals while being immersed visually in the surgical site. The spatial arrangement of 4 pedals sounding the feet from 4 different orthogonal directions provides an easier registration between the feet and the pedals and a richer medium as an input device 34.
  • The feet tactile interface can include a passive gimbal mechanism 84 to support the heel of the foot. The gimbal mechanism 84 allows the surgeon to move freely in any direction while avoiding gamble lock (the rotation in any direction will never exceed 90°). The four pedals can be arranged around the distal end of the feet in four orthogonal planes. Flexing the ankle joint will press the top pedal while extending the joint will press the bottom pedal. Moving the distal part of the foot left and right will press the left and right pedals.
  • A dead zone can be implemented in the design preventing a situation in which the feet activate two opposing functions simultaneously (e.g., suction and irrigation). Each pedal can be controlled by a single DC servo motor. Through the software, the force displacement characteristics can be defined. Displacement characteristics of each pedal provide the opportunity to change the function of this interface and assign functions to each pedal based on a specific operation (similar to the third finger of the hand interface). Moreover, the software can allow the surgeon to change the nature of the pedal from an on/off switch to a gradual control switch.
  • The distribution of functions between the hand's third finger and the foot can be assigned as desired. Functions such as controlling the cameras can be implemented by linking the ankle flexion/extension to the camera pitch movements and ankle rotation to the camera yaw movements. Haptics can be added to the pedals. For example, force feedback applied through the pedals can be correlated with irrigation or suction pressure.
  • As noted previously, the cockpits 2 herein can also comprise two or more different surgical consoles in one cockpit or two or more different surgical consoles in two or more different surgical cockpits that are operably connected to each other either locally or at the remote surgical site (or otherwise as desired). This allows, e.g., surgeons in completely different locales to operate simultaneously on a single surgical site. This also allows, e.g., surgeons in the same or different locales to relieve each other in a single surgery at a single surgical site. This still further allows, e.g., exceptional training of surgeons one by another, including providing tactile feedback to a student surgeon, for example from the movements of the surgical instruments controlled by the teaching surgeon or to the student surgeon from a surgical site being operated on by the teaching surgeon.
  • FIG. 15 depicts an isometric view of a system comprising four remote robotics arms 86 a-86 d holding remote surgical instruments 88 a-88 d in a remote operating room, which arms 86 can be teleoperated by one or two (or more) surgeons in multiple modes of operation, such as (1) solo by a single surgeon from a local or a remote location (2) by two surgeons in which one or both are located locally with the robot or in one or two remote sites.
  • The four robotic arms can contain a variety of peripheral devices and/or functions, such as an endoscope 100 comprising an endoscopic camera 102 in remote surgical instrument 88 a coupled with positioning and/or zooming the camera, camera angle, camera focus. Similarly, remote surgical instrument 88 b can contain a suction device 104 and irrigation device 106, and remote surgical instrument 88 d comprises a cautery device 108 for, for example, electric coagulation or laser photocoagulation. The robotic arms can also be under the control of robot brakes 110 and can provide a staple applier 114.
  • The four arms 86 system duplicates two surgeons collaborating and simultaneously interacting with the remote surgical site. The new system provides a new opportunity to explore collaborative surgery which to some extent was not possible before due to the limited number of available remote surgical arms 86. Moreover, the control over one or two of the arms can be assigned to an artificial agent (software) and to facilitate new methods of automation. In certain embodiments, any robotic arm can be assigned at any point by the primary surgeon to the other surgeon(s) or the artificial agent regardless of its location (local or remote).
  • In certain embodiments, the four remote robotic arms are held in a sole arm-retention structure 116. The sole arm-retention structure can be configured to hold the four remote robotic arms such that the arms cannot collide with each other.
  • The cockpit can be part of a system comprising at least two local surgical cockpits each configured for an operator, and the system can be configured such that each operator can simultaneously hold a single remote robotic arm, and/or so that the operators can switch control of a remote robotic arm between each other.
  • FIG. 15 also depicts an external, remote operating room camera 98. The functions of this camera such as position, zooming, angle, focus, etc., and other remote operating room devices and peripherals can also be controlled by the surgeon or other operator in the surgical cockpit 2.
  • FIG. 6 depicts an exemplary headset 90 comprising a 3-D auditory input device 80. The Auditory Interface can be any suitable 3-D auditory system such as a system comprising two sonic beam focused speakers. 3-D auditory technologies include Hypersonic Sound (HSS) by American Technology Corp and Audio Spotlight by Holosonics. These technologies allow a focal beam of sound to be transmitted into a specific point in space (surgeon's ears) and avoid sound “pollution” in the operating room (OR). Both can still provide desired sound input from the operating room.
  • 3-D auditory input devices 80 permit local surgical areas/cockpits to become consistent virtual listening areas without the pervasive omnidirectionality of conventional loudspeakers. The high-precision targeting of directional beam of sound significantly minimizes the levels of noise pollution in the local operating rooms while still allowing the surgeon to respond to other sources of auditory inputs from the remote operating room.
  • The 3-D auditory input devices 80 can be used, for example, to convey the following information via either natural or synthetic sound cues: (1) collisions between the surgical tools, (2) contact between the tool tip and some types of tissue, (3) stress levels applied to the tissue, (4) vital signs and emergency limits, (5) local pulse and vascular blood flow to denote vessels that may need to be preserved or ligated.
  • There are several types of 3D audio effects, such as (1) widening the stereo image by modifying phase information; (2) placing sounds outside the stereo basis and (3) complete 3D simulation. Sound can be complementary to haptics because little kinesthetic information extends into the audio frequency range. Audio may play an important role in perceiving the important information generated by the surgical site or the OR by the surgeon located in a remote site.
  • FIG. 16 is a block diagram depicting certain aspects of an exemplary software architecture that can be used with the local surgical cockpits 2 and remote surgical procedures herein. In this example, the software architecture includes two layers: high level software 92 and low level software 94. The software comprises multiple interfaces to peripheral hardware and software components such as seat 8, display 50 and 3-D audio input devices 80. The software can also be used for the surgical cockpit 2 to communicate with any desired surgical module (surgical robot) that shares a proper communication channel such as a common and universally accepted Transmission Control Protocol/Internet Protocol (TCP/IP). Thus, the surgical cockpit can not only be used with nearby remote surgical sites but also with distant remote surgery sites located outside at least one, and possibly any, building containing at least one of the surgical cockpit(s).
  • The high level software 92 can comprise a software layer that smoothly interfaces the teleoperation capability with image-guided software modules residing either at the remote or local surgeon site, and a software module that facilitates collaborative communication among multiple surgical consoles. The high-level software module 92 can merge the information provided by the hand, foot, and peripheral input device 34. Image data and vital signs are presented continuously and/or through computer generated audio cues. The high level module can be divided into processes, and if desired each process can be dedicated to one of the peripheral elements depicted in FIG. 8.
  • The low level software 94 can comprise a robust software layer that leverages control techniques to support haptic feedback via Internet connections at local, national, and global scales. The low-level software module can be responsible for the interpretation of the low-level input signals acquired from the hand and feet interfaces. Signals are translated to and from the surgical robot over a network using TCP/IP.
  • Network communications can update the software modules, for example the high level module that manages visual and audio display.
  • The low level software can be primarily a real-time segment of the software running on a RT Linux operating system. The high level module can be non real-time software.
  • If desired, the interaction between the high level and low level modules can be primarily unidirectional, in which position and orientation of the surgical tool location can be sent from the low level to the high level, with the only exception in which information can be sent back to the low level module being when a tool reaches a position that exceeds a predefined safe zone.
  • As may be seen from the foregoing, the present invention provides an improved surgical device that permits surgical instruments to reach remote portions of the body with reduced trauma. The device sheath may be steered to a surgical site around sensitive or critical tissue. The surgical tool components may be removed for replacement or cleaning without the device having to be straightened or removed from the body. Further, the tool deflection assemblies and methodology renders precise control of the surgical tool components in all required degrees and directions of movement. The present invention is thus well suited for use in many different applications, including robotic surgical systems.
  • From the foregoing, it will be appreciated that, although specific embodiments have been discussed herein for purposes of illustration, various modifications may be made without deviating from the spirit and scope of the discussion herein. Accordingly, the systems and methods, etc., include such modifications as well as all permutations and combinations of the subject matter set forth herein and are not limited except as by the appended claims or other claim having adequate support in the discussion and figures herein.

Claims (81)

1. A local surgical cockpit comprising:
a base, a frame disposed on the base, a seat for an operator disposed on the frame, and a remote surgical console configured such that the operator can operate the console for remote surgery while in the seat, wherein the seat is ergonomic and is operably connected to the frame such that the seat can be retainably tilted from a substantially upright position to a substantially supine position.
2. The local surgical cockpit of claim 1 wherein the seat comprises an independently movable headrest, backrest, seating plate and footrest, the seat further comprising positioning elements operably connected to the independently movable headrest, backrest, seating plate and footrest and providing at least three axes of retainable positioning movement for each of the independently movable headrest, backrest, seating plate and footrest.
3. The local surgical cockpit of claim 2 wherein the seat includes a lumbar support comprising retainable positioning movement for support of the lower back.
4. The local surgical cockpit of claim 1 or 2 wherein a reference body posture of the seat corresponds to a human body posture that is fully relaxed in micro gravity.
5. The local surgical cockpit of claim 1 or 2 wherein the cockpit further comprises at least one peripheral device operably connected to move with the seat when the seat is moved so that the location of the peripheral device relative to the operator in the seat is substantially unchanged.
6. The local surgical cockpit of claim 4 wherein the peripheral device is at least one of a monitor facing an operator in the seat and operably linked to display a remote surgical site, a heads-up display disposed in front of the local operator's eyes, and an input device disposed at a hand of the operator and operably linked to provide input to a corresponding device located at the remote surgical site.
7. The local surgical cockpit of claim 6 wherein the cockpit comprises all of the monitor, the heads-up display and the input device.
8. A local surgical cockpit comprising:
a local surgical cockpit comprising a local surgical console configured for transmitting surgical movements of local operator operating the local surgical console to a remote surgery site, and a head-mounted display disposed in front of the local operator's eyes in surgical position in the cockpit to operate the console for surgery, wherein the head-mounted display is configured to depict at least images of a remote surgical site under remote operation by the operator.
9. The local surgical cockpit of claim 8 wherein the local surgical cockpit is part of a system and the system further comprises remote image sensors operably connected to the head-mounted display to transmit the image of the remote surgical site.
10. The local surgical cockpit of claim 8 wherein the head-mounted display extends to the local operator's eyes from an articulated boom disposed in front of the local operator's eyes.
11. The local surgical cockpit of claim 8 wherein the articulated boom is actuated by at least one hand control located on the cockpit.
12. The local surgical cockpit of claim 8 wherein the boom is actuated by voice control.
13. The local surgical cockpit of claim 8 wherein the head-mounted display is disposed on a head-mounted frame configured to rest on an operator's head and to maintain the images in front of the local operator's eyes when the operator's head moves.
14. The local surgical cockpit of claim 8 wherein the head-mounted display comprises two separate streams of video displayed to each eye of the local operator's eyes, each stream comprising corresponding right and left eye views of a remote surgical site to provide a 3-D image of the site.
15. The local surgical cockpit of claim 8 wherein the cockpit further comprises at least one monitor.
16. A local surgical cockpit comprising:
a local surgical cockpit comprising a local surgical console configured for transmitting surgical movements of an operator operating the local surgical console to a remote surgery site, and comprising at least one image display device configured to depict at least one image of the remote surgical site, the display device further depicting augmented reality for the operator comprising augmented information shown on the display and superimposed over the image of the remote surgical site.
17. The local surgical cockpit of claim 16 wherein the local surgical cockpit is part of a system and the system further comprises remote image sensors operably connected to the head-mounted display to transmit the image of the remote surgical site.
18. The local surgical cockpit of claim 16 wherein the augmented information comprises at least one of preselected margins to dissect during the surgery and a mask of vital structures in the remote surgical site overlaid over the images of the remote surgical site.
19. The local surgical cockpit of claim 16 or 18 wherein the display device further displays further augmented information either to a side of or superimposed over the image of the remote surgical site and the further augmented information comprises at least one of blood pressure, temperature, O2 level, CO2 level, intracranial pressure, a preplanned trajectory for a surgical tool, tool type, suction on/off, a bottom task bar, recording capabilities, current time, and elapsed time.
20. The local surgical cockpit of claim 16 or 18 wherein the image of the remote surgical site and the augmented information comprise blending graphical images with real-world views of the remote surgical site.
21. The local surgical cockpit of claim 16 or 18 wherein the image of the remote surgical site is provided by at least one of an endoscopic camera, a remote surgical site camera, or a camera showing an operating room.
22. A local surgical cockpit comprising:
a local surgical cockpit comprising a local surgical console configured for transmitting surgical movements of an operator operating the local surgical console to a remote surgery site, and comprising a local surgical instrument comprising local input surgical fingers configured to provide input to corresponding remote surgical fingers configured to manipulate a remote surgical instrument at a remote operation site, wherein the local fingers are high frequency haptic fingers configured to provide tactile feedback to the operator based on acceleration of the remote surgical instrument manipulated by the remote surgical fingers.
23. The local surgical cockpit of claim 22 wherein the local surgical cockpit is part of a system and the system further comprises the remote surgical fingers, and wherein the remote surgical fingers are haptic fingers configured to provide tactile feedback to the operator based on acceleration of the remote surgical instrument manipulated by the remote surgical fingers.
24. The local surgical cockpit of claim 23 wherein the local surgical cockpits of the system are configured such that operators in different locales can operate simultaneously on a single surgical site.
25. The local surgical cockpit of claim 24 wherein the local surgical cockpits of the system are configured such that operators can relieve each other in a single surgery at a single surgical site.
26. The local surgical cockpit of claim 23 wherein the local surgical cockpits of the system are configured to provide a teaching surgical cockpit and a student surgical cockpit providing haptic feedback to a student operator generated by a teaching operator.
27. The local surgical cockpit of claim 23 wherein the haptic feedback to the student comprises movements of a remote surgical instrument controlled by the teaching operator.
28. The local surgical cockpit of claim 23 wherein the haptic feedback to the student comprises tactile feedback from a surgical site being operated on by the teaching operator.
29. A local surgical cockpit comprising:
a local surgical cockpit comprising a local surgical console configured for transmitting surgical movements of an operator operating the local surgical console to a remote surgery site, and comprising at least seven degrees of freedom for a local surgical instrument manipulated by a robotic arm manipulated by the operator, wherein the console is configured such that the seven degrees of freedom are transmissible to a remote surgical instrument located at a remote surgical site and manipulated by the operator operating the console.
30. The local surgical cockpit of claim 29 wherein the local surgical cockpit is part of a system and the system further comprises the remote surgical instrument operably connected to the local surgical instrument such that the remote surgical instrument precisely responds in at least seven corresponding degrees of freedom to movements of the local surgical instrument.
31. The local surgical cockpit of claim 29 wherein the degrees of freedom comprise at least nine degrees of freedom for the local surgical instrument manipulated by the operator and a corresponding nine degrees of freedom for the remote surgical instrument.
32. The local surgical cockpit of claim 29 wherein the degrees of freedom comprise at least twelve degrees of freedom for the local surgical instrument manipulated by the operator and a corresponding twelve degrees of freedom for the remote surgical instrument, wherein the local robotic arm comprises a shoulder joint, an elbow joint, a wrist joint and the three fingers, each comprising at least the following degrees of freedom: shoulder comprises 2 degrees of freedom; elbow comprises 1 degree of freedom; wrist comprises 3 degrees of freedom; the three fingers comprise 2 degrees of freedom each.
33. The local surgical cockpit of claim 29 wherein local surgical instrument comprises at least three input fingers configured to provide input to a corresponding at least three remote surgical fingers configured to manipulate a remote surgical instrument at a remote operation site, wherein the at least three input fingers are configured to be manipulated by a single hand of an operator operating the local surgical instrument, and wherein the at least seven degrees of freedom comprise at least two degrees of freedom for two of the three remote surgical fingers and at least three degrees of freedom for a third of the three remote surgical fingers.
34. The local surgical cockpit of claim 29 wherein local surgical instrument comprises at least three input fingers configured to provide input to a corresponding at least three remote surgical fingers configured to manipulate a remote surgical instrument at a remote operation site, wherein the at least three input fingers are configured to be manipulated by a single hand of an operator operating the local surgical instrument, and wherein the degrees of freedom comprise at least nine degrees of freedom comprising at least three degrees of freedom for each of the three remote surgical fingers.
35. The local surgical cockpit of claim 29 wherein local surgical instrument comprises at least three input fingers configured to provide input to a corresponding at least three remote surgical fingers configured to manipulate a remote surgical instrument at a remote operation site, wherein the at least three input fingers are configured to be manipulated by a single hand of an operator operating the local surgical instrument, wherein the local robotic arm comprises a shoulder joint, an elbow joint, a wrist joint and the three fingers, each comprising at least the following degrees of freedom: shoulder comprises 2 degrees of freedom; elbow comprises 1 degree of freedom; wrist comprises 3 degrees of freedom; the three fingers comprise 2 degrees of freedom each.
36. The local surgical cockpit of claim 35 wherein the three fingers comprise 3 degrees of freedom each.
37. The local surgical cockpit of claim 29 wherein the degrees of freedom provide for variable desired positioning and orientation of a tip of the remote surgical instrument in space in 6 parameters including Cartesian position (x,y,z), and angular orientation (x y z θ, θ, θ).
38. The local surgical cockpit of claim 29 wherein control of the remote surgical instrument further comprises scaling factors configured such that motion input by the operator is attenuated or amplified with respect to the remote surgical instrument.
39. The local surgical cockpit of claim 29 wherein control of the remote surgical instrument further comprises indexing configured to allow the operator to disengage the input device from the remote surgical instrument to reposition his/her arms and engage again.
40. A local surgical cockpit comprising:
a local surgical cockpit comprising a local surgical console configured for transmitting surgical movements of an operator operating the local surgical console to a remote surgery site, and comprising a local surgical instrument comprising at least three input fingers configured to provide input to a corresponding at least three remote surgical fingers configured to manipulate a remote surgical instrument at a remote operation site, wherein the at least three input fingers are configured to be manipulated by a single hand of an operator operating the local surgical instrument.
41. The local surgical cockpit of claim 40 wherein the local surgical cockpit is part of a system and the system further comprises the three remote surgical fingers operably connected to the three input fingers such that the three remote surgical fingers precisely respond to movements of the three input fingers.
42. The local surgical cockpit of claim 40 wherein the at least three input fingers are configured to correspond respectively to a) an operator's thumb, b) an operator's index and middle fingers, and c) an operator's ring and little fingers.
43. The local surgical cockpit of claim 40 wherein the at least three input fingers are configured to correspond respectively to a) an operator's thumb, b) an operator's index finger, and c) an operator's middle, ring and little fingers.
44. The local surgical cockpit of claim 40 wherein the at least three input fingers are haptic fingers configured to provide tactile feedback to the operator based on acceleration of a remote surgical instrument manipulated by the remote surgical fingers.
45. The local surgical cockpit of claim 40 wherein the three input fingers are operably connected so that two fingers control remote surgical fingers and the remaining third finger controls an external device.
46. The local surgical cockpit of claim 40 wherein the external device is a one or more of an electrocautery device, a laser photocoagulator, a staple applier.
47. The local surgical cockpit of claim 40 wherein the external device is an optical aspect of the camera system such as focus, zoom, rotation, or field-of-view.
48. A local surgical cockpit comprising:
a local surgical cockpit comprising a local surgical console configured for transmitting surgical movements of an operator operating the local surgical console to a remote surgery site, and comprising at least one haptic foot pedal configured to be operably connected to at least one remote device at a remote operation site, wherein the at least one haptic foot pedal is configured to be manipulated by a foot of the operator operating the local surgical console to cause a movement or control change in the remote device.
49. The local surgical cockpit of claim 48 wherein the local surgical cockpit is part of a system and the system further comprises the at least one remote device operably connected to the at least one haptic foot pedal.
50. The local surgical cockpit of claim 48 wherein the local surgical cockpit further comprises at least two haptic foot pedals configured to be operably connected to the at least one remote device at the remote operation site.
51. The local surgical cockpit of claim 48 wherein the local surgical cockpit further comprises at least eight haptic foot pedals configured to be operably connected to at least two remote devices at the remote operation site, the at least eight haptic foot pedals divided to provide at least a first foot pedal set and second foot pedal set, wherein a first foot pedal set and second foot pedal set are each disposed to be manipulated by a right foot of the operator and by a left foot of the operator, respectively.
52. The local surgical cockpit of claim 51 wherein, for each of the first foot pedal set and second foot pedal set, the sets each contain four pedals with each of the four pedals in one of four quadrants of a circle.
53. The local surgical cockpit of claim 51 wherein opposed pairs of the four pedals are assigned opposed functions at the remote surgical site.
54. The local surgical cockpit of claim 51 wherein the opposed functions are suction and irrigation.
55. The local surgical cockpit of claim 51 wherein each of the four pedals is assigned complementary functions for a remote instrument at the remote surgical site.
56. The local surgical cockpit of claim 55 wherein each of the four pedals controls the viewing angles of an endoscopic camera.
57. The local surgical cockpit of claim 48 wherein the local surgical cockpit further comprises a dead zone that prevents two opposing functions being implemented simultaneously.
58. The local surgical cockpit of claim 48 wherein the at least one haptic foot pedal controls at least one of camera angle, camera zoom, camera focus, irrigation, suction, robot brakes, electric coagulation, laser photocoagulation.
59. A local surgical cockpit comprising:
a local surgical cockpit comprising a local surgical console configured for transmitting surgical movements of an operator operating the local surgical console to a remote surgery site, and comprising at least one virtual console control knob presented virtually to the operator and configured to be manipulated by the operator to generate control signals for a corresponding remote device at a remote operation site.
60. The local surgical cockpit of claim 59 wherein the local surgical cockpit is part of a system and the system further comprises the corresponding remote device.
61. The local surgical cockpit of claim 60 wherein the at least one virtual console control knob is a binary switch configured to provide on/off signals to the corresponding remote device.
62. The local surgical cockpit of claim 60 wherein the at least one virtual console control knob is a gradual control knob configured to provide gradual control signals to the corresponding remote device.
63. The local surgical cockpit of claim 59 wherein the virtual control knob is operably connected to one of the three fingers of the haptic device.
64. The local surgical cockpit of claim 59 wherein the virtual control knob must be virtually gripped by two or more fingers of the haptic device before it may be rotated.
65. A local surgical cockpit system comprising:
at least two local surgical cockpits each comprising a surgical console configured for transmitting surgical movements of an operator operating the local surgical console to a distant remote surgery site located outside at least one building containing at least one of the surgical cockpits, wherein each cockpit comprises a respective first and second set of at least two local robotic input arms configured to provide input to corresponding first and second remote sets of at least two corresponding remote robotic arms each configured to manipulate a remote surgical instrument at a single remote operation site, wherein the respective first and second set of local robotic input arms are configured to be manipulated by respective first and second operators working in concert on the remote surgical site.
66. The local surgical cockpit system of claim 65 wherein the local surgical cockpit system is part of a further system and the further system further comprises the first and second remote sets of at least two corresponding remote robotic arms.
67. The local surgical cockpit system of claim 65 wherein the distant remote surgery site is located outside any building containing any of the local surgical cockpits.
68. The local surgical cockpit of claim 65 wherein the local surgical cockpits of the system are configured such that operators in different locales can operate simultaneously on a single surgical site.
69. The local surgical cockpit of claim 65 wherein the local surgical cockpits of the system are configured such that operators can relieve each other in a single surgery at a single surgical site.
70. The local surgical cockpit of claim 65 wherein the local surgical cockpits of the system are configured to provide a teaching surgical cockpit and a student surgical cockpit providing haptic feedback to a student operator generated by a teaching operator.
71. The local surgical cockpit of claim 70 wherein the haptic feedback to the student comprises movements of a remote surgical instrument controlled by the teaching operator.
72. The local surgical cockpit of claim 70 wherein the haptic feedback to the student comprises tactile feedback from a surgical site being operated on by the teaching operator.
73. A local surgical cockpit comprising:
a local surgical cockpit comprising a local surgical console configured for transmitting surgical movements of an operator operating the local surgical console to a remote surgery site, and comprising at least four local robotic input arms configured to provide input to a corresponding at least four remote robotic arms each configured to manipulate a remote surgical instrument at a remote operation site, wherein the at least four local robotic input arms are configured to be manipulated by at least one operator operating the local surgical console.
74. The local surgical cockpit of claim 73 wherein the local surgical cockpit is part of a system and the system further comprises the four remote robotic arms operably connected to the four local robotic input arms such that the four remote robotic arms precisely respond to movements of the four local robotic input arms.
75. The local surgical cockpit of claim 73 wherein the four remote robotic arms are held in a sole arm-retention structure.
76. The local surgical cockpit of claim 75 wherein the sole arm-retention structure is configured to hold the four remote robotic arms such that the arms cannot collide with each other.
77. The local surgical cockpit of claim 73 wherein the cockpit is part of a system comprising at least two local surgical cockpits each configured for an operator, and wherein the system is configured such that each operator can simultaneously hold a single remote robotic arm.
78. The local surgical cockpit of claim 73 wherein the cockpit is part of a system comprising at least two local surgical cockpits each configured for an operator, and wherein the system is configured such that the operators can switch control of a remote robotic arm between each other.
79. A local surgical cockpit comprising:
a local surgical cockpit comprising a local surgical console configured for transmitting surgical movements of an operator operating the local surgical console to a remote surgery site, and comprising local 3-dimensional audio configured to obtain remote 3-dimensional audio input from a remote operation site and provide corresponding local 3-dimensional audio to an operator operating the console.
80. The local surgical cockpit of claim 79 wherein the local surgical cockpit is part of a system and the system further comprises remote 3-dimensional audio sensors operably connected to the local 3-dimensional audio such that the local 3-dimensional audio precisely transmit 3-dimensional audio signals from the remote 3-dimensional audio sensors.
81. The local surgical cockpit of claim 79 wherein the 3-dimensional audio signals are correlated with tactile feedback to provide correlated response to haptic input devices at the local surgical cockpit.
US13/051,921 2010-03-18 2011-03-18 Surgical Cockpit Comprising Multisensory and Multimodal Interfaces for Robotic Surgery and Methods Related Thereto Abandoned US20110238079A1 (en)

Priority Applications (2)

Application Number Priority Date Filing Date Title
US13/051,921 US20110238079A1 (en) 2010-03-18 2011-03-18 Surgical Cockpit Comprising Multisensory and Multimodal Interfaces for Robotic Surgery and Methods Related Thereto
US14/181,388 US9474580B2 (en) 2010-03-18 2014-02-14 Surgical cockpit comprising multisensory and multimodal interfaces for robotic surgery and methods related thereto

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US31501810P 2010-03-18 2010-03-18
US13/051,921 US20110238079A1 (en) 2010-03-18 2011-03-18 Surgical Cockpit Comprising Multisensory and Multimodal Interfaces for Robotic Surgery and Methods Related Thereto

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US14/181,388 Continuation US9474580B2 (en) 2010-03-18 2014-02-14 Surgical cockpit comprising multisensory and multimodal interfaces for robotic surgery and methods related thereto

Publications (1)

Publication Number Publication Date
US20110238079A1 true US20110238079A1 (en) 2011-09-29

Family

ID=44649855

Family Applications (2)

Application Number Title Priority Date Filing Date
US13/051,921 Abandoned US20110238079A1 (en) 2010-03-18 2011-03-18 Surgical Cockpit Comprising Multisensory and Multimodal Interfaces for Robotic Surgery and Methods Related Thereto
US14/181,388 Expired - Fee Related US9474580B2 (en) 2010-03-18 2014-02-14 Surgical cockpit comprising multisensory and multimodal interfaces for robotic surgery and methods related thereto

Family Applications After (1)

Application Number Title Priority Date Filing Date
US14/181,388 Expired - Fee Related US9474580B2 (en) 2010-03-18 2014-02-14 Surgical cockpit comprising multisensory and multimodal interfaces for robotic surgery and methods related thereto

Country Status (2)

Country Link
US (2) US20110238079A1 (en)
WO (1) WO2011116332A2 (en)

Cited By (120)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20100228264A1 (en) * 2009-03-09 2010-09-09 David Robinson Adaptable integrated energy control system for electrosurgical tools in robotic surgical systems
US20110234484A1 (en) * 2010-03-29 2011-09-29 Olympus Corporation Operation input unit and manipulator system
US20120010749A1 (en) * 2010-04-09 2012-01-12 Deka Products Limited Partnership System and apparatus for robotic device and methods of using thereof
US20120130739A1 (en) * 2010-11-21 2012-05-24 David Crane Unsupervised Telemedical Office for Remote &/or Autonomous & Automated Medical Care of Patients
US20140018819A1 (en) * 2012-07-16 2014-01-16 Anil K Raj Anthro-Centric Multisensory Interface for Sensory Augmentation of Telesurgery
WO2014047402A1 (en) * 2012-09-20 2014-03-27 MUSC Foundation for Research and Development Head-mounted systems and methods for providing inspection, evaluation or assessment of an event or location
US20140148818A1 (en) * 2011-08-04 2014-05-29 Olympus Corporation Surgical assistant system
US20140204079A1 (en) * 2011-06-17 2014-07-24 Immersion System for colocating a touch screen and a virtual object, and device for manipulating virtual objects implementing such a system
WO2015023513A1 (en) * 2013-08-14 2015-02-19 Intuitive Surgical Operations, Inc. Endoscope control system
WO2015084837A1 (en) * 2013-12-02 2015-06-11 Immersive Touch, Inc. Improvements for haptic augmented and virtual reality system for simulation of surgical procedures
WO2014052158A3 (en) * 2012-09-27 2015-07-16 Immersive Touch, Inc. Haptic augmented reality for simulation of surgical procedure
US20150199106A1 (en) * 2014-01-14 2015-07-16 Caterpillar Inc. Augmented Reality Display System
CN104840253A (en) * 2014-02-18 2015-08-19 三星电子株式会社 Master devices for surgical robots and control methods thereof
US20150293362A1 (en) * 2012-11-13 2015-10-15 Sony Corporation Image display apparatus, image display method, mobile apparatus, image display system, and computer program
US9161772B2 (en) 2011-08-04 2015-10-20 Olympus Corporation Surgical instrument and medical manipulator
US20150317910A1 (en) * 2013-05-03 2015-11-05 John James Daniels Accelerated Learning, Entertainment and Cognitive Therapy Using Augmented Reality Comprising Combined Haptic, Auditory, and Visual Stimulation
US9244524B2 (en) 2011-08-04 2016-01-26 Olympus Corporation Surgical instrument and control method thereof
US9244523B2 (en) 2011-08-04 2016-01-26 Olympus Corporation Manipulator system
US9298884B1 (en) * 2014-12-17 2016-03-29 Vitaax Llc Remote instruction and monitoring of health care
US9301811B2 (en) 2012-09-17 2016-04-05 Intuitive Surgical Operations, Inc. Methods and systems for assigning input devices to teleoperated surgical instrument functions
WO2016077531A1 (en) * 2014-11-13 2016-05-19 Intuitive Surgical Operations, Inc. Integrated user environments
US9423869B2 (en) 2011-08-04 2016-08-23 Olympus Corporation Operation support device
US9474580B2 (en) 2010-03-18 2016-10-25 SPI Surgical, Inc. Surgical cockpit comprising multisensory and multimodal interfaces for robotic surgery and methods related thereto
US9477301B2 (en) 2011-08-04 2016-10-25 Olympus Corporation Operation support device and assembly method thereof
US20160320862A1 (en) * 2014-05-01 2016-11-03 Aaron Schradin Motion control seat input device
US9519341B2 (en) 2011-08-04 2016-12-13 Olympus Corporation Medical manipulator and surgical support apparatus
US9524022B2 (en) 2011-08-04 2016-12-20 Olympus Corporation Medical equipment
US20170036684A1 (en) * 2015-08-06 2017-02-09 Progress Rail Services Corporation Cabin monitoring system for train locomotive
US9568992B2 (en) 2011-08-04 2017-02-14 Olympus Corporation Medical manipulator
US9632577B2 (en) 2011-08-04 2017-04-25 Olympus Corporation Operation support device and control method thereof
US9632573B2 (en) 2011-08-04 2017-04-25 Olympus Corporation Medical manipulator and method of controlling the same
US9671860B2 (en) 2011-08-04 2017-06-06 Olympus Corporation Manipulation input device and manipulator system having the same
USD789105S1 (en) 2015-09-18 2017-06-13 Sv Tool Corporation Ergonomic chair
US20170186157A1 (en) * 2015-12-23 2017-06-29 Siemens Healthcare Gmbh Method and system for outputting augmented reality information
CN107049523A (en) * 2017-05-08 2017-08-18 成都中科博恩思医学机器人有限公司 Operational control component and surgical operation robot system
CN107334550A (en) * 2017-07-31 2017-11-10 成都中科博恩思医学机器人有限公司 Operating robot surgeon console slidably helps elbow platform
US9851782B2 (en) 2011-08-04 2017-12-26 Olympus Corporation Operation support device and attachment and detachment method thereof
US20180098813A1 (en) * 2016-10-07 2018-04-12 Simbionix Ltd. Method and system for rendering a medical simulation in an operating room in virtual reality or augmented reality environment
US20180157317A1 (en) * 2016-08-18 2018-06-07 Technische Universität Dresden System and method for haptic interaction with virtual objects
US20180243910A1 (en) * 2015-08-25 2018-08-30 Kawasaki Jukogyo Kabushiki Kaisha Remote control robot system and method of operating the same
US10108266B2 (en) 2012-09-27 2018-10-23 The Board Of Trustees Of The University Of Illinois Haptic augmented and virtual reality system for simulation of surgical procedures
US10166061B2 (en) 2014-03-17 2019-01-01 Intuitive Surgical Operations, Inc. Teleoperated surgical system equipment with user interface
US10215989B2 (en) 2012-12-19 2019-02-26 Lockheed Martin Corporation System, method and computer program product for real-time alignment of an augmented reality device
US20190086998A1 (en) * 2016-03-11 2019-03-21 Limbic Life Ag Occupant support device and system for controlling objects
EP3434219A4 (en) * 2016-04-28 2019-04-10 Sony Corporation Control device, control method, program, and sound output system
US20190201146A1 (en) * 2017-12-28 2019-07-04 Ethicon Llc Safety systems for smart powered surgical stapling
US20190244537A1 (en) * 2018-02-02 2019-08-08 Access Virtual, LLC Virtual reality based pilot training system
US10437335B2 (en) 2015-04-14 2019-10-08 John James Daniels Wearable electronic, multi-sensory, human/machine, human/human interfaces
US10492873B2 (en) * 2016-10-25 2019-12-03 Novartis Ag Medical spatial orientation system
US10568703B2 (en) * 2016-09-21 2020-02-25 Verb Surgical Inc. User arm support for use in a robotic surgical system
US10631939B2 (en) 2012-11-02 2020-04-28 Intuitive Surgical Operations, Inc. Systems and methods for mapping flux supply paths
JP2020144803A (en) * 2019-03-08 2020-09-10 株式会社フジ医療器 Controller chair
US20200281674A1 (en) * 2017-09-08 2020-09-10 Covidien Lp High precision instrument control mode for robotic surgical systems
US10864048B2 (en) 2012-11-02 2020-12-15 Intuitive Surgical Operations, Inc. Flux disambiguation for teleoperated surgical systems
US10874469B2 (en) 2017-05-22 2020-12-29 Tsinghua University Remotely operated orthopedic surgical robot system for fracture reduction with visual-servo control method
CN112168354A (en) * 2020-10-14 2021-01-05 北京科迈启元科技有限公司 Waterproof and lightweight surgical robot actuator and surgical robot system
DE102019127631A1 (en) * 2019-10-14 2021-04-15 Bayerische Motoren Werke Aktiengesellschaft Display system for a vehicle
US11032125B2 (en) * 2011-07-11 2021-06-08 Board Of Regents Of The University Of Nebraska Robotic surgical devices, systems and related methods
US11036985B2 (en) * 2014-05-15 2021-06-15 Fenwal, Inc. Head mounted display device for use in a medical facility
US11080634B2 (en) 2016-03-29 2021-08-03 Locatee Ag Device, system and method for monitoring usage of functional facilities
US11083528B2 (en) 2017-03-09 2021-08-10 Memic Innovative Surgery Ltd. Input arm for control of a surgical mechanical arm
US11116589B2 (en) * 2014-09-04 2021-09-14 Memic Innovative Surgery Ltd. Control of device including mechanical arms
GB2593473A (en) * 2020-03-23 2021-09-29 Cmr Surgical Ltd Virtual console for controlling a surgical robot
US20210370506A1 (en) * 2020-05-29 2021-12-02 Honda Motor Co., Ltd. Database construction for control of robotic manipulator
WO2021250580A1 (en) * 2020-06-10 2021-12-16 Mazor Robotics Ltd. Multi-arm robotic system enabling multiportal endoscopic surgery
US11229787B2 (en) 2016-11-25 2022-01-25 Kinaptic, LLC Haptic human machine interface and wearable electronics methods and apparatus
US11468793B2 (en) 2020-02-14 2022-10-11 Simbionix Ltd. Airway management virtual reality training
US11488381B2 (en) 2014-05-15 2022-11-01 Fenwal, Inc. Medical device with camera for imaging disposable
US20220398936A1 (en) * 2021-06-15 2022-12-15 Richard Parker Aircraft training aid systems and processes
US11559347B2 (en) 2015-09-30 2023-01-24 Cilag Gmbh International Techniques for circuit topologies for combined generator
US11589933B2 (en) * 2017-06-29 2023-02-28 Ix Innovation Llc Guiding a robotic surgical system to perform a surgical procedure
US11589916B2 (en) 2019-12-30 2023-02-28 Cilag Gmbh International Electrosurgical instruments with electrodes having variable energy densities
WO2023046185A1 (en) * 2021-09-26 2023-03-30 武汉联影智融医疗科技有限公司 Master control station for puncture operation, and puncture robot
US11633087B2 (en) * 2018-08-07 2023-04-25 The Chinese University Of Hong Kong Endoscope manipulator and method for controlling the same
US11660089B2 (en) 2019-12-30 2023-05-30 Cilag Gmbh International Surgical instrument comprising a sensing system
US11666375B2 (en) 2015-10-16 2023-06-06 Cilag Gmbh International Electrode wiping surgical device
US11684412B2 (en) 2019-12-30 2023-06-27 Cilag Gmbh International Surgical instrument with rotatable and articulatable surgical end effector
US11684402B2 (en) 2016-01-15 2023-06-27 Cilag Gmbh International Modular battery powered handheld surgical instrument with selective application of energy based on tissue characterization
US11696776B2 (en) 2019-12-30 2023-07-11 Cilag Gmbh International Articulatable surgical instrument
US11717706B2 (en) 2009-07-15 2023-08-08 Cilag Gmbh International Ultrasonic surgical instruments
US11723716B2 (en) 2019-12-30 2023-08-15 Cilag Gmbh International Electrosurgical instrument with variable control mechanisms
US11744604B2 (en) 2017-12-28 2023-09-05 Cilag Gmbh International Surgical instrument with a hardware-only control circuit
US11751958B2 (en) 2017-12-28 2023-09-12 Cilag Gmbh International Surgical hub coordination of control and communication of operating room devices
US11759251B2 (en) 2019-12-30 2023-09-19 Cilag Gmbh International Control program adaptation based on device status and user input
US11771487B2 (en) 2017-12-28 2023-10-03 Cilag Gmbh International Mechanisms for controlling different electromechanical systems of an electrosurgical instrument
US11771511B2 (en) 2016-03-09 2023-10-03 Momentis Surgical Ltd Modular device comprising mechanical arms
US11775682B2 (en) 2017-12-28 2023-10-03 Cilag Gmbh International Data stripping method to interrogate patient records and create anonymized record
US11779387B2 (en) 2019-12-30 2023-10-10 Cilag Gmbh International Clamp arm jaw to minimize tissue sticking and improve tissue control
US11779410B2 (en) 2017-03-09 2023-10-10 Momentis Surgical Ltd Control console including an input arm for control of a surgical mechanical arm
US11779329B2 (en) 2019-12-30 2023-10-10 Cilag Gmbh International Surgical instrument comprising a flex circuit including a sensor system
US11779337B2 (en) 2017-12-28 2023-10-10 Cilag Gmbh International Method of using reinforced flexible circuits with multiple sensors to optimize performance of radio frequency devices
US11786251B2 (en) 2017-12-28 2023-10-17 Cilag Gmbh International Method for adaptive control schemes for surgical network control and interaction
US11786291B2 (en) 2019-12-30 2023-10-17 Cilag Gmbh International Deflectable support of RF energy electrode with respect to opposing ultrasonic blade
US11793537B2 (en) 2017-10-30 2023-10-24 Cilag Gmbh International Surgical instrument comprising an adaptive electrical system
US11801098B2 (en) 2017-10-30 2023-10-31 Cilag Gmbh International Method of hub communication with surgical instrument systems
US11812924B2 (en) * 2018-11-02 2023-11-14 Verb Surgical Inc. Surgical robotic system
US11812957B2 (en) 2019-12-30 2023-11-14 Cilag Gmbh International Surgical instrument comprising a signal interference resolution system
US11818052B2 (en) 2017-12-28 2023-11-14 Cilag Gmbh International Surgical network determination of prioritization of communication, interaction, or processing based on system or device needs
US11819299B2 (en) 2012-05-01 2023-11-21 Board Of Regents Of The University Of Nebraska Single site robotic device and related systems and methods
US11826032B2 (en) 2013-07-17 2023-11-28 Virtual Incision Corporation Robotic surgical devices, systems and related methods
US11826014B2 (en) 2016-05-18 2023-11-28 Virtual Incision Corporation Robotic surgical devices, systems and related methods
US11832902B2 (en) 2012-08-08 2023-12-05 Virtual Incision Corporation Robotic surgical devices, systems, and related methods
US11832899B2 (en) 2017-12-28 2023-12-05 Cilag Gmbh International Surgical systems with autonomously adjustable control programs
US11839396B2 (en) 2018-03-08 2023-12-12 Cilag Gmbh International Fine dissection mode for tissue classification
US11844579B2 (en) 2017-12-28 2023-12-19 Cilag Gmbh International Adjustments based on airborne particle properties
US11857152B2 (en) 2017-12-28 2024-01-02 Cilag Gmbh International Surgical hub spatial awareness to determine devices in operating theater
US11864728B2 (en) 2017-12-28 2024-01-09 Cilag Gmbh International Characterization of tissue irregularities through the use of mono-chromatic light refractivity
US11864820B2 (en) 2016-05-03 2024-01-09 Cilag Gmbh International Medical device with a bilateral jaw configuration for nerve stimulation
US11864845B2 (en) 2017-12-28 2024-01-09 Cilag Gmbh International Sterile field interactive control displays
US11871955B2 (en) 2012-06-29 2024-01-16 Cilag Gmbh International Surgical instruments with articulating shafts
US11871982B2 (en) 2009-10-09 2024-01-16 Cilag Gmbh International Surgical generator for ultrasonic and electrosurgical devices
US11872090B2 (en) 2015-08-03 2024-01-16 Virtual Incision Corporation Robotic surgical devices, systems, and related methods
US11871901B2 (en) 2012-05-20 2024-01-16 Cilag Gmbh International Method for situational awareness for surgical network or surgical network connected device capable of adjusting function based on a sensed situation or usage
US11890065B2 (en) 2017-12-28 2024-02-06 Cilag Gmbh International Surgical system to limit displacement
US11896322B2 (en) 2017-12-28 2024-02-13 Cilag Gmbh International Sensing the patient position and contact utilizing the mono-polar return pad electrode to provide situational awareness to the hub
US11896443B2 (en) 2017-12-28 2024-02-13 Cilag Gmbh International Control of a surgical system through a surgical barrier
US11903658B2 (en) 2019-01-07 2024-02-20 Virtual Incision Corporation Robotically assisted surgical system and related devices and methods
US11903587B2 (en) 2017-12-28 2024-02-20 Cilag Gmbh International Adjustment to the surgical stapling control based on situational awareness
US11911045B2 (en) 2017-10-30 2024-02-27 Cllag GmbH International Method for operating a powered articulating multi-clip applier
US11918302B2 (en) 2021-03-31 2024-03-05 Cilag Gmbh International Sterile field interactive control displays

Families Citing this family (42)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10092359B2 (en) 2010-10-11 2018-10-09 Ecole Polytechnique Federale De Lausanne Mechanical manipulator for surgical instruments
WO2013014621A2 (en) 2011-07-27 2013-01-31 Ecole Polytechnique Federale De Lausanne (Epfl) Mechanical teleoperated device for remote manipulation
DK2919948T3 (en) 2012-11-14 2017-11-27 European Space Agency (Esa) Hand control device
EP2939632B1 (en) * 2012-12-25 2021-10-13 Kawasaki Jukogyo Kabushiki Kaisha Surgical robot
CO6740182A1 (en) * 2013-02-28 2013-08-30 Univ Militar Nueva Granada Dynamic device with capacity to support the pelvic region
WO2016030767A1 (en) * 2014-08-27 2016-03-03 Distalmotion Sa Surgical system for microsurgical techniques
US10864049B2 (en) 2014-12-19 2020-12-15 Distalmotion Sa Docking system for mechanical telemanipulator
US10548680B2 (en) 2014-12-19 2020-02-04 Distalmotion Sa Articulated handle for mechanical telemanipulator
US10646294B2 (en) 2014-12-19 2020-05-12 Distalmotion Sa Reusable surgical instrument for minimally invasive procedures
US11039820B2 (en) 2014-12-19 2021-06-22 Distalmotion Sa Sterile interface for articulated surgical instruments
EP4289385A2 (en) 2014-12-19 2023-12-13 DistalMotion SA Surgical instrument with articulated end-effector
US11497668B1 (en) * 2015-02-24 2022-11-15 Baromedical Research Institute, Ltd. Resuscitation chamber
WO2016162751A1 (en) 2015-04-09 2016-10-13 Distalmotion Sa Articulated hand-held instrument
WO2016162752A1 (en) 2015-04-09 2016-10-13 Distalmotion Sa Mechanical teleoperated device for remote manipulation
CN108135361A (en) * 2015-07-01 2018-06-08 帕雷维迪有限责任公司 Motion control seat input equipment
US10786272B2 (en) 2015-08-28 2020-09-29 Distalmotion Sa Surgical instrument with increased actuation force
DE102015121017A1 (en) 2015-12-03 2017-06-08 Karl Storz Gmbh & Co. Kg Observation device, in particular medical observation device, with an operating unit and use of an input module
JP6673684B2 (en) * 2015-12-11 2020-03-25 株式会社メディカロイド Remote control device and remote operation system
US10272572B2 (en) * 2016-06-10 2019-04-30 The Boeing Company Remotely controlling robotic platforms based on multi-modal sensory data
US10023250B2 (en) 2016-06-10 2018-07-17 The Boeing Company Multi-tread vehicles and methods of operating thereof
US10571902B2 (en) * 2016-10-12 2020-02-25 Sisu Devices Llc Robotic programming and motion control
US10368955B2 (en) 2017-03-31 2019-08-06 Johnson & Johnson Innovation-Jjdc, Inc. Multi-functional foot pedal assembly for controlling a robotic surgical system
US10646394B2 (en) * 2017-04-21 2020-05-12 Johnson & Johnson Surgical Vision, Inc. Apparatus, system and method of providing a foot pedal platform for ophthalmic surgery
US11058503B2 (en) 2017-05-11 2021-07-13 Distalmotion Sa Translational instrument interface for surgical robot and surgical robot systems comprising the same
JP2018202032A (en) 2017-06-08 2018-12-27 株式会社メディカロイド Remote control apparatus for medical equipment
CN107443374A (en) * 2017-07-20 2017-12-08 深圳市易成自动驾驶技术有限公司 Manipulator control system and its control method, actuation means, storage medium
JP6757340B2 (en) * 2018-01-25 2020-09-16 株式会社メディカロイド Remote control device for medical equipment
JP6960505B2 (en) * 2018-01-25 2021-11-05 株式会社メディカロイド Remote control device
AU2019218707A1 (en) 2018-02-07 2020-08-13 Distalmotion Sa Surgical robot systems comprising robotic telemanipulators and integrated laparoscopy
US20190254753A1 (en) * 2018-02-19 2019-08-22 Globus Medical, Inc. Augmented reality navigation systems for use with robotic surgical systems and methods of their use
US11353967B2 (en) 2018-05-31 2022-06-07 Arkh Litho Holdings, LLC Interacting with a virtual environment using a pointing controller
US10895757B2 (en) * 2018-07-03 2021-01-19 Verb Surgical Inc. Systems and methods for three-dimensional visualization during robotic surgery
US10503199B1 (en) * 2018-07-17 2019-12-10 Verb Surgical Inc. Pedal with sliding and locking mechanisms for surgical robots
US10888383B2 (en) 2018-07-17 2021-01-12 Verb Surgical Inc. Robotic surgical pedal with integrated foot sensor
JP7248271B2 (en) * 2018-09-14 2023-03-29 国立大学法人 東京大学 Information processing device, robot hand control system, and robot hand control program
WO2020142049A2 (en) * 2018-12-31 2020-07-09 Havelsan Hava Elektronik Sanayi Ve Ticaret Anonim Sirketi Mixed-reality haptic simulation system with real device instrument interaction and serial or parallel manipulator
WO2021230834A1 (en) * 2019-12-31 2021-11-18 Havelsan Hava Elektronik Sanayi Ve Ticaret Anonim Sirketi Tool-independent 3-dimensional surface haptic interface
US11755111B2 (en) 2020-03-16 2023-09-12 Arkh, Inc. Spatially aware computing hub and environment
JP7317917B2 (en) * 2020-06-10 2023-07-31 株式会社メディカロイド Remote control device
CN112022357B (en) * 2020-09-16 2022-08-05 上海微创医疗机器人(集团)股份有限公司 Doctor console, surgical robot system, and control method for doctor console
USD1009861S1 (en) 2020-09-25 2024-01-02 Arkh, Inc. Smart ring
US11844585B1 (en) 2023-02-10 2023-12-19 Distalmotion Sa Surgical robotics systems and devices having a sterile restart, and methods thereof

Citations (21)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5163949A (en) * 1990-03-02 1992-11-17 Bonutti Peter M Fluid operated retractors
US5373854A (en) * 1993-07-15 1994-12-20 Kolozsi; William Z. Biopsy apparatus for use in endoscopy
US5514153A (en) * 1990-03-02 1996-05-07 General Surgical Innovations, Inc. Method of dissecting tissue layers
US5776126A (en) * 1993-09-23 1998-07-07 Wilk; Peter J. Laparoscopic surgical apparatus and associated method
US6179776B1 (en) * 1999-03-12 2001-01-30 Scimed Life Systems, Inc. Controllable endoscopic sheath apparatus and related method of use
US6203559B1 (en) * 1998-10-05 2001-03-20 Origin Medsystems Method and apparatus for tissue dissection
US6352503B1 (en) * 1998-07-17 2002-03-05 Olympus Optical Co., Ltd. Endoscopic surgery apparatus
US20020128633A1 (en) * 1998-02-24 2002-09-12 Brock David L. Surgical instrument
US6451042B1 (en) * 1990-03-02 2002-09-17 General Surgical Innovations, Inc. Method and apparatus for dissecting tissue layers
US20030013949A1 (en) * 1998-11-20 2003-01-16 Frederic H. Moll Cooperative minimally invasive telesurgical system
US20050001838A1 (en) * 2003-04-28 2005-01-06 Pedro Gregorio Systems and methods for user interfaces designed for rotary input devices
US20050085691A1 (en) * 2003-10-16 2005-04-21 Nakao Naomi L. Endoscope having multiple working segments
US20050137460A1 (en) * 2003-11-12 2005-06-23 Bertolero Arthur A. Retractor with inflatable blades
US20050206583A1 (en) * 1996-10-02 2005-09-22 Lemelson Jerome H Selectively controllable heads-up display system
US20060052684A1 (en) * 2002-05-07 2006-03-09 Takashi Takahashi Medical cockpit system
US20080058590A1 (en) * 2006-09-01 2008-03-06 Nidus Medical, Llc. Tissue visualization device having multi-segmented frame
US20080167662A1 (en) * 2007-01-08 2008-07-10 Kurtz Anthony D Tactile feel apparatus for use with robotic operations
US20090000626A1 (en) * 2002-03-06 2009-01-01 Mako Surgical Corp. Haptic guidance system and method
US20090085878A1 (en) * 2007-09-28 2009-04-02 Immersion Corporation Multi-Touch Device Having Dynamic Haptic Effects
US20090245600A1 (en) * 2008-03-28 2009-10-01 Intuitive Surgical, Inc. Automated panning and digital zooming for robotic surgical systems
US20100073150A1 (en) * 2008-09-24 2010-03-25 Olson Eric S Robotic catheter system including haptic feedback

Family Cites Families (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2002078772A (en) * 2000-09-06 2002-03-19 Sukeyuki Kawaguchi Remote computer diagnostic image method and laser acupuncture therapeutic instrument using muscle tonus or muscle potential sensor
US7864161B2 (en) * 2004-06-17 2011-01-04 Adrea, LLC Use of a two finger input on touch screens
US20110046659A1 (en) * 2007-07-09 2011-02-24 Immersion Corporation Minimally Invasive Surgical Tools With Haptic Feedback
US20110238079A1 (en) 2010-03-18 2011-09-29 SPI Surgical, Inc. Surgical Cockpit Comprising Multisensory and Multimodal Interfaces for Robotic Surgery and Methods Related Thereto

Patent Citations (21)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5514153A (en) * 1990-03-02 1996-05-07 General Surgical Innovations, Inc. Method of dissecting tissue layers
US5163949A (en) * 1990-03-02 1992-11-17 Bonutti Peter M Fluid operated retractors
US6451042B1 (en) * 1990-03-02 2002-09-17 General Surgical Innovations, Inc. Method and apparatus for dissecting tissue layers
US5373854A (en) * 1993-07-15 1994-12-20 Kolozsi; William Z. Biopsy apparatus for use in endoscopy
US5776126A (en) * 1993-09-23 1998-07-07 Wilk; Peter J. Laparoscopic surgical apparatus and associated method
US20050206583A1 (en) * 1996-10-02 2005-09-22 Lemelson Jerome H Selectively controllable heads-up display system
US20020128633A1 (en) * 1998-02-24 2002-09-12 Brock David L. Surgical instrument
US6352503B1 (en) * 1998-07-17 2002-03-05 Olympus Optical Co., Ltd. Endoscopic surgery apparatus
US6203559B1 (en) * 1998-10-05 2001-03-20 Origin Medsystems Method and apparatus for tissue dissection
US20030013949A1 (en) * 1998-11-20 2003-01-16 Frederic H. Moll Cooperative minimally invasive telesurgical system
US6179776B1 (en) * 1999-03-12 2001-01-30 Scimed Life Systems, Inc. Controllable endoscopic sheath apparatus and related method of use
US20090000626A1 (en) * 2002-03-06 2009-01-01 Mako Surgical Corp. Haptic guidance system and method
US20060052684A1 (en) * 2002-05-07 2006-03-09 Takashi Takahashi Medical cockpit system
US20050001838A1 (en) * 2003-04-28 2005-01-06 Pedro Gregorio Systems and methods for user interfaces designed for rotary input devices
US20050085691A1 (en) * 2003-10-16 2005-04-21 Nakao Naomi L. Endoscope having multiple working segments
US20050137460A1 (en) * 2003-11-12 2005-06-23 Bertolero Arthur A. Retractor with inflatable blades
US20080058590A1 (en) * 2006-09-01 2008-03-06 Nidus Medical, Llc. Tissue visualization device having multi-segmented frame
US20080167662A1 (en) * 2007-01-08 2008-07-10 Kurtz Anthony D Tactile feel apparatus for use with robotic operations
US20090085878A1 (en) * 2007-09-28 2009-04-02 Immersion Corporation Multi-Touch Device Having Dynamic Haptic Effects
US20090245600A1 (en) * 2008-03-28 2009-10-01 Intuitive Surgical, Inc. Automated panning and digital zooming for robotic surgical systems
US20100073150A1 (en) * 2008-09-24 2010-03-25 Olson Eric S Robotic catheter system including haptic feedback

Cited By (187)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9375288B2 (en) * 2009-03-09 2016-06-28 Intuitive Surgical Operations, Inc. Adaptable integrated energy control system for electrosurgical tools in robotic surgical systems
US20100228264A1 (en) * 2009-03-09 2010-09-09 David Robinson Adaptable integrated energy control system for electrosurgical tools in robotic surgical systems
US20150173849A1 (en) * 2009-03-09 2015-06-25 Intuitive Surgical Operations, Inc. Adaptable integrated energy control system for electrosurgical tools in robotic surgical systems
US9827059B2 (en) * 2009-03-09 2017-11-28 Intuitive Surgical Operations, Inc. Adaptable integrated energy control system for electrosurgical tools in robotic surgical systems
US20160338786A1 (en) * 2009-03-09 2016-11-24 Intuitive Surgical Operations, Inc. Adaptable integrated energy control system for electrosurgical tools in robotic surgical systems
US20130231681A1 (en) * 2009-03-09 2013-09-05 Intuitive Surgical Operations, Inc. Adaptable integrated energy control system for electrosurgical tools in robotic surgical systems
US10575909B2 (en) 2009-03-09 2020-03-03 Intuitive Surgical Operations, Inc. Adaptable integrated energy control system for electrosurgical tools in robotic surgical systems
US8862268B2 (en) * 2009-03-09 2014-10-14 Intuitive Surgical Operations, Inc. Adaptable integrated energy control system for electrosurgical tools in robotic surgical systems
US8423182B2 (en) * 2009-03-09 2013-04-16 Intuitive Surgical Operations, Inc. Adaptable integrated energy control system for electrosurgical tools in robotic surgical systems
US10898287B2 (en) 2009-03-09 2021-01-26 Intuitive Surgical Operations, Inc. Adaptable integrated energy control system for electrosurgical tools in robotic surgical systems
US11717706B2 (en) 2009-07-15 2023-08-08 Cilag Gmbh International Ultrasonic surgical instruments
US11871982B2 (en) 2009-10-09 2024-01-16 Cilag Gmbh International Surgical generator for ultrasonic and electrosurgical devices
US9474580B2 (en) 2010-03-18 2016-10-25 SPI Surgical, Inc. Surgical cockpit comprising multisensory and multimodal interfaces for robotic surgery and methods related thereto
US20110234484A1 (en) * 2010-03-29 2011-09-29 Olympus Corporation Operation input unit and manipulator system
US8681095B2 (en) * 2010-03-29 2014-03-25 Olympus Corporation Operation input unit and manipulator system
US20120010749A1 (en) * 2010-04-09 2012-01-12 Deka Products Limited Partnership System and apparatus for robotic device and methods of using thereof
US10888439B2 (en) * 2010-04-09 2021-01-12 Deka Products Limited Partnership System and apparatus for robotic device and methods of using thereof
US10646355B2 (en) * 2010-04-09 2020-05-12 Deka Products Limited Partnership System and apparatus for robotic device and methods of using thereof
US20190175362A1 (en) * 2010-04-09 2019-06-13 Deka Products Limited Partnership System and Apparatus for Robotic Device and Methods of Using Thereof
US10201435B2 (en) * 2010-04-09 2019-02-12 Deka Products Limited Partnership System and apparatus for robotic device and methods of using thereof
US11628072B2 (en) * 2010-04-09 2023-04-18 Deka Products Limited Partnership System and apparatus for robotic device and methods of using thereof
US20210128322A1 (en) * 2010-04-09 2021-05-06 Deka Products Limited Partnership System and apparatus for robotic device and methods of using thereof
US9844447B2 (en) * 2010-04-09 2017-12-19 Deka Products Limited Partnership System and apparatus for robotic device and methods of using thereof
US20120130739A1 (en) * 2010-11-21 2012-05-24 David Crane Unsupervised Telemedical Office for Remote &/or Autonomous & Automated Medical Care of Patients
US20140204079A1 (en) * 2011-06-17 2014-07-24 Immersion System for colocating a touch screen and a virtual object, and device for manipulating virtual objects implementing such a system
US9786090B2 (en) * 2011-06-17 2017-10-10 INRIA—Institut National de Recherche en Informatique et en Automatique System for colocating a touch screen and a virtual object, and device for manipulating virtual objects implementing such a system
US11595242B2 (en) 2011-07-11 2023-02-28 Board Of Regents Of The University Of Nebraska Robotic surgical devices, systems and related methods
US11032125B2 (en) * 2011-07-11 2021-06-08 Board Of Regents Of The University Of Nebraska Robotic surgical devices, systems and related methods
US11909576B2 (en) 2011-07-11 2024-02-20 Board Of Regents Of The University Of Nebraska Robotic surgical devices, systems, and related methods
US20140148818A1 (en) * 2011-08-04 2014-05-29 Olympus Corporation Surgical assistant system
US9671860B2 (en) 2011-08-04 2017-06-06 Olympus Corporation Manipulation input device and manipulator system having the same
US9244524B2 (en) 2011-08-04 2016-01-26 Olympus Corporation Surgical instrument and control method thereof
US9218053B2 (en) * 2011-08-04 2015-12-22 Olympus Corporation Surgical assistant system
US9632573B2 (en) 2011-08-04 2017-04-25 Olympus Corporation Medical manipulator and method of controlling the same
US9477301B2 (en) 2011-08-04 2016-10-25 Olympus Corporation Operation support device and assembly method thereof
US9161772B2 (en) 2011-08-04 2015-10-20 Olympus Corporation Surgical instrument and medical manipulator
US9423869B2 (en) 2011-08-04 2016-08-23 Olympus Corporation Operation support device
US9519341B2 (en) 2011-08-04 2016-12-13 Olympus Corporation Medical manipulator and surgical support apparatus
US9524022B2 (en) 2011-08-04 2016-12-20 Olympus Corporation Medical equipment
US9244523B2 (en) 2011-08-04 2016-01-26 Olympus Corporation Manipulator system
US9851782B2 (en) 2011-08-04 2017-12-26 Olympus Corporation Operation support device and attachment and detachment method thereof
US9568992B2 (en) 2011-08-04 2017-02-14 Olympus Corporation Medical manipulator
US9632577B2 (en) 2011-08-04 2017-04-25 Olympus Corporation Operation support device and control method thereof
US11819299B2 (en) 2012-05-01 2023-11-21 Board Of Regents Of The University Of Nebraska Single site robotic device and related systems and methods
US11871901B2 (en) 2012-05-20 2024-01-16 Cilag Gmbh International Method for situational awareness for surgical network or surgical network connected device capable of adjusting function based on a sensed situation or usage
US11871955B2 (en) 2012-06-29 2024-01-16 Cilag Gmbh International Surgical instruments with articulating shafts
US20140018819A1 (en) * 2012-07-16 2014-01-16 Anil K Raj Anthro-Centric Multisensory Interface for Sensory Augmentation of Telesurgery
US11832902B2 (en) 2012-08-08 2023-12-05 Virtual Incision Corporation Robotic surgical devices, systems, and related methods
US10624706B2 (en) 2012-09-17 2020-04-21 Intuitive Surgical Operations, Inc. Methods and systems for assigning input devices to teleoperated surgical instrument functions
EP4082468A1 (en) * 2012-09-17 2022-11-02 Intuitive Surgical Operations, Inc. Methods and systems for assigning input devices to teleoperated surgical instrument functions
US9301811B2 (en) 2012-09-17 2016-04-05 Intuitive Surgical Operations, Inc. Methods and systems for assigning input devices to teleoperated surgical instrument functions
US9814536B2 (en) 2012-09-17 2017-11-14 Intuitive Surgical Operations, Inc. Methods and systems for assigning input devices to teleoperated surgical instrument functions
US11160622B2 (en) 2012-09-17 2021-11-02 Intuitive Surgical Operations, Inc. Methods and systems for assigning input devices to teleoperated surgical instrument functions
WO2014047402A1 (en) * 2012-09-20 2014-03-27 MUSC Foundation for Research and Development Head-mounted systems and methods for providing inspection, evaluation or assessment of an event or location
US9819843B2 (en) 2012-09-20 2017-11-14 Zeriscope Inc. Head-mounted systems and methods for providing inspection, evaluation or assessment of an event or location
CN105264459A (en) * 2012-09-27 2016-01-20 沉浸式触感有限公司 Haptic augmented and virtual reality system for simulation of surgical procedures
US9563266B2 (en) 2012-09-27 2017-02-07 Immersivetouch, Inc. Haptic augmented and virtual reality system for simulation of surgical procedures
WO2014052158A3 (en) * 2012-09-27 2015-07-16 Immersive Touch, Inc. Haptic augmented reality for simulation of surgical procedure
US10108266B2 (en) 2012-09-27 2018-10-23 The Board Of Trustees Of The University Of Illinois Haptic augmented and virtual reality system for simulation of surgical procedures
US10437339B2 (en) 2012-09-27 2019-10-08 The Board Of Trustees Of The University Of Illinois Haptic augmented and virtual reality system for simulation of surgical procedures
US10864048B2 (en) 2012-11-02 2020-12-15 Intuitive Surgical Operations, Inc. Flux disambiguation for teleoperated surgical systems
US10631939B2 (en) 2012-11-02 2020-04-28 Intuitive Surgical Operations, Inc. Systems and methods for mapping flux supply paths
US20150293362A1 (en) * 2012-11-13 2015-10-15 Sony Corporation Image display apparatus, image display method, mobile apparatus, image display system, and computer program
US10108018B2 (en) * 2012-11-13 2018-10-23 Sony Corporation Image display apparatus for displaying an image captured by a mobile apparatus
US10215989B2 (en) 2012-12-19 2019-02-26 Lockheed Martin Corporation System, method and computer program product for real-time alignment of an augmented reality device
US9390630B2 (en) * 2013-05-03 2016-07-12 John James Daniels Accelerated learning, entertainment and cognitive therapy using augmented reality comprising combined haptic, auditory, and visual stimulation
US20150317910A1 (en) * 2013-05-03 2015-11-05 John James Daniels Accelerated Learning, Entertainment and Cognitive Therapy Using Augmented Reality Comprising Combined Haptic, Auditory, and Visual Stimulation
US11826032B2 (en) 2013-07-17 2023-11-28 Virtual Incision Corporation Robotic surgical devices, systems and related methods
WO2015023513A1 (en) * 2013-08-14 2015-02-19 Intuitive Surgical Operations, Inc. Endoscope control system
US10265057B2 (en) 2013-08-14 2019-04-23 Intuitive Surgical Operations, Inc. Endoscope control system
US10925586B2 (en) 2013-08-14 2021-02-23 Intuitive Surgical Operations, Inc Endoscope control system
WO2015084837A1 (en) * 2013-12-02 2015-06-11 Immersive Touch, Inc. Improvements for haptic augmented and virtual reality system for simulation of surgical procedures
US20150199106A1 (en) * 2014-01-14 2015-07-16 Caterpillar Inc. Augmented Reality Display System
JP2015150425A (en) * 2014-02-18 2015-08-24 三星電子株式会社Samsung Electronics Co.,Ltd. Master devices for surgical robots and control methods thereof
US9655680B2 (en) * 2014-02-18 2017-05-23 Samsung Electronics Co., Ltd. Master devices for surgical robots and control methods thereof
US20150230869A1 (en) * 2014-02-18 2015-08-20 Samsung Electronics Co., Ltd. Master devices for surgical robots and control methods thereof
CN104840253A (en) * 2014-02-18 2015-08-19 三星电子株式会社 Master devices for surgical robots and control methods thereof
US10166061B2 (en) 2014-03-17 2019-01-01 Intuitive Surgical Operations, Inc. Teleoperated surgical system equipment with user interface
US11439454B2 (en) 2014-03-17 2022-09-13 Intuitive Surgical Operations, Inc. Teleoperated surgical system equipment with user interface
US20160320862A1 (en) * 2014-05-01 2016-11-03 Aaron Schradin Motion control seat input device
US11488381B2 (en) 2014-05-15 2022-11-01 Fenwal, Inc. Medical device with camera for imaging disposable
US11036985B2 (en) * 2014-05-15 2021-06-15 Fenwal, Inc. Head mounted display device for use in a medical facility
US11837360B2 (en) 2014-05-15 2023-12-05 Fenwal, Inc. Head-mounted display device for use in a medical facility
US11436829B2 (en) 2014-05-15 2022-09-06 Fenwal, Inc. Head-mounted display device for use in a medical facility
US11116589B2 (en) * 2014-09-04 2021-09-14 Memic Innovative Surgery Ltd. Control of device including mechanical arms
US11517378B2 (en) 2014-09-04 2022-12-06 Momentis Surgical Ltd Device and system including mechanical arms
WO2016077531A1 (en) * 2014-11-13 2016-05-19 Intuitive Surgical Operations, Inc. Integrated user environments
US9298884B1 (en) * 2014-12-17 2016-03-29 Vitaax Llc Remote instruction and monitoring of health care
US10437335B2 (en) 2015-04-14 2019-10-08 John James Daniels Wearable electronic, multi-sensory, human/machine, human/human interfaces
US11872090B2 (en) 2015-08-03 2024-01-16 Virtual Incision Corporation Robotic surgical devices, systems, and related methods
US20170036684A1 (en) * 2015-08-06 2017-02-09 Progress Rail Services Corporation Cabin monitoring system for train locomotive
US10792114B2 (en) * 2015-08-25 2020-10-06 Kawasaki Jukogyo Kabushiki Kaisha Remote control robot system and method of operating the same
US20180243910A1 (en) * 2015-08-25 2018-08-30 Kawasaki Jukogyo Kabushiki Kaisha Remote control robot system and method of operating the same
USD789105S1 (en) 2015-09-18 2017-06-13 Sv Tool Corporation Ergonomic chair
US11766287B2 (en) 2015-09-30 2023-09-26 Cilag Gmbh International Methods for operating generator for digitally generating electrical signal waveforms and surgical instruments
US11559347B2 (en) 2015-09-30 2023-01-24 Cilag Gmbh International Techniques for circuit topologies for combined generator
US11666375B2 (en) 2015-10-16 2023-06-06 Cilag Gmbh International Electrode wiping surgical device
US10366489B2 (en) * 2015-12-23 2019-07-30 Siemens Healthcare Gmbh Method and system for outputting augmented reality information
US11694328B2 (en) 2015-12-23 2023-07-04 Siemens Healthcare Gmbh Method and system for outputting augmented reality information
US20170186157A1 (en) * 2015-12-23 2017-06-29 Siemens Healthcare Gmbh Method and system for outputting augmented reality information
CN106909771A (en) * 2015-12-23 2017-06-30 西门子医疗有限公司 Method and system for exporting augmented reality information
US10846851B2 (en) 2015-12-23 2020-11-24 Siemens Healthcare Gmbh Method and system for outputting augmented reality information
US11896280B2 (en) 2016-01-15 2024-02-13 Cilag Gmbh International Clamp arm comprising a circuit
US11684402B2 (en) 2016-01-15 2023-06-27 Cilag Gmbh International Modular battery powered handheld surgical instrument with selective application of energy based on tissue characterization
US11751929B2 (en) 2016-01-15 2023-09-12 Cilag Gmbh International Modular battery powered handheld surgical instrument with selective application of energy based on tissue characterization
US11771511B2 (en) 2016-03-09 2023-10-03 Momentis Surgical Ltd Modular device comprising mechanical arms
US20190086998A1 (en) * 2016-03-11 2019-03-21 Limbic Life Ag Occupant support device and system for controlling objects
US11893147B2 (en) * 2016-03-11 2024-02-06 Limbic Life Ag Occupant support device and system for controlling objects
US11386372B2 (en) 2016-03-29 2022-07-12 Locatee Ag Device, system and method for monitoring usage of functional facilities
US11080634B2 (en) 2016-03-29 2021-08-03 Locatee Ag Device, system and method for monitoring usage of functional facilities
EP3434219A4 (en) * 2016-04-28 2019-04-10 Sony Corporation Control device, control method, program, and sound output system
US11864820B2 (en) 2016-05-03 2024-01-09 Cilag Gmbh International Medical device with a bilateral jaw configuration for nerve stimulation
US11826014B2 (en) 2016-05-18 2023-11-28 Virtual Incision Corporation Robotic surgical devices, systems and related methods
US20180157317A1 (en) * 2016-08-18 2018-06-07 Technische Universität Dresden System and method for haptic interaction with virtual objects
US10521010B2 (en) * 2016-08-18 2019-12-31 Technische Universitaet Dresden System and method for haptic interaction with virtual objects
US11446097B2 (en) 2016-09-21 2022-09-20 Verb Surgical Inc. User console system for robotic surgery
US10568703B2 (en) * 2016-09-21 2020-02-25 Verb Surgical Inc. User arm support for use in a robotic surgical system
US20180098813A1 (en) * 2016-10-07 2018-04-12 Simbionix Ltd. Method and system for rendering a medical simulation in an operating room in virtual reality or augmented reality environment
US10492873B2 (en) * 2016-10-25 2019-12-03 Novartis Ag Medical spatial orientation system
US11229787B2 (en) 2016-11-25 2022-01-25 Kinaptic, LLC Haptic human machine interface and wearable electronics methods and apparatus
US11779410B2 (en) 2017-03-09 2023-10-10 Momentis Surgical Ltd Control console including an input arm for control of a surgical mechanical arm
US11083528B2 (en) 2017-03-09 2021-08-10 Memic Innovative Surgery Ltd. Input arm for control of a surgical mechanical arm
CN107049523A (en) * 2017-05-08 2017-08-18 成都中科博恩思医学机器人有限公司 Operational control component and surgical operation robot system
US10874469B2 (en) 2017-05-22 2020-12-29 Tsinghua University Remotely operated orthopedic surgical robot system for fracture reduction with visual-servo control method
US11589933B2 (en) * 2017-06-29 2023-02-28 Ix Innovation Llc Guiding a robotic surgical system to perform a surgical procedure
US20230225810A1 (en) * 2017-06-29 2023-07-20 Ix Innovation Llc Guiding a robotic surgical system to perform a surgical procedure
CN107334550A (en) * 2017-07-31 2017-11-10 成都中科博恩思医学机器人有限公司 Operating robot surgeon console slidably helps elbow platform
US11717363B2 (en) * 2017-09-08 2023-08-08 Covidien Lp High precision instrument control mode for robotic surgical systems
US20200281674A1 (en) * 2017-09-08 2020-09-10 Covidien Lp High precision instrument control mode for robotic surgical systems
US11911045B2 (en) 2017-10-30 2024-02-27 Cllag GmbH International Method for operating a powered articulating multi-clip applier
US11793537B2 (en) 2017-10-30 2023-10-24 Cilag Gmbh International Surgical instrument comprising an adaptive electrical system
US11801098B2 (en) 2017-10-30 2023-10-31 Cilag Gmbh International Method of hub communication with surgical instrument systems
US11819231B2 (en) 2017-10-30 2023-11-21 Cilag Gmbh International Adaptive control programs for a surgical system comprising more than one type of cartridge
US11751958B2 (en) 2017-12-28 2023-09-12 Cilag Gmbh International Surgical hub coordination of control and communication of operating room devices
US11890065B2 (en) 2017-12-28 2024-02-06 Cilag Gmbh International Surgical system to limit displacement
US11844579B2 (en) 2017-12-28 2023-12-19 Cilag Gmbh International Adjustments based on airborne particle properties
US11744604B2 (en) 2017-12-28 2023-09-05 Cilag Gmbh International Surgical instrument with a hardware-only control circuit
US20190201146A1 (en) * 2017-12-28 2019-07-04 Ethicon Llc Safety systems for smart powered surgical stapling
US11857152B2 (en) 2017-12-28 2024-01-02 Cilag Gmbh International Surgical hub spatial awareness to determine devices in operating theater
US11903587B2 (en) 2017-12-28 2024-02-20 Cilag Gmbh International Adjustment to the surgical stapling control based on situational awareness
US11818052B2 (en) 2017-12-28 2023-11-14 Cilag Gmbh International Surgical network determination of prioritization of communication, interaction, or processing based on system or device needs
US11771487B2 (en) 2017-12-28 2023-10-03 Cilag Gmbh International Mechanisms for controlling different electromechanical systems of an electrosurgical instrument
US11896443B2 (en) 2017-12-28 2024-02-13 Cilag Gmbh International Control of a surgical system through a surgical barrier
US11775682B2 (en) 2017-12-28 2023-10-03 Cilag Gmbh International Data stripping method to interrogate patient records and create anonymized record
US11864728B2 (en) 2017-12-28 2024-01-09 Cilag Gmbh International Characterization of tissue irregularities through the use of mono-chromatic light refractivity
US11896322B2 (en) 2017-12-28 2024-02-13 Cilag Gmbh International Sensing the patient position and contact utilizing the mono-polar return pad electrode to provide situational awareness to the hub
US11864845B2 (en) 2017-12-28 2024-01-09 Cilag Gmbh International Sterile field interactive control displays
US11779337B2 (en) 2017-12-28 2023-10-10 Cilag Gmbh International Method of using reinforced flexible circuits with multiple sensors to optimize performance of radio frequency devices
US11832899B2 (en) 2017-12-28 2023-12-05 Cilag Gmbh International Surgical systems with autonomously adjustable control programs
US11786251B2 (en) 2017-12-28 2023-10-17 Cilag Gmbh International Method for adaptive control schemes for surgical network control and interaction
US20210074173A1 (en) * 2018-02-02 2021-03-11 Access Virtual, LLC Virtual reality based pilot training system
US10878714B2 (en) * 2018-02-02 2020-12-29 Access Virtual, LLC Virtual reality based pilot training system
US20190244537A1 (en) * 2018-02-02 2019-08-08 Access Virtual, LLC Virtual reality based pilot training system
US11830382B2 (en) * 2018-02-02 2023-11-28 Access Virtual, LLC Virtual reality based pilot training system
US11844545B2 (en) 2018-03-08 2023-12-19 Cilag Gmbh International Calcified vessel identification
US11839396B2 (en) 2018-03-08 2023-12-12 Cilag Gmbh International Fine dissection mode for tissue classification
US11633087B2 (en) * 2018-08-07 2023-04-25 The Chinese University Of Hong Kong Endoscope manipulator and method for controlling the same
US11925373B2 (en) 2018-08-24 2024-03-12 Cilag Gmbh International Surgical suturing instrument comprising a non-circular needle
US11812924B2 (en) * 2018-11-02 2023-11-14 Verb Surgical Inc. Surgical robotic system
US11903658B2 (en) 2019-01-07 2024-02-20 Virtual Incision Corporation Robotically assisted surgical system and related devices and methods
JP2020144803A (en) * 2019-03-08 2020-09-10 株式会社フジ医療器 Controller chair
JP7189811B2 (en) 2019-03-08 2022-12-14 株式会社フジ医療器 controller chair
DE102019127631A1 (en) * 2019-10-14 2021-04-15 Bayerische Motoren Werke Aktiengesellschaft Display system for a vehicle
US11759251B2 (en) 2019-12-30 2023-09-19 Cilag Gmbh International Control program adaptation based on device status and user input
US11696776B2 (en) 2019-12-30 2023-07-11 Cilag Gmbh International Articulatable surgical instrument
US11786291B2 (en) 2019-12-30 2023-10-17 Cilag Gmbh International Deflectable support of RF energy electrode with respect to opposing ultrasonic blade
US11589916B2 (en) 2019-12-30 2023-02-28 Cilag Gmbh International Electrosurgical instruments with electrodes having variable energy densities
US11707318B2 (en) 2019-12-30 2023-07-25 Cilag Gmbh International Surgical instrument with jaw alignment features
US11812957B2 (en) 2019-12-30 2023-11-14 Cilag Gmbh International Surgical instrument comprising a signal interference resolution system
US11779387B2 (en) 2019-12-30 2023-10-10 Cilag Gmbh International Clamp arm jaw to minimize tissue sticking and improve tissue control
US11660089B2 (en) 2019-12-30 2023-05-30 Cilag Gmbh International Surgical instrument comprising a sensing system
US11786294B2 (en) 2019-12-30 2023-10-17 Cilag Gmbh International Control program for modular combination energy device
US11723716B2 (en) 2019-12-30 2023-08-15 Cilag Gmbh International Electrosurgical instrument with variable control mechanisms
US11744636B2 (en) 2019-12-30 2023-09-05 Cilag Gmbh International Electrosurgical systems with integrated and external power sources
US11684412B2 (en) 2019-12-30 2023-06-27 Cilag Gmbh International Surgical instrument with rotatable and articulatable surgical end effector
US11779329B2 (en) 2019-12-30 2023-10-10 Cilag Gmbh International Surgical instrument comprising a flex circuit including a sensor system
US11651706B2 (en) 2020-02-14 2023-05-16 Simbionix Ltd. Airway management virtual reality training
US11468793B2 (en) 2020-02-14 2022-10-11 Simbionix Ltd. Airway management virtual reality training
GB2593473A (en) * 2020-03-23 2021-09-29 Cmr Surgical Ltd Virtual console for controlling a surgical robot
US20210370506A1 (en) * 2020-05-29 2021-12-02 Honda Motor Co., Ltd. Database construction for control of robotic manipulator
US11642784B2 (en) * 2020-05-29 2023-05-09 Honda Motor Co., Ltd. Database construction for control of robotic manipulator
WO2021250580A1 (en) * 2020-06-10 2021-12-16 Mazor Robotics Ltd. Multi-arm robotic system enabling multiportal endoscopic surgery
CN112168354A (en) * 2020-10-14 2021-01-05 北京科迈启元科技有限公司 Waterproof and lightweight surgical robot actuator and surgical robot system
US11918302B2 (en) 2021-03-31 2024-03-05 Cilag Gmbh International Sterile field interactive control displays
US20220398936A1 (en) * 2021-06-15 2022-12-15 Richard Parker Aircraft training aid systems and processes
WO2023046185A1 (en) * 2021-09-26 2023-03-30 武汉联影智融医疗科技有限公司 Master control station for puncture operation, and puncture robot
US11925350B2 (en) 2021-11-04 2024-03-12 Cilag Gmbh International Method for providing an authentication lockout in a surgical stapler with a replaceable cartridge

Also Published As

Publication number Publication date
WO2011116332A2 (en) 2011-09-22
US9474580B2 (en) 2016-10-25
WO2011116332A3 (en) 2012-04-19
US20150025547A1 (en) 2015-01-22

Similar Documents

Publication Publication Date Title
US9474580B2 (en) Surgical cockpit comprising multisensory and multimodal interfaces for robotic surgery and methods related thereto
JP7175943B2 (en) Immersive 3D viewing for robotic surgery
US9867671B2 (en) Multi-user medical robotic system for collaboration or training in minimally invasive surgical procedures
EP2132007B1 (en) Systems for surgical visualization and device manipulation
JP6373440B2 (en) Patient side surgeon interface for minimally invasive teleoperated surgical instruments
Simorov et al. Review of surgical robotics user interface: what is the best way to control robotic surgery?
EP3620128B1 (en) Multi-port surgical robotic system architecture
US6788999B2 (en) Surgical system
Hills et al. Telepresence technology in medicine: principles and applications
US20230157525A1 (en) System and method for reversing orientation and view of selected components of a miniaturized surgical robotic unit in vivo
Urban et al. Robot‐assisted surgery system with kinesthetic feedback
Rassweiler et al. Robotic-Assisted Surgery: Low-Cost Options
Jensen et al. Development of a Telepresence Surgery System

Legal Events

Date Code Title Description
AS Assignment

Owner name: SPI SURGICAL, INC., WASHINGTON

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:HANNAFORD, BLAKE;KIM, LOUIS;LENDVAY, THOMAS S.;AND OTHERS;SIGNING DATES FROM 20110401 TO 20110504;REEL/FRAME:027228/0186

STCB Information on status: application discontinuation

Free format text: ABANDONED -- FAILURE TO RESPOND TO AN OFFICE ACTION