EP4178473A1 - Système comprenant une matrice de caméras déployables hors d'un canal d'un dispositif chirurgical pénétrant un tissu - Google Patents

Système comprenant une matrice de caméras déployables hors d'un canal d'un dispositif chirurgical pénétrant un tissu

Info

Publication number
EP4178473A1
EP4178473A1 EP22718802.6A EP22718802A EP4178473A1 EP 4178473 A1 EP4178473 A1 EP 4178473A1 EP 22718802 A EP22718802 A EP 22718802A EP 4178473 A1 EP4178473 A1 EP 4178473A1
Authority
EP
European Patent Office
Prior art keywords
surgical
camera array
hub
display
camera
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
EP22718802.6A
Other languages
German (de)
English (en)
Inventor
Iv Frederick E. Shelton
Matthew D. COWPERTHWAIT
Kevin M. Fiebig
Jason L. Harris
Cory G. Kimball
Monica L.Z. Rivard
Leonardo N. Rossoni
Risto KOJCEV
Felix J. BORK
Demetrius N. HARRIS
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Cilag GmbH International
Original Assignee
Cilag GmbH International
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority claimed from US17/688,660 external-priority patent/US20220331052A1/en
Application filed by Cilag GmbH International filed Critical Cilag GmbH International
Publication of EP4178473A1 publication Critical patent/EP4178473A1/fr
Pending legal-status Critical Current

Links

Classifications

    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B90/00—Instruments, implements or accessories specially adapted for surgery or diagnosis and not covered by any of the groups A61B1/00 - A61B50/00, e.g. for luxation treatment or for protecting wound edges
    • A61B90/36—Image-producing devices or illumination devices not otherwise provided for
    • A61B90/361—Image-producing devices, e.g. surgical cameras
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B17/00—Surgical instruments, devices or methods, e.g. tourniquets
    • A61B17/34—Trocars; Puncturing needles
    • A61B17/3403—Needle locating or guiding means
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B17/00—Surgical instruments, devices or methods, e.g. tourniquets
    • A61B17/34—Trocars; Puncturing needles
    • A61B17/3417—Details of tips or shafts, e.g. grooves, expandable, bendable; Multiple coaxial sliding cannulas, e.g. for dilating
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B17/00—Surgical instruments, devices or methods, e.g. tourniquets
    • A61B17/34—Trocars; Puncturing needles
    • A61B17/3417—Details of tips or shafts, e.g. grooves, expandable, bendable; Multiple coaxial sliding cannulas, e.g. for dilating
    • A61B17/3421—Cannulas
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B17/00—Surgical instruments, devices or methods, e.g. tourniquets
    • A61B17/34—Trocars; Puncturing needles
    • A61B17/3474—Insufflating needles, e.g. Veress needles
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B90/00—Instruments, implements or accessories specially adapted for surgery or diagnosis and not covered by any of the groups A61B1/00 - A61B50/00, e.g. for luxation treatment or for protecting wound edges
    • A61B90/10—Instruments, implements or accessories specially adapted for surgery or diagnosis and not covered by any of the groups A61B1/00 - A61B50/00, e.g. for luxation treatment or for protecting wound edges for stereotaxic surgery, e.g. frame-based stereotaxis
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B17/00—Surgical instruments, devices or methods, e.g. tourniquets
    • A61B2017/00017—Electrical control of surgical instruments
    • A61B2017/00022—Sensing or detecting at the treatment site
    • A61B2017/00026—Conductivity or impedance, e.g. of tissue
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B17/00—Surgical instruments, devices or methods, e.g. tourniquets
    • A61B2017/00017—Electrical control of surgical instruments
    • A61B2017/00022—Sensing or detecting at the treatment site
    • A61B2017/00084—Temperature
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B17/00—Surgical instruments, devices or methods, e.g. tourniquets
    • A61B2017/00681—Aspects not otherwise provided for
    • A61B2017/00734—Aspects not otherwise provided for battery operated
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B18/00—Surgical instruments, devices or methods for transferring non-mechanical forms of energy to or from the body
    • A61B18/04—Surgical instruments, devices or methods for transferring non-mechanical forms of energy to or from the body by heating
    • A61B18/12—Surgical instruments, devices or methods for transferring non-mechanical forms of energy to or from the body by heating by passing a current through the tissue to be heated, e.g. high-frequency current
    • A61B18/1206—Generators therefor
    • A61B2018/1246—Generators therefor characterised by the output polarity
    • A61B2018/1253—Generators therefor characterised by the output polarity monopolar
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B18/00—Surgical instruments, devices or methods for transferring non-mechanical forms of energy to or from the body
    • A61B18/04—Surgical instruments, devices or methods for transferring non-mechanical forms of energy to or from the body by heating
    • A61B18/12—Surgical instruments, devices or methods for transferring non-mechanical forms of energy to or from the body by heating by passing a current through the tissue to be heated, e.g. high-frequency current
    • A61B18/1206—Generators therefor
    • A61B2018/1246—Generators therefor characterised by the output polarity
    • A61B2018/126—Generators therefor characterised by the output polarity bipolar
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B34/00—Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
    • A61B34/20—Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
    • A61B2034/2046—Tracking techniques
    • A61B2034/2048—Tracking techniques using an accelerometer or inertia sensor
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B34/00—Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
    • A61B34/20—Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
    • A61B2034/2046—Tracking techniques
    • A61B2034/2055—Optical tracking systems
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B34/00—Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
    • A61B34/20—Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
    • A61B2034/2046—Tracking techniques
    • A61B2034/2065—Tracking using image or pattern recognition
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B34/00—Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
    • A61B34/20—Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
    • A61B2034/2068—Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis using pointers, e.g. pointers having reference marks for determining coordinates of body points
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B90/00—Instruments, implements or accessories specially adapted for surgery or diagnosis and not covered by any of the groups A61B1/00 - A61B50/00, e.g. for luxation treatment or for protecting wound edges
    • A61B90/08—Accessories or related features not otherwise provided for
    • A61B2090/0807—Indication means
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B90/00—Instruments, implements or accessories specially adapted for surgery or diagnosis and not covered by any of the groups A61B1/00 - A61B50/00, e.g. for luxation treatment or for protecting wound edges
    • A61B90/30—Devices for illuminating a surgical field, the devices having an interrelation with other surgical devices or with a surgical procedure
    • A61B2090/304—Devices for illuminating a surgical field, the devices having an interrelation with other surgical devices or with a surgical procedure using chemi-luminescent materials
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B90/00—Instruments, implements or accessories specially adapted for surgery or diagnosis and not covered by any of the groups A61B1/00 - A61B50/00, e.g. for luxation treatment or for protecting wound edges
    • A61B90/36—Image-producing devices or illumination devices not otherwise provided for
    • A61B2090/363—Use of fiducial points
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B90/00—Instruments, implements or accessories specially adapted for surgery or diagnosis and not covered by any of the groups A61B1/00 - A61B50/00, e.g. for luxation treatment or for protecting wound edges
    • A61B90/36—Image-producing devices or illumination devices not otherwise provided for
    • A61B2090/364—Correlation of different images or relation of image positions in respect to the body
    • A61B2090/365—Correlation of different images or relation of image positions in respect to the body augmented reality, i.e. correlating a live optical image with another image
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B90/00—Instruments, implements or accessories specially adapted for surgery or diagnosis and not covered by any of the groups A61B1/00 - A61B50/00, e.g. for luxation treatment or for protecting wound edges
    • A61B90/36—Image-producing devices or illumination devices not otherwise provided for
    • A61B90/37—Surgical systems with images on a monitor during operation
    • A61B2090/372—Details of monitor hardware
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B90/00—Instruments, implements or accessories specially adapted for surgery or diagnosis and not covered by any of the groups A61B1/00 - A61B50/00, e.g. for luxation treatment or for protecting wound edges
    • A61B90/39—Markers, e.g. radio-opaque or breast lesions markers
    • A61B2090/3937—Visible markers
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B90/00—Instruments, implements or accessories specially adapted for surgery or diagnosis and not covered by any of the groups A61B1/00 - A61B50/00, e.g. for luxation treatment or for protecting wound edges
    • A61B90/39—Markers, e.g. radio-opaque or breast lesions markers
    • A61B2090/3937—Visible markers
    • A61B2090/3941—Photoluminescent markers
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B90/00—Instruments, implements or accessories specially adapted for surgery or diagnosis and not covered by any of the groups A61B1/00 - A61B50/00, e.g. for luxation treatment or for protecting wound edges
    • A61B90/39—Markers, e.g. radio-opaque or breast lesions markers
    • A61B2090/3937—Visible markers
    • A61B2090/395—Visible markers with marking agent for marking skin or other tissue
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B90/00—Instruments, implements or accessories specially adapted for surgery or diagnosis and not covered by any of the groups A61B1/00 - A61B50/00, e.g. for luxation treatment or for protecting wound edges
    • A61B90/50—Supports for surgical instruments, e.g. articulated arms
    • A61B2090/502—Headgear, e.g. helmet, spectacles
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B2218/00—Details of surgical instruments, devices or methods for transferring non-mechanical forms of energy to or from the body
    • A61B2218/001—Details of surgical instruments, devices or methods for transferring non-mechanical forms of energy to or from the body having means for irrigation and/or aspiration of substances to and/or from the surgical site
    • A61B2218/002—Irrigation
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B2218/00—Details of surgical instruments, devices or methods for transferring non-mechanical forms of energy to or from the body
    • A61B2218/001—Details of surgical instruments, devices or methods for transferring non-mechanical forms of energy to or from the body having means for irrigation and/or aspiration of substances to and/or from the surgical site
    • A61B2218/007—Aspiration
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B34/00—Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
    • A61B34/30—Surgical robots
    • A61B34/37—Master-slave robots

Definitions

  • This disclosure relates to apparatuses, systems, and methods for providing an augmented reality interactive experience during a surgical procedure.
  • a surgical procedure it would be desirable to provide an augmented reality interactive experience of a real- world environment where objects that reside in the real world are enhanced by overlaying computer-generated perceptual information, sometimes across multiple sensory modalities, including visual, auditory, haptic, somatosensory, and olfactory.
  • images of a surgical field and surgical instruments and other objects appearing in the surgical field are enhanced by overlaying computer-generated visual, auditory, haptic, somatosensory, olfactory, or other sensory information onto the real world images of the surgical field and instruments or other objects appearing in the surgical field.
  • the images may be streamed in real time or may be still images.
  • Real world surgical instruments include a variety of surgical devices including energy, staplers, or combined energy and stapler.
  • Energy based medical devices include, without limitation, radio-frequency (RF) based monopolar and bipolar electrosurgical instruments, ultrasonic surgical instruments, combination RF electrosurgical and ultrasonic instruments, combination RF electrosurgical and mechanical staplers, among others.
  • Surgical stapler devices are surgical instruments used to cut and staple tissue in a variety of surgical procedures, including bariatric, thoracic, colorectal, gynecologic, urologic and general surgery.
  • the present disclosure provides a surgical system comprising: a surgical device comprising: an axial passage defining an outer diameter and an inner diameter; a proximal end; a distal end configured to penetrate tissue; a camera array comprising individual cameras connected in a ring configuration with an elastic connection; a removable installation trigger configured to extend the camera array from a first recessed position from the inner diameter of the distal end of the axial passage to a second deployed position with the camera array circumferentially positioned around the outer diameter of the distal end of the axial passage; an augmented reality (AR) device; and a surgical hub communicatively coupled to the camera array and the AR device, wherein the surgical hub comprises a control circuit coupled to a memory, and wherein the control circuit is configured to: receive a plurality of video feed from the camera array; identify a physical marker on the video feed; and display the physical marker on the AR display.
  • AR augmented reality
  • the present disclosure provides a surgical device comprising: a camera array comprising individual cameras connected in a ring configuration with an elastic connection, wherein the camera array is communicatively couplable to a surgical hub; an elongated penetration member having a proximal end and a distal end, wherein the distal end further comprises a tissue penetrating tip; an axial passage through the elongated penetration member and the tissue penetrating tip, and wherein an inner diameter of the axial passage is sized to house the camera array in a first recessed position; and a removable installation trigger configured to extend the camera array from a first recessed position from the inner diameter of the distal end of the elongated penetration member to a second deployed position with the camera array circumferentially positioned around an outer diameter of the distal end of the elongated penetration member.
  • the present disclosure provides a method for displaying a surgical location inside of a patient, the method comprising: receiving, by a surgical hub, a video feed from a camera located inside a patient; identifying, by the surgical hub, a physical marker inside of the patient; determining, by the surgical hub, a target location based on the relationship to the physical marker; generating, by the surgical hub, a virtual element corresponding to the target location; and displaying, by an augmented reality (AR) device coupled to the surgical hub, the virtual element overlaid on the video feed on an AR display.
  • AR augmented reality
  • FIG. 1 is a block diagram of a computer-implemented interactive surgical system, according to one aspect of this disclosure.
  • FIG. 2 is a surgical system being used to perform a surgical procedure in an operating room, according to one aspect of this disclosure.
  • FIG. 3 is a surgical hub paired with a visualization system, a robotic system, and an intelligent instrument, according to one aspect of this disclosure.
  • FIG. 4 illustrates a surgical data network comprising a modular communication hub configured to connect modular devices located in one or more operating theaters of a healthcare facility, or any room in a healthcare facility specially equipped for surgical operations, to the cloud, according to one aspect of this disclosure.
  • FIG. 5 illustrates a computer-implemented interactive surgical system, according to one aspect of this disclosure.
  • FIG. 6 illustrates a surgical hub comprising a plurality of modules coupled to the modular control tower, according to one aspect of this disclosure.
  • FIG. 7 illustrates an augmented reality (AR) system comprising an intermediate signal combiner positioned in the communication path between an imaging module and a surgical hub display, according to one aspect of this disclosure.
  • AR augmented reality
  • FIG. 8 illustrates an augmented reality (AR) system comprising an intermediate signal combiner positioned in the communication path between an imaging module and a surgical hub display, according to one aspect of this disclosure.
  • AR augmented reality
  • FIG. 9 illustrates an augmented reality (AR) device worn by a surgeon to communicate data to the surgical hub, according to one aspect of this disclosure.
  • AR augmented reality
  • FIG. 10 illustrates a system for augmenting surgical instrument information using an augmented reality display, according to one aspect of this disclosure.
  • FIG. 11 illustrates a timeline of a situational awareness surgical procedure, according to one aspect of this disclosure.
  • FIG. 12 shows a structural surface comprising a plurality of fiducial markers, according to one aspect of this disclosure.
  • FIG. 13 shows a process for surface matching external structure of a patient with fiducial markers, according to one aspect of this disclosure.
  • FIG. 14 shows a process for surface matching internal structure of a patient with fiducial markers, according to one aspect of this disclosure.
  • FIG. 15 shows a stereotactic frame external surgical alignment instruments to aid a surgeon in a surgical procedure, according to one aspect of this disclosure.
  • FIG. 16 shows a starfix platform external surgical alignment instruments to aid a surgeon in a surgical procedure, according to one aspect of this disclosure.
  • FIG. 17 shows a microtable external surgical alignment instruments to aid a surgeon in a surgical procedure, according to one aspect of this disclosure.
  • FIG. 18 shows a flow diagram for identifying objects based on a plurality of a registration parameters, according to one aspect of this disclosure.
  • FIG. 19 shows a flow diagram for classifying unknown surgical instruments based on a partial information of known and unknown parameters, according to one aspect of this disclosure.
  • FIG. 20 shows a trocar comprising an internal camera system, according to one aspect of this disclosure.
  • FIG. 21 shows a reusable installation tool, configured to insert into the proximal end of the trocar, deploy and retract the camera system around the outer diameter of the trocar, according to one aspect of this disclosure.
  • FIG. 22 shows a plurality fiducial markers tagged to areas of interest, in a pre operative computerized tomography (CT) scan, according to one aspect of this disclosure.
  • FIG. 23 shows a laparoscopic surgical procedure that utilizes a plurality of fiducial markers to aid a surgeon in locating a surgical site, according to one aspect of this disclosure.
  • FIG. 24 shows a physical marker applied by injection into a vascular system of a patient with an indocyanine dye, according to one aspect of this disclosure.
  • FIG. 25 further shows exemplary tissue injected with a dye and illuminated to show vasculature, according to one aspect of this disclosure.
  • FIG. 26 shows a system configured to monitor the change in pressure or fluid in a body cavity according to an impendence measurement by a probe, according to one aspect of this disclosure.
  • FIG. 27 shows an infrared (IR) heat detection system comprising an IR camera system configured direct IR light on a treated region of tissue and identify a temperature difference in a surgical environment, according to one aspect of this disclosure.
  • IR infrared
  • FIG. 28 shows a surgical procedure employing three end effectors configured to grasp and transect tissue, according to one aspect of this disclosure.
  • FIG. 29 shows the third end effector sliding along the tissue from a first position to a second position, according to one aspect of this disclosure.
  • FIG. 30 shows the third end effector positioned adjacent to the second end effector, according to one aspect of this disclosure.
  • FIG. 31 shows a surgical procedure comprising three static clamps and a dynamic clamp configured to transfer tissue between stationary, according to one aspect of this disclosure.
  • FIG. 32 shows a logic flow diagram of a method for displaying a surgical location inside of a patient, according to one aspect of this disclosure.
  • Energy based medical devices include, without limitation, radio-frequency (RF) based monopolar and bipolar electrosurgical instruments, ultrasonic surgical instruments, combination RF electrosurgical and ultrasonic instruments, combination RF electrosurgical and mechanical staplers, among others.
  • Surgical stapler devices include and combined surgical staplers with electrosurgical and / or ultrasonic devices.
  • Aspects of the ultrasonic surgical devices can be configured for transecting and / or coagulating tissue during surgical procedures, for example.
  • Aspects of the electrosurgical devices can be configured for transecting, coagulating, sealing, welding and / or desiccating tissue during surgical procedures, for example.
  • the surgical stapler devices can be configured for transecting and stapling tissue during surgical procedures and in some aspects, the surgical stapler devices may be configured to delivery RF energy to the tissue during surgical procedures. Electrosurgical devices are configured to deliver therapeutic and / or nontherapeutic RF energy to the tissue. Elements of surgical staplers, electrosurgical, and ultrasonic devices may be used in combination in a single surgical instrument.
  • the present disclosure provides onscreen displays of real time information to the OR team during a surgical procedure.
  • many new and unique onscreen displays are provided to display onscreen a variety of visual information feedback to the OR team.
  • visual information may comprise one or more than one of various visual media with or without sound.
  • visual information comprises still photography, motion picture photography, video or audio recording, graphic arts, visual aids, models, display, visual presentation services, and the support processes.
  • the visual information can be communicated on any number of display options such as the primary OR screen, the energy or surgical stapler device itself, a tablet, augmented reality glasses, among others, for example.
  • the present disclosure provides a large list of potential options to communicate visual information in real time to the OR team, without overwhelming the OR team with too much visual information.
  • the present disclosure provides onscreen displays of visual information to enable the surgeon, or other members of the OR team, to selectively activate onscreen displays such as icons surrounding the screen option to manage a wealth of visual information.
  • One or a combination of factors can be used to determine the active display, these may include energy based (e.g., electrosurgical, ultrasonic) or mechanical based (e.g., staplers) surgical devices in use, the estimated risk associated with a given display, the experience level of the surgeon and the surgeons’ choice among other things.
  • the visual information may comprises rich data overlaid or superimposed into the surgical field of view to manage the visual information.
  • the visual information may comprise rich data overlaid or superimposed into the surgical field of view to manage the visual information.
  • the visual information may comprise superimposed imagery that requires video analysis and tracking to properly overlay the data.
  • Visual information data communicated in this manner may provide additional useful visual information in a more concise and easy to understand way to the OR team.
  • the present disclosure provides techniques for selectively activating onscreen displays such as icons surrounding the screen to manage visual information during a surgical procedure.
  • the present disclosure provides techniques for determining the active display using one or a combination of factors.
  • the techniques according to the resent disclosure may comprise selecting the energy based or mechanical based surgical device in use as the active display, estimating risk associated with a given display, utilizing the experience level of the surgeon or OR team making the selection, among other things.
  • the techniques according to the present disclosure may comprise overlaying or superimposing rich data onto the surgical field of view to manage the visual information.
  • a number of the display arrangements described by the present disclosure involve overlaying various visual representations of surgical data onto a livestream of a surgical field.
  • the term overlay comprises a translucent overlay, a partial overlay, and / or a moving overlay.
  • Graphical overlays may be in the form of a transparent graphic, semitransparent graphic, or opaque graphic, or a combination of transparent, semitransparent, and opaque elements or effects.
  • the overlay can be positioned on, or at least partially on, or near an object in the surgical field such as, for example, an end effector and/or a critical surgical structure.
  • Certain display arrangements may comprise a change in one or more display elements of an overlay including a change in color, size, shape, display time, display location, display frequency, highlighting, or a combination thereof, based on changes in display priority values.
  • the graphical overlays are rendered on top of the active display monitor to convey important information quickly and efficiently to the OR team.
  • the techniques according to the present disclosure may comprise superimposing imagery that requires analyzing video and tracking for properly overlaying the visual information data.
  • the techniques according to the present disclosure may comprise communicating rich visual information, as opposed to simple static icons, to provide additional visual information to the OR team in a more concise and easy to understand manner.
  • the visual overlays may be used in combination with audible and / or somatosensory overlays such as thermal, chemical, and mechanical devices, and combinations thereof.
  • AR augmented reality
  • images of a surgical field and surgical instruments and other objects appearing in the surgical field are enhanced by overlaying computer-generated visual, auditory, haptic, somatosensory, olfactory, or other sensory information onto the real world images of the surgical field, instruments, and/or other objects appearing in the surgical field.
  • the images may be streamed in real time or may be still images.
  • Augmented reality is a technology for rendering and displaying virtual or “augmented” virtual objects, data, or visual effects overlaid on a real environment.
  • the real environment may include a surgical field.
  • the virtual objects overlaid on the real environment may be represented as anchored or in a set position relative to one or more aspects of the real environment.
  • a real world object exits the real environment field of view
  • a virtual object anchored to the real world object would also exit the augmented reality field of view.
  • a number of the display arrangements described by the present disclosure involve overlaying various visual representations of surgical data onto a livestream of a surgical field.
  • overlaying comprises a translucent overlay, a partial overlay, and/or a moving overlay.
  • the overlay can be positioned on, or at least partially on, or near an object in the surgical field such as, for example, an end effector and/or a critical surgical structure.
  • Certain display arrangements may comprise a change in one or more display elements of an overlay including a change in color, size, shape, display time, display location, display frequency, highlighting, or a combination thereof, based on changes in display priority values.
  • AR is an enhanced version of the real physical world that is achieved through the use of digital visual elements, sound, or other sensory stimuli delivered via technology.
  • Virtual Reality VR is a computer-generated environment with scenes and objects that appear to be real, making the user feel they are immersed in their surroundings. This environment is perceived through a device known as a Virtual Reality headset or helmet.
  • Mixed reality (MR) and AR are both considered immersive technologies, but they aren't the same.
  • MR is an extension of Mixed reality that allows real and virtual elements to interact in an environment. While AR adds digital elements to a live view often by using a camera, an MR experience combines elements of both AR and VR, where real-world and digital objects interact.
  • one or more computer-generated virtual objects may be displayed along with one or more real (i.e. , so-called “real world”) elements.
  • a real-time image or video of a surrounding environment may be shown on a computer screen display with one or more overlaying virtual objects.
  • Such virtual objects may provide complementary information relating to the environment or generally enhance a user's perception and engagement with the environment.
  • the real-time image or video of the surrounding environment may additionally or alternatively enhance a user's engagement with the virtual objects shown on the display.
  • the apparatuses, systems, and methods in the context of this disclosure enhance images received from one or more imaging devices during a surgical procedure.
  • the imaging devices may include a variety of scopes used during non-invasive and minimally invasive surgical procedures, an AR device, and/or a camera to provide images during open surgical procedures.
  • the images may be streamed in real time or may be still images.
  • the apparatuses, systems, and methods provide an augmented reality interactive experience by enhancing images of the real world surgical environment by overlaying virtual objects or representations of data and/or real objects onto the real surgical environment.
  • the augmented reality experience may be viewed on a display and/or an AR device that allows a user to view the overlaid virtual objects onto the real world surgical environment.
  • the display may be located in the operating room or remote from the operating room.
  • AR devices are worn on the head of the surgeon or other operating room personnel and typically include two stereo-display lenses or screens, including one for each eye of the user. Natural light is permitted to pass through the two transparent or semi-transparent display lenses such that aspects of the real environment are visible while also projecting light to make virtual objects visible to the user of the AR device.
  • Two or more displays and AR devices may be used in a coordinated manner, for example with a first display or AR device controlling one or more additional displays or AR devices in a system with defined roles.
  • a user when activating display or an AR device, a user may select a role (e.g., surgeon, surgical assistant, nurse, etc., during a surgical procedure) and the display or AR device may display information relevant to that role.
  • a surgical assistant may have a virtual representation of an instrument displayed that the surgeon needs to perform for a next step of a surgical procedure. A surgeon’s focus on the current step may see different information displayed than the surgical assistant.
  • augmented reality interactive experiences include visual, auditory, haptic, somatosensory, olfactory, or other sensory feedback information to the surgical team inside or outside the operating room.
  • the virtual feedback information overlaid onto the real world surgical environment may be provided to an operating room (OR) team, including personnel inside the OR including, without limitation, the operating surgeon, assistants to the surgeon, a scrub person, an anesthesiologist and a circulating nurse, among others, for example.
  • the virtual feedback information can be communicated on any number of display options such as a primary OR screen display, an AR device, the energy or surgical stapler instrument, a tablet, augmented reality glasses, device etc.
  • FIG. 1 depicts a computer-implemented interactive surgical system 1 that includes one or more surgical systems 2 and a cloud-based system 4.
  • the cloud-based system 4 may include a remote server 13 coupled to a storage device 5.
  • Each surgical system 2 includes at least one surgical hub 6 in communication with the cloud 4.
  • the surgical system 2 may include a visualization system 8, a robotic system 10, and handheld intelligent surgical instruments 12, each configured to communicate with one another and/or the hub 6.
  • a surgical system 2 may include an M number of hubs 6, an N number of visualization systems 8, an O number of robotic systems 10, and a P number of handheld intelligent surgical instruments 12, where M, N, O, and P are integers greater than or equal to one.
  • the computer- implemented interactive surgical system 1 may be configured to provide an augmented reality interactive experience during a surgical procedure as described herein.
  • FIG. 2 depicts an example of a surgical system 2 to perform a surgical procedure on a patient lying down on an operating table 14 in a surgical operating room 16.
  • a robotic system 10 is used in the surgical procedure as a part of the surgical system 2.
  • the robotic system 10 includes a surgeon’s console 18, a patient side cart 20 (surgical robot), and a surgical robotic hub 22.
  • the patient side cart 20 can manipulate at least one removably coupled surgical tool 17 through a minimally invasive incision in the body of the patient while the surgeon views the surgical site through the surgeon’s console 18 or an augmented reality (AR) device 66 worn by the surgeon.
  • AR augmented reality
  • An image (e.g., still or live streamed in real time) of the surgical site during a minimally invasive procedure can be obtained by a medical imaging device 24.
  • the patient side cart 20 can manipulate the imaging device 24 to orient the imaging device 24.
  • An image of an open surgical procedure can be obtained by a medical imaging device 96.
  • the robotic hub 22 processes the images of the surgical site for subsequent display on the surgeon’s console 18 or the AR device 66 worn by the surgeon, or other person in the surgical operating room 16.
  • the optical components of the imaging device 24, 96 or AR device 66 may include one or more illumination sources and/or one or more lenses.
  • the one or more illumination sources may be directed to illuminate portions of the surgical field.
  • One or more image sensors may receive light reflected or refracted from tissue and instruments in the surgical field.
  • the imaging device 24 is configured for use in a minimally invasive surgical procedure.
  • imaging devices suitable for use with this disclosure include, but not limited to, an arthroscope, angioscope, bronchoscope, choledochoscope, colonoscope, cytoscope, duodenoscope, enteroscope, esophagogastro-duodenoscope (gastroscope), endoscope, laryngoscope, nasopharyngo-neproscope, sigmoidoscope, thoracoscope, and ureteroscope.
  • the imaging device 96 is configured for use in an open (invasive) surgical procedure.
  • the visualization system 8 includes one or more imaging sensors, one or more image-processing units, one or more storage arrays, and one or more displays that are strategically arranged with respect to the sterile field.
  • the visualization system 8 includes an interface for HL7, PACS, and EMR.
  • the imaging device 24 may employ multi-spectrum monitoring to discriminate topography and underlying structures.
  • a multi- spectral image captures image data within specific wavelength ranges in the electromagnetic spectrum. Wavelengths are separated by filters or instruments sensitive to particular wavelengths, including light from frequencies beyond the visible light range, e.g., IR and ultraviolet.
  • Spectral imaging can extract information not visible to the human eye.
  • Multi-spectrum monitoring can relocate a surgical field after a surgical task is completed to perform tests on the treated tissue.
  • FIG. 2 depicts a primary display 19 positioned in the sterile field to be visible to an operator at the operating table 14.
  • a visualization tower 11 is positioned outside the sterile field and includes a first non-sterile display 7 and a second non-sterile display 9, which face away from each other.
  • the visualization system 8, guided by the hub 6, is configured to utilize the displays 7, 9, 19 to coordinate information flow to operators inside and outside the sterile field.
  • the hub 6 may cause the visualization system 8 to display AR images of the surgical site, as recorded by an imaging device 24, 96 on a non-sterile display 7, 9, or through the AR device 66, while maintaining a live feed of the surgical site on the primary display 19 or the AR device 66.
  • the non-sterile display 7, 9 can permit a non-sterile operator to perform a diagnostic step relevant to the surgical procedure, for example.
  • FIG. 3 depicts a hub 6 in communication with a visualization system 8, a robotic system 10, and a handheld intelligent surgical instrument 12.
  • the hub 6 includes a hub display 35, an imaging module 38, a generator module 40, a communication module 30, a processor module 32, a storage array 34, and an operating room mapping module 33.
  • the hub 6 further includes a smoke evacuation module 26 and/or a suction/irrigation module 28.
  • the imaging module 38 comprises an AR device 66 and the processor module 32 comprises an integrated video processor and an augmented reality modeler (e.g., as shown in FIG. 10).
  • a modular light source may be adapted for use with various imaging devices.
  • multiple imaging devices may be placed at different positions in the surgical field to provide multiple views (e.g., non-invasive, minimally invasive, invasive or open surgical procedures).
  • the imaging module 38 can be configured to switch between the imaging devices to provide an optimal view.
  • the imaging module 38 can be configured to integrate the images from the different imaging devices and provide an augmented reality interactive experience during a surgical procedure as described herein.
  • FIG. 4 shows a surgical data network 51 comprising a modular communication hub 53 configured to connect modular devices located in one or more operating theaters/rooms of a healthcare facility to a cloud-based system.
  • the cloud 54 may include a remote server 63 (FIG. 5) coupled to a storage device 55.
  • the modular communication hub 53 comprises a network hub 57 and/or a network switch 59 in communication with a network router 61 .
  • the modular communication hub 53 is coupled to a local computer system 60 to process data.
  • Modular devices 1a-1 n in the operating theater may be coupled to the modular communication hub 53.
  • the network hub 57 and/or the network switch 59 may be coupled to a network router 61 to connect the devices 1a-1 n to the cloud 54 or the local computer system 60.
  • Data associated with the devices 1a-1 n may be transferred to cloud-based computers via the router for remote data processing and manipulation.
  • the operating theater devices 1a-1n may be connected to the modular communication hub 53 over a wired channel or a wireless channel.
  • the surgical data network 51 environment may be employed to provide an augmented reality interactive experience during a surgical procedure as described herein and in particular providing augmented images if the surgical field to one or more than one remote display 58.
  • FIG. 5 illustrates a computer-implemented interactive surgical system 50.
  • the computer-implemented interactive surgical system 50 is similar in many respects to the computer-implemented interactive surgical system 1 .
  • the computer-implemented interactive surgical system 50 includes one or more surgical systems 52, which are similar in many respects to the surgical systems 2.
  • Each surgical system 52 includes at least one surgical hub 56 in communication with a cloud 54 that may include a remote server 63.
  • the computer-implemented interactive surgical system 50 comprises a modular control tower 23 connected to multiple operating theater devices such as, for example, intelligent surgical instruments, robots, and other computerized devices located in the operating theater.
  • the modular control tower 23 comprises a modular communication hub 53 coupled to a computer system 60.
  • the modular control tower 23 is coupled to an imaging module 38 that is coupled to an endoscope 98, a generator module 27 that is coupled to an energy device 99, a smoke evacuator module 76, a suction/irrigation module 78, a communication module 13, a processor module 15, a storage array 16, a smart device/instrument 21 optionally coupled to a display 39, and a sensor module 29.
  • the operating theater devices are coupled to cloud computing resources such as server 63, data storage 55, and displays 58 via the modular control tower 23.
  • a robot hub 72 also may be connected to the modular control tower 23 and to the servers 63, data storage 55, and displays 58.
  • the devices/instruments 21 , visualization systems 58, among others, may be coupled to the modular control tower 23 via wired or wireless communication standards or protocols, as described herein.
  • the modular control tower 23 may be coupled to a hub display 65 (e.g., monitor, screen) to display augmented images received comprising overlaid virtual objects on the real surgical field received from the imaging module 38, device/instrument display 39, and/or other visualization systems 58.
  • the hub display 65 also may display data received from devices connected to the modular control tower 23 in conjunction with images and overlaid images.
  • FIG. 6 illustrates a surgical hub 56 comprising a plurality of modules coupled to the modular control tower 23.
  • the modular control tower 23 comprises a modular communication hub 53, e.g., a network connectivity device, and a computer system 60 to provide local processing, visualization, and imaging of augmented surgical information, for example.
  • the modular communication hub 53 may be connected in a tiered configuration to expand the number of modules (e.g., devices) that may be connected to the modular communication hub 53 and transfer data associated with the modules to the computer system 60, cloud computing resources, or both.
  • Each of the network hubs/switches 57, 59 in the modular communication hub 53 may include three downstream ports and one upstream port.
  • the upstream network hub/switch 57, 59 is connected to a processor 31 to provide a communication connection to the cloud computing resources and a local display 67. Communication to the cloud 54 may be made either through a wired or a wireless communication channel.
  • the computer system 60 comprises a processor 31 and a network interface 37.
  • the processor 31 is coupled to a communication module 41 , storage 45, memory 46, non-volatile memory 47, and input/output interface 48 via a system bus.
  • the system bus can be any of several types of bus structure(s) including the memory bus or memory controller, a peripheral bus or external bus, and/or a local bus using any variety of available bus architectures.
  • the processor 31 comprises an augmented reality modeler (e.g., as shown in FIG. 10) and may be implemented as a single-core or multicore processor such as those known under the trade name ARM Cortex by Texas Instruments.
  • the processor may be an LM4F230H5QR ARM Cortex-M4F Processor Core, available from Texas Instruments, for example, comprising an on-chip memory of 256 KB single-cycle flash memory, or other non volatile memory, up to 40 MHz, a prefetch buffer to improve performance above 40 MHz, a 32 KB single-cycle serial random access memory (SRAM), an internal read-only memory (ROM) loaded with StellarisWare® software, a 2 KB electrically erasable programmable read-only memory (EEPROM), and/or one or more pulse width modulation (PWM) modules, one or more quadrature encoder inputs (QEI) analogs, one or more 12-bit analog-to-digital converters (ADCs) with 12 analog input channels
  • QEI
  • the system memory includes volatile memory and non-volatile memory.
  • the basic input/output system (BIOS) containing the basic routines to transfer information between elements within the computer system, such as during start-up, is stored in non-volatile memory.
  • the non-volatile memory can include ROM, programmable ROM (PROM), electrically programmable ROM (EPROM), EEPROM, or flash memory.
  • Volatile memory includes random-access memory (RAM), which acts as external cache memory.
  • RAM random-access memory
  • RAM is available in many forms such as SRAM, dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), and direct Rambus RAM (DRRAM).
  • DRAM dynamic RAM
  • SDRAM synchronous DRAM
  • DDR SDRAM double data rate SDRAM
  • ESDRAM enhanced SDRAM
  • SLDRAM Synchlink DRAM
  • DRRAM direct Rambus RAM
  • the computer system 60 also includes removable/non-removable, volatile/non volatile computer storage media, such as for example disk storage.
  • the disk storage includes, but is not limited to, devices like a magnetic disk drive, floppy disk drive, tape drive, Jaz drive, Zip drive, LS-60 drive, flash memory card, or memory stick.
  • the disk storage can include storage media separately or in combination with other storage media including, but not limited to, an optical disc drive such as a compact disc ROM device (CD-ROM), compact disc recordable drive (CD-R Drive), compact disc rewritable drive (CD-RW Drive), or a digital versatile disc ROM drive (DVD-ROM).
  • CD-ROM compact disc ROM
  • CD-R Drive compact disc recordable drive
  • CD-RW Drive compact disc rewritable drive
  • DVD-ROM digital versatile disc ROM drive
  • a removable or non-removable interface may be employed.
  • the computer system 60 of FIG. 6, the imaging module 38 and/or visualization system 58, and/or the processor module 15 of FIGS. 4-6 may comprise an image processor, image-processing engine, graphics processing unit (GPU), media processor, or any specialized digital signal processor (DSP) used for the processing of digital images.
  • the image processor may employ parallel computing with single instruction, multiple data (SIMD) or multiple instruction, multiple data (MIMD) technologies to increase speed and efficiency.
  • SIMD single instruction, multiple data
  • MIMD multiple instruction, multiple data
  • the digital image-processing engine can perform a range of tasks.
  • the image processor may be a system on a chip with multicore processor architecture.
  • FIG. 7 illustrates an augmented reality system 263 comprising an intermediate signal combiner 64 positioned in the communication path between an imaging module 38 and a surgical hub display 67.
  • the signal combiner 64 combines audio and/or image data received from an imaging module 38 and/or an AR device 66.
  • the surgical hub 56 receives the combined data from the combiner 64 and overlays the data provided to the display 67, where the overlaid data is displayed.
  • the imaging device 68 may be a digital video camera and the audio device 69 may be a microphone.
  • the signal combiner 64 may comprise a wireless heads-up display adapter to couple to the AR device 66 placed into the communication path of the display 67 to a console allowing the surgical hub 56 to overlay data on the display 67.
  • FIG. 8 illustrates an augmented reality (AR) system comprising an intermediate signal combiner positioned in the communication path between an imaging module and a surgical hub display.
  • FIG. 8 illustrates an AR device 66 worn by a surgeon 73 to communicate data to the surgical hub 56.
  • Peripheral information of the AR device 66 does not include active video. Rather, the peripheral information includes only device settings, or signals that do not have same demands of refresh rates. Interaction may augment the surgeon's 73 information based on linkage with preoperative computerized tomography (CT) or other data linked in the surgical hub 56.
  • CT computerized tomography
  • the AR device 66 can identify structure - ask whether instrument is touching a nerve, vessel, or adhesion, for example.
  • the AR device 66 may include pre-operative scan data, an optical view, tissue interrogation properties acquired throughout procedure, and/or processing in the surgical hub 56 used to provide an answer.
  • the surgeon 73 can dictate notes to the AR device 66 to be saved with patient data in the hub storage 45 for later use in report or in follow up.
  • the AR device 66 worn by the surgeon 73 links to the surgical hub 56 with audio and visual information to avoid the need for overlays, and allows customization of displayed information around periphery of view.
  • the AR device 66 provides signals from devices (e.g., instruments), answers queries about device settings, or positional information linked with video to identify quadrant or position.
  • the AR device 66 has audio control and audio feedback from the AR device 66.
  • the AR device 66 is able to interact with other systems in the operating theater and have feedback and interaction available wherever the surgeon 73 is viewing.
  • the AR device 66 may receive voice or gesture initiated commands and queries from a surgeon, and the AR device 66 may provide feedback in the form of one or more modalities including audio, visual, or haptic touch.
  • FIG. 9 illustrates a surgeon 73 wearing an AR device 66, a patient 74, and may include a camera 96 in an operating room 75.
  • the AR device 66 worn by the surgeon 73 may be used to present to the surgeon 73 a virtual object overlaid on a real time image of the surgical field through augmented reality display 89 or through the hub connected display 67.
  • the real time image may include a portion of a surgical instrument 77.
  • the virtual object may not be visible to others within the operating room 75 (e.g., surgical assistant or nurse), though they also may wear AR devices 66. Even if another person is viewing the operating room 75 with an AR device 66, the person may not be able to see the virtual object or may be able to see the virtual object in a shared augmented reality with the surgeon 73, or may be able to see a modified version of the virtual object (e.g., according to customizations unique to the surgeon 73) or may see different virtual objects.
  • a virtual object and/or data may be configured to appear on a portion of a surgical instrument 77 or in a surgical field of view captured by an imaging module 38, an imaging device 68 during minimally invasive surgical procedures, and/or the camera 96 during open surgical procedures.
  • the imaging module 38 is a laparoscopic camera that provides a live feed of a surgical area during a minimally invasive surgical procedure.
  • An AR system may present virtual objects that are fixed to a real object without regard to a perspective of a viewer or viewers of the AR system (e.g., the surgeon 73).
  • a virtual object may be visible to a viewer of the AR system inside the operating room 75 and not visible to a viewer of the AR system outside the operating room 75.
  • the virtual object may be displayed to the viewer outside the operating room 75 when the viewer enters the operating room 75.
  • the augmented image may be displayed on the surgical hub display 67 or the augmented reality display 89.
  • the AR device 66 may include one or more screens or lens, such as a single screen or two screens (e.g., one per eye of a user).
  • the screens may allow light to pass through the screens such that aspects of the real environment are visible while displaying the virtual object.
  • the virtual object may be made visible to the surgeon 73 by projecting light.
  • a virtual object may appear to have a degree of transparency or may be opaque (i.e., blocking aspects of the real environment).
  • An AR system may be viewable to one or more viewers, and may include differences among views available for the one or more viewers while retaining some aspects as universal among the views.
  • a heads-up display may change between two views while virtual objects and/or data may be fixed to a real object or area in both views. Aspects such as a color of an object, lighting, or other changes may be made among the views without changing a fixed position of at least one virtual object.
  • a user may see a virtual object and/or data presented in an AR system as opaque or as including some level of transparency.
  • the user may interact with the virtual object, such as by moving the virtual object from a first position to a second position.
  • the user may move an object with his or her hand. This may be done in the AR system virtually by determining that the hand has moved into a position coincident or adjacent to the object (e.g., using one or more cameras, which may be mounted on the AR device 66, such as AR device camera 79 or separate 96, and which may be static or may be controlled to move), and causing the object to move in response.
  • Virtual aspects may include virtual representations of real world objects or may include visual effects, such as lighting effects, etc.
  • the AR system may include rules to govern the behavior of virtual objects, such as subjecting a virtual object to gravity or friction, or may include other predefined rules that defy real world physical constraints (e.g., floating objects, perpetual motion, etc.).
  • the AR device 66 may include a camera 79 on the AR device 66 (not to be confused with the camera 96, separate from the AR device 66).
  • the AR device camera 79 or the camera 96 may include an infrared camera, an infrared filter, a visible light filter, a plurality of cameras, a depth camera, etc.
  • the AR device 66 may project virtual items over a representation of a real environment, which may be viewed by a user.
  • the AR device 66 may be used in the operating room 75 during a surgical procedure, for example performed by the surgeon 73 on the patient 74.
  • the AR device 66 may project or display virtual objects, such as a virtual object during the surgical procedure to augment the surgeon's vision.
  • the surgeon 73 may view a virtual object using the AR device 66, a remote controller for the AR device 66, or may interact with a virtual object, for example, using a hand to “interact” with a virtual object or a gesture recognized by the camera 79 of the AR device 66.
  • a virtual object may augment a surgical tool such as the surgical instrument 77.
  • the virtual object may appear (to the surgeon 73 viewing the virtual object through the AR device 66) to be coupled with or remain a fixed distance from the surgical instrument 77.
  • the virtual object may be used to guide the surgical instrument 77, and may appear to be fixed to the patient 74.
  • a virtual object may react to movements of other virtual or real-world objects in the surgical field.
  • the virtual object may be altered when a surgeon is manipulating a surgical instrument in proximity to the virtual object.
  • the augmented reality display system imaging device 38 capture a real image of a surgical area during a surgical procedure.
  • An augmented reality display 89, 67 presents an overlay of an operational aspect of the surgical instrument 77 onto the real image of the surgical area.
  • the surgical instrument 77 includes communications circuitry 231 to communicate operational aspects and functional data from the surgical instrument 77 to the AR device 66 via communication communications circuitry 233 on the AR device 66.
  • the surgical instrument 77 and the AR device 66 are shown in RF wireless communication between circuits 231 , 233 as indicated by arrows B, C, other communication techniques may employed (e.g., wired, ultrasonic, infrared, etc.).
  • the overlay is related to the operational aspect of the surgical instrument 77 being actively visualized.
  • the overlay combines aspects of tissue interaction in the surgical area with functional data from the surgical instrument 77.
  • a processor portion of the AR device 66 is configured to receive the operational aspects and functional data from the surgical instrument 77, determine the overlay related to the operation of the surgical instrument 77, and combine the aspect of the tissue in the surgical area with the functional data from the surgical instrument 77.
  • the augmented images indicate alerts relative to device performance considerations, alerts of incompatible usage, alerts on incomplete capture. Incompatible usage includes tissue out range conditions and tissue incorrectly balanced within the jaws of the end effector. Additional augmented images provide an indication of collateral events including indication of tissue tension and indication of foreign object detection. Other augmented images indicate device status overlays and instrument indication.
  • FIG. 10 illustrates a system 83 for augmenting images of a surgical field with information using an AR display 89, in accordance with at least one aspect of this disclosure.
  • the system 83 may be used to perform the techniques described hereinbelow, for example, by using the processor 85.
  • the system 83 includes one aspect of an AR device 66 that may be in communication with a database 93.
  • the AR device 66 includes a processor 85, memory 87, an AR display 89, and a camera 79.
  • the AR device 66 may include a sensor 90, a speaker 91 , and/or a haptic controller 92.
  • the database 93 may include image storage 94 or preoperative plan storage 95.
  • the processor 85 of the AR device 66 includes an augmented reality modeler 86.
  • the augmented reality modeler 86 may be used by the processor 85 to create the augmented reality environment.
  • the augmented reality modeler 86 may receive images of the instrument in a surgical field, such as from the camera 79 or sensor 90, and create the augmented reality environment to fit within a display image of the surgical field of view.
  • physical objects and/or date may be overlaid on the surgical field of view and/or the surgical instruments images and the augmented reality modeler 86 may use physical objects and data to present the augmented reality display of virtual object s and/or data in the augmented reality environment.
  • the augmented reality modeler 86 may use or detect an instrument at a surgical site of the patient and present a virtual object and/or data on the surgical instrument and/or an image of the surgical site in the surgical field of view captured by the camera 79.
  • the AR display 89 may display the AR environment overlaid on a real environment.
  • the display 89 may show a virtual object and/or data, using the AR device 66, such as in a fixed position in the AR environment.
  • the AR device 66 may include a sensor 90, such as an infrared sensor.
  • the camera 79 or the sensor 90 may be used to detect movement, such as a gesture by a surgeon or other user, that may be interpreted by the processor 85 as attempted or intended interaction by the user with the virtual target.
  • the processor 85 may identify an object in a real environment, such as through processing information received using the camera 79.
  • the sensor 90 may be a tactile, audible, chemical, or thermal sensor to generate corresponding signals that may combined with various data feeds to create the augmented environment.
  • the sensor 90 may include binaural audio sensors (spatial sound), inertial measurement (accelerometer, gyroscope, magnetometer) sensors, environmental sensors, depth camera sensors, hand and eye tracking sensors, and voice command recognition functions.
  • the AR display 89 may present, such as within a surgical field while permitting the surgical field to be viewed through the AR display 89, a virtual feature corresponding to a physical feature hidden by an anatomical aspect of a patient.
  • the virtual feature may have a virtual position or orientation corresponding to a first physical position or orientation of the physical feature.
  • the virtual position or orientation of the virtual feature may include an offset from the first physical position or orientation of the physical feature.
  • the offset may include a predetermined distance from the augmented reality display, a relative distance from the augmented reality display to the anatomical aspect, or the like.
  • the AR device 66 may be an individual AR device.
  • the AR device 66 may be a HoloLens 2 AR device manufactured by Microsoft of Redmond, Wash.
  • This AR device 66 includes a visor with lenses and binaural audio features (spatial sound), inertial measurement (accelerometer, gyroscope, magnetometer), environmental sensors, depth camera, and video camera, hand and eye tracking, and voice command recognition functions. It provides an improved field of view with high resolution by using mirrors to direct waveguides in front of wearer's eyes. Images can be enlarged by changing angles of mirrors. It also provides eye tracking to recognize users and adjust lens widths for specific users.
  • the AR device 66 may be a Snapchat Spectacles 3 AR device.
  • This AR device provides the ability to capture paired images and recreate 3D depth mapping, add in virtual effects, and replay 3D videos.
  • the AR device includes two HD cameras to capture 3D photos and videos at 60 fps — while four built-in microphones record immersive, high-fidelity audio. Images from both cameras combine to build out a geometric map of the real world around the user to provide a new sense of depth perception. Photos and videos may be wirelessly synchronized to external display devices.
  • the AR device 66 may be a Glass 2 AR device by Google. This AR device provides inertial measurement (accelerometer, gyroscope, magnetometer) information overlaid on lens (out of view) to supplement information.
  • the AR device 66 may be an Echo Frames AR device by Amazon. This AR device does not have cameras/displays. A microphone and speaker are linked to Alexa. This AR device provides less functionality than a heads-up display.
  • the AR device 66 may be a Focals AR device by North (Google). This AR device provides notification pusher/smartwatch analog; inertial measurement, screen overlay of information (weather, calendar, messages), voice control (Alexa) integration. This AR device provides basic heads-up display functionality.
  • the AR device 66 may be an Nreal AR device.
  • This AR device includes spatial sound, two environmental cameras, a photo camera, IMU (accelerometer, gyroscope), ambient light sensor, proximity sensor functionality.
  • IMU accelerometer, gyroscope
  • ambient light sensor proximity sensor functionality.
  • a nebula projects application information on lenses.
  • the AR device 66 may be any one of the following commercially available AR devices: Magic Leap 1 , Epson Moverio, Vuzix Blade AR, ZenFone AR, Microsoft AR glasses prototype, EyeT ap to create collinear light to that of the environment directly into the retina.
  • a beam splitter makes the same light seen by the eye available to the computer to process and overlay information, for example.
  • AR visualization systems include HUD, contact lenses , glasses, virtual reality (VR) headsets, virtual retinal display, on in operating room displays, and/or smart contact lenses (bionic lenses).
  • Multi-user interfaces for the AR device 66 include virtual retinal displays such as raster displays drawn directly on retinas instead of on a screen in front of the eye, smart televisions, smart phones, and / or spatial displays such as Sony spatial display systems.
  • AR technology may include, for example, AR capture devices and software applications, AR creation devices and software applications, and AR cloud devices and software applications.
  • AR capture devices and software applications include, for example, Apple Polycam app, Ubiquity 6 (Mirrorworld using Display. land app) - users can scan and get 3d image of real world (to create 3D model).
  • AR creation devices and software applications include, for example, Adobe Aero, Vuforia, ARToolKit, Google ARCore, Apple ARKit, MAXST, Aurasma, Zappar, Blippar.
  • AR cloud devices and software applications include, for example, Facebook, Google (world geometry, objection recognition, predictive data), Amazon AR Cloud (commerce), Microsoft Azure, Samsung Project Whare, Niantic, Magic Leap.
  • Situational awareness is the ability of some aspects of a surgical system to determine or infer information related to a surgical procedure from data received from databases and/or instruments.
  • the information can include the type of procedure being undertaken, the type of tissue being operated on, or the body cavity that is the subject of the procedure.
  • the surgical system can, for example, improve the manner in which it controls the modular devices (e.g., a robotic arm and/or robotic surgical tool) that are connected to it and provide contextualized information or suggestions to the surgeon during the course of the surgical procedure.
  • FIG. 11 illustrates a timeline of a situational awareness surgical procedure.
  • FIG. 11 illustrates a timeline 5200 of an illustrative surgical procedure and the contextual information that a surgical hub 5104 can derive from the data received from the data sources 5126 at each step in the surgical procedure.
  • the timeline 5200 depicts the typical steps that would be taken by the nurses, surgeons, and other medical personnel during the course of a lung segmentectomy procedure, beginning with setting up the operating theater and ending with transferring the patient to a post-operative recovery room.
  • the situationally aware surgical hub 5104 receives data from the data sources 5126 throughout the course of the surgical procedure, including data generated each time medical personnel utilize a modular device 5102 that is paired with the surgical hub 5104.
  • the surgical hub 5104 can receive this data from the paired modular devices 5102 and other data sources 5126 and continually derive inferences (i.e., contextual information) about the ongoing procedure as new data is received, such as which step of the procedure is being performed at any given time.
  • the situational awareness system of the surgical hub 5104 is able to, for example, record data pertaining to the procedure for generating reports, verify the steps being taken by the medical personnel, provide data or prompts (e.g., via a display screen) that may be pertinent for the particular procedural step, adjust modular devices 5102 based on the context (e.g., activate monitors, adjust the FOV of the medical imaging device, or change the energy level of an ultrasonic surgical instrument or RF electrosurgical instrument), and take any other such action described above.
  • the context e.g., activate monitors, adjust the FOV of the medical imaging device, or change the energy level of an ultrasonic surgical instrument or RF electrosurgical instrument
  • the hospital staff members retrieve the patient’s EMR from the hospital’s EMR database. Based on select patient data in the EMR, the surgical hub 5104 determines that the procedure to be performed is a thoracic procedure.
  • Second 5204 the staff members scan the incoming medical supplies for the procedure.
  • the surgical hub 5104 cross-references the scanned supplies with a list of supplies that are utilized in various types of procedures and confirms that the mix of supplies corresponds to a thoracic procedure. Further, the surgical hub 5104 is also able to determine that the procedure is not a wedge procedure (because the incoming supplies either lack certain supplies that are necessary for a thoracic wedge procedure or do not otherwise correspond to a thoracic wedge procedure).
  • the medical personnel scan the patient band via a scanner 5128 that is communicably connected to the surgical hub 5104.
  • the surgical hub 5104 can then confirm the patient’s identity based on the scanned data.
  • the medical staff turns on the auxiliary equipment.
  • the auxiliary equipment being utilized can vary according to the type of surgical procedure and the techniques to be used by the surgeon, but in this illustrative case they include a smoke evacuator, insufflator, and medical imaging device.
  • the auxiliary equipment that are modular devices 5102 can automatically pair with the surgical hub 5104 that is located within a particular vicinity of the modular devices 5102 as part of their initialization process.
  • the surgical hub 5104 can then derive contextual information about the surgical procedure by detecting the types of modular devices 5102 that pair with it during this pre-operative or initialization phase. In this particular example, the surgical hub 5104 determines that the surgical procedure is a VATS procedure based on this particular combination of paired modular devices 5102.
  • the surgical hub 5104 can generally infer the specific procedure that the surgical team will be performing. Once the surgical hub 5104 knows what specific procedure is being performed, the surgical hub 5104 can then retrieve the steps of that procedure from a memory or from the cloud and then cross-reference the data it subsequently receives from the connected data sources 5126 (e.g., modular devices 5102 and patient monitoring devices 5124) to infer what step of the surgical procedure the surgical team is performing.
  • the connected data sources 5126 e.g., modular devices 5102 and patient monitoring devices 5124
  • the staff members attach the EKG electrodes and other patient monitoring devices 5124 to the patient.
  • the EKG electrodes and other patient monitoring devices 5124 are able to pair with the surgical hub 5104.
  • the surgical hub 5104 begins receiving data from the patient monitoring devices 5124, the surgical hub 5104 thus confirms that the patient is in the operating theater.
  • the medical personnel induce anesthesia in the patient.
  • the surgical hub 5104 can infer that the patient is under anesthesia based on data from the modular devices 5102 and/or patient monitoring devices 5124, including EKG data, blood pressure data, ventilator data, or combinations.
  • the pre-operative portion of the lung segmentectomy procedure is completed and the operative portion begins.
  • the patient’s lung that is being operated on is collapsed (while ventilation is switched to the contralateral lung).
  • the surgical hub 5104 can infer from the ventilator data that the patient’s lung has been collapsed.
  • the surgical hub 5104 can infer that the operative portion of the procedure has commenced as it can compare the detection of the patient’s lung collapsing to the expected steps of the procedure (which can be accessed or retrieved previously) and thereby determine that collapsing the lung is the first operative step in this particular procedure.
  • the medical imaging device 5108 e.g., a scope
  • receives the medical imaging device data i.e., still image data or live streamed video in real time
  • the surgical hub 5104 can determine that the laparoscopic portion of the surgical procedure has commenced. Further, the surgical hub 5104 can determine that the particular procedure being performed is a segmentectomy, as opposed to a lobectomy (note that a wedge procedure has already been discounted by the surgical hub 5104 based on data received at the second step 5204 of the procedure).
  • the data from the medical imaging device 124 (FIG.
  • one technique for performing a VATS lobectomy places the camera in the lower anterior corner of the patient’s chest cavity above the diaphragm, whereas one technique for performing a VATS segmentectomy places the camera in an anterior intercostal position relative to the segmental fissure.
  • the situational awareness system can be trained to recognize the positioning of the medical imaging device according to the visualization of the patient’s anatomy.
  • one technique for performing a VATS lobectomy utilizes a single medical imaging device, whereas another technique for performing a VATS segmentectomy utilizes multiple cameras.
  • one technique for performing a VATS segmentectomy utilizes an infrared light source (which can be communicably coupled to the surgical hub as part of the visualization system) to visualize the segmental fissure, which is not utilized in a VATS lobectomy.
  • an infrared light source which can be communicably coupled to the surgical hub as part of the visualization system
  • the surgical hub 5104 can thereby determine the specific type of surgical procedure being performed and/or the technique being used for a particular type of surgical procedure.
  • the surgical hub 5104 can infer that the surgeon is in the process of dissecting to mobilize the patient’s lung because it receives data from the RF or ultrasonic generator indicating that an energy instrument is being fired.
  • the surgical hub 5104 can cross-reference the received data with the retrieved steps of the surgical procedure to determine that an energy instrument being fired at this point in the process (i.e. , after the completion of the previously discussed steps of the procedure) corresponds to the dissection step.
  • the surgical team proceeds to the ligation step of the procedure.
  • the surgical hub 5104 can infer that the surgeon is ligating arteries and veins because it receives data from the surgical stapling and cutting instrument indicating that the instrument is being fired. Similarly to the prior step, the surgical hub 5104 can derive this inference by cross- referencing the receipt of data from the surgical stapling and cutting instrument with the retrieved steps in the process.
  • the segmentectomy portion of the procedure is performed.
  • the surgical hub 5104 infers that the surgeon is transecting the parenchyma based on data from the surgical instrument, including data from a staple cartridge.
  • the cartridge data may correspond to size or type of staple being fired by the instrument.
  • the cartridge data can indicate the type of tissue being stapled and/or transected for different types of staples utilized in different types of tissues.
  • the type of staple being fired is utilized for parenchyma or other tissue types to allow the surgical hub 5104 to infer that the segmentectomy procedure is being performed.
  • the node dissection step is then performed.
  • the surgical hub 5104 can infer that the surgical team is dissecting the node and performing a leak test based on data received from the generator indicating that an RF or ultrasonic instrument is being fired.
  • an RF or ultrasonic instrument being utilized after parenchyma was transected corresponds to the node dissection step, which allows the surgical hub 5104 to make this inference.
  • surgeons regularly switch back and forth between surgical stapling/cutting instruments and surgical energy (i.e. , RF or ultrasonic) instruments depending upon the particular step in the procedure because different instruments are better adapted for particular tasks.
  • the particular sequence in which the stapling/cutting instruments and surgical energy instruments are used can indicate what step of the procedure the surgeon is performing.
  • the incisions and closed up and the post operative portion of the procedure begins.
  • the surgical hub 5104 can infer that the patient is emerging from the anesthesia based on the ventilator data (i.e., the patient’s breathing rate begins increasing), for example.
  • the surgical hub 5104 can thus infer that the patient is being transferred to a recovery room when the hub loses EKG, BP, and other data from the patient monitoring devices 5124.
  • the surgical hub 5104 can determine or infer when each step of a given surgical procedure is taking place according to data received from the various data sources 5126 that are communicably coupled to the surgical hub 5104.
  • the patient data can also be utilized by a situationally aware surgical hub 5104 to generate control adjustments for the paired modular devices 5102.
  • the present disclosure describes a method and system for tracking tissue, identifying marked areas of interest, and generate virtual element indicative of the areas of interest in an augmented reality environment.
  • the patient may be virtually or physically tagged with fiducial markers to aid the surgeon in an operation.
  • a surgical procedure may require that a patient undergoes a pre-operative fiducial marking process.
  • FIG. 12 shows an example of a structural surface 18000 comprising a plurality of fiducial markers.
  • a plurality of points 18010a-d are identified and tagged to the structural surface 18000.
  • a computing system such as remote server 63 (FIG. 5) or the surgical hub 56 (FIG. 6) evaluates the structural surface and assigns fiducial marker 18010a-d, based on a target registration error (TRE) model 18008.
  • the TRE model uses a sample data set to estimate the placement of fiducial markers 18010a-d.
  • 18002 show an initial view of the structure surface 18000.
  • 18004 shows the TRE model representation of the structural surface 18000.
  • Transformation 18006 shows the resultant placement of fiducial markers 18010a-d of the structural surface 18000.
  • FIG. 13 shows a process 18020 for surface matching external structure of a patient with fiducial markers.
  • a computing system such as remote server 63 (FIG. 5) or the surgical hub 56 (FIG. 6) system generates 18022 an initial mapping of a surface based on a digital representation of the surface.
  • the system uses facial recognition features such as eyes and the bridge of the nose to recognize 18024 a plurality of anatomic landmarks 18028 to map the surface and distance between points. Based on the determined distance between anatomic landmarks 18028, the system generates 18026 the fiducial markers 18030.
  • FIG. 14 shows a process 18040 for surface matching internal structure 18044 of a patient with fiducial markers 18042.
  • the internal structure is displayed on an output display 18046 which allows a technician to tag areas of the internal structure 18044 with a stylus 18048.
  • the fiducial markers 18042 may correspond to a path or the location of the surgical procedure.
  • FIGS. 15-17 show external surgical alignment instruments to aid a surgeon in a surgical procedure.
  • FIG. 15 shows a stereotactic frame 18060
  • FIG. 16 shows a starfix platform 18080
  • FIG. 17 shows a microtable 18100.
  • the surgical hub 56 (FIG. 6) may register and catalog the dimensions of the external surgical aids and assign fiducial markers to a plurality of points on the surgical aids.
  • the surgical hub 56 may then align fiducial markers of the external surgical alignment instruments with surface markers on the patient.
  • the surgical hub 56 (FIG. 6) is configured to track the location and position of surgical instruments 12 (FIGS. 1 , 2), 21 (FIGS. 5, 6) in the operating room 16 (FIG. 2).
  • the surgical instrument 12, 21 may comprise a plurality of fiducial markers, strategically placed on the external housing of the instrument 12, 21 to convey the special parameters of the instrument 12, 21.
  • the fiducial markers may be used by a passive tracking system, in combination with an infrared (IR) light source.
  • the surgical instrument 12, 21 may comprise sensors that indicate when the surgical instrument 12, 21 is in use and when then surgical instrument 12, 21 in inside the patient.
  • a trocar may comprises one or more internal patient sensors that indicate when the device is inserted into the body cavity.
  • the trocar may automatically or manually detect and track the location of another surgical instrument 12, 21 .
  • the trocar may comprise an internal camera system that identifies fiducial markers on other surgical instruments 12, 21.
  • the internal camera system may receive commands to locate an end effector and tag an end effector with a marker in relation to the tip of the trocar.
  • the tag provides a registration point that may be used to monitor the tip of the end effector throughout the surgical procedure, and may be associated with a virtual element that is rendered on an AR display 89 of an AR device 66 (FIG. 10). In situations when the tip of the end effector may be out of view of the internal camera system, the virtual element may continuously display the tracked position of the end effector tip, based on the tag.
  • FIG. 18 shows a flow diagram for identifying objects based on a plurality of a registration parameters.
  • the surgical hub receives physical characteristic of an object from one or more object detection cameras in an operating room. Camera provides 18122 raw imaging data of an object. The surgical hub performs 18124 image processing to remove the back of the image or fame and perform edge detection. Once the surgical hub performs the image processing, the surgical hub compares 18126 the detected object to a catalog of objects. The surgical hub reviews 18128 the properties of each object and determines 18130 the object the most closely fits the identified parameters of the image.
  • FIG. 19 shows a flow diagram 18140 for classifying unknown surgical instruments based on a partial information of known and unknown parameters.
  • An object recognition system may implemented by the remote server 63 (FIG. 5) or the surgical hub 56 (FIG. 6). The object recognition system may be unable to affirmatively determine an object, but may have it narrowed down to a plurality of candidates.
  • the system inputs 18142 partial object information and uses known parameters to evaluate the object. If the system determines physical characteristics of the object, it can perform 18144 the geometric-uncertain analysis. If the system determines geometric characteristics of the object, the system can perform 18146 the physical-uncertain analysis 18146. However, if the system does not have enough information, the system may require manual identification, and classifies the object as unknown 18148.
  • the surgical hub receives spatial and physical parameters associated with an operating room or an external environment.
  • the physical parameters may be registered to a specific room or environment.
  • an external environment may be classified according to certain features such as a sterile or non-sterile environment; pre-op, OR, or post-op room; and specific equipment in the room (e.g. MRI, CT scanner).
  • Trocar Comprising An Outer Diameter Mounted Camera System Keeps Port Free For Instruments And Increases Field Of View
  • the present disclosure further describes a camera system integrated into a trocar.
  • the camera system allows for wide field of view of an internal surgical site and 3D mapping of fiducial markers during a laparoscopic procedure.
  • the camera system Upon entry into the patient, the camera system is configured to deploy from a recessed position at the distal end of the trocar.
  • the internal camera system is configured to keep the trocar ports free for surgical instruments and provide the surgical staff members with an increased view of the surgical environment.
  • FIG. 20 shows a trocar 18160 comprising an internal camera system 18166.
  • the internal camera system 18166 comprises a plurality of cameras 18166a-n connected together with an elastic member 18168.
  • the elastic connection 18168 allows the camera system 18166 to collapse together and fit through a passage defined in the center of the trocar.
  • the camera system 18166 emits light in the non-visible spectrum allowing the cameras 18166a-n to detect various types of fiducial markers (e.g., IR fiducial markers).
  • the camera system 18166 attaches to the outer diameter 18170 of the distal end 18162 of the trocar 18160.
  • the internal camera system 18166 When the trocar is inserted and removed from the body cavity of the patient, the internal camera system 18166 is in the retracted position 18160b. In the retracted position 18160b, the camera system 18166 recesses and attaches to the inner diameter 18172 of the distal end 18162 of the trocar 18160.
  • FIG. 21 shows a reusable installation tool 18176, configured to insert into the proximal end 18164 of the trocar 18160, deploy and retract the camera system 18166 around the outer diameter 18170 of the trocar 18160.
  • the camera system 18166 is communicably coupled to a surgical hub 56 (FIG. 6) through a wired or wireless connection.
  • each camera 18166a-n may have its own power supply (e.g., rechargeable battery), the camera system 18166 may have a single external power supply that connects to each camera 18166a-n through an elastically deformable wired connection.
  • the wired connection may be inserted externally to the trocar 18160, while the trocar 18160 is inserted, keeping the inner diameter of the trocar 18160 free for surgical instruments.
  • the camera system 18160 may attach to the outer diameter of the trocar 18160 through a squeeze and friction of elastic connection, magnet in the case of a metal trocar, or a separate connection on the outer diameter of the trocar 18160.
  • FIG. 21 also shows a profile view 18178a of the installation plunger 18174a in a full depressed position.
  • a conical distal end 18158 of the trocar 18160 releases the camera system 18166.
  • the plunger 18174b is pulled in the proximal direction which causes the conical distal end 18158 to force the camera system along the outer diameter 18170 of the trocar 18160.
  • the camera system 18166 attaches to the outer diameter 18170 of the trocar 18160.
  • the reusable installation tool 18176 is removed so that a laparoscopic surgical procedure may begin.
  • the present disclosure further describes a system configured to generate a 3D model for a surgeon to navigate through the internal tissue structure of a patient.
  • the system identifies and marks target tissue or structure of interest in a pre-operative CT-scan.
  • the system generates an initial 3D-model based on the CT-scan that is used by the surgeon to aid in their navigation of internal structure.
  • the 3D-model may be continuously updated in real-time based on additional data points received intra-operatively.
  • the system may determine the proximity of distance from a surgical instrument, and update the model to reflect the tissue movement or change in tissue location.
  • the system generates a 3D rendering of the internal tissue structure with virtual elements, and displays the 3D-model on an augmented reality display.
  • the system may generate a live feed of the surgical environment or provide virtual elements, overlaid on top of a real world live feed of the surgical site.
  • the 3D-model indicates areas of interest, areas to avoid. Additionally, the markers can indicate tissue that needs to be sealed or tissue that is difficult to find, such as pulmonary veins and pulmonary arteries.
  • FIG. 22 shows a plurality fiducial markers 18180 tagged to areas of interest, in a pre op CT scan.
  • the fiducial markers may be placed to create a centroid at a critical structure 18182.
  • a centroid value is determined based on the relative distance between each of the fiducial markers in a set.
  • FIG. 23 shows a laparoscopic surgical procedure that utilizes a plurality of fiducial markers to aid a surgeon in locating a surgical site.
  • the pre-operative determination of a critical structure 18184 is generally a close approximation of the structure location, but may not be the exact location.
  • an internal camera may be used in conjunction with the fiducial markers to provide a real-time updated location of critical structure 18186, with an updated model or updated fiducial marker.
  • fiducial markers are located on the surgical instrument 18190 and help provide an updated location 18186 based on a relationship between other points.
  • the surgical instrument may further comprise an integrated mapping sensor 18188.
  • the present disclosure further describes various methods and systems for marking and tracking tissue movement with physical markers.
  • the tracking system comprises a camera system configured to detect and track the physical markers.
  • the physical markers comprise magnetic ink, visible ink at visible light spectrums, invisible ink at invisible light spectrums, or other detectable ink by a camera system.
  • FIG. 24 shows a physical marker applied by injection into a vascular system of a patient with an indocyanine dye 18202.
  • the dye 18202 is illuminated by a light source 18204 which allows a camera 18206 to capture and record the vascular structure of the tissue 18210.
  • the light source 18204 may be a fluorescent light source.
  • the camera system 18206 may use various light frequencies or lasers to visualize the dye 18202.
  • the camera system 18206 is further configured to identify various overlaying paths of the ink and display a 3D rendering on an output display 18208.
  • vasculature can be used like a fingerprint to unique identify structure and track the structure as it moves.
  • pre-op CT imaging may be used to help the system generate a 3D map of the structure.
  • the dye also may be used to track organs and warn the surgical staff if they are about to grasp highly vascularized tissue.
  • FIG. 25 further shows exemplary tissue injected with a dye and illuminated to show vasculature.
  • the light source 18204 may emit light at a wavelength outside of the visible spectrum such as IR.
  • the dye 18202 may comprise magnetic ink used as a marker to distinguish areas of interest inside and outside the field of view of the camera 18206.
  • the dye 18202 may be splatter sprayed in a surgical area in a no-visible spectrum, such that the body can easily absorb the dye 18202. The splatter creates a unique pattern that allows the camera 18206 to easily track the location and movement of the tissue 18210.
  • FIG. 26 shows a system 18300 configured to monitor the change in pressure or fluid in a body cavity according to an impendence measurement by a probe 18302.
  • the probe 18302 is coupled to a surgical instrument 18304 and measures an impedance value based on a pressure created by fluid and/or gas in a body cavity 18306.
  • a surgical hub 56 (FIG.
  • the probe 18302 is configured to maintain a fixed gap to measure a change in pressure at a potential leak site 18308.
  • the surgical hub 56 may notify the surgical staff members of a detected leak by generating a warning through AR content.
  • a virtual element may be rendered on an AR display 89 of an AR device 66 (FIG. 10) that anchors to the location of the detected event.
  • the virtual element may be identified with a contrasting color that is solid, flashing, or semi-transparent.
  • the virtual element may be accompanied with a textual warning that identifies the type of event and/or the severity of the event.
  • FIG. 27 shows an infrared (IR) heat detection system 18310 comprising an IR camera system 18312 configured to direct IR light 18314 on a treated region of tissue 18316 and identify a temperature difference in a surgical environment 18304.
  • the IR camera system 18312 may be configure to identify the location of a leak in a pressurized cavity due to the change in temperature of air surrounding the leak.
  • the body cavity is pressurized with air at a temperature that is colder or warmer than the fluid in the peritoneal cavity.
  • the IR camera system 18312 will observe a leak by seeing a gas at a temperature that is warmer or colder than the cavity.
  • the surgical hub 56 may notify the surgical staff members.
  • the IR camera system 18312 may determine that an area of tissue was recently sealed.
  • the sealed tissue may be at a different temperature and allows the IR camera system 18312 to distinguish the sealed tissue as a sensitive treated area.
  • the surgical hub 56 (FIG. 6) may render a virtual element that overlays on top of the treated area to indicate to a surgeon that the sealed tissue was recently treated and is a sensitive area.
  • the sealed tissue is identified based on a predetermined tissue temperature threshold at the time the tissue was sealed.
  • the tissue temperature may slowly cool however, the IR camera system 18312 may mark the region with a non-fixed marker that is maintained even after the tissue temperature drops below the initial threshold temperature.
  • the present disclosure further describes a tissue tracking system to prevent excess tension from being exerted on tissue.
  • the system is configured to track markers applied to tissue at specific location that indicate motion, force, and tension.
  • a surgical hub 56 (FIG. 6) may continuously monitor tissue tension and motion parameters. The surgical hub 56 may determine that tissue tension at a specific location has reached a predetermined threshold, and provides a notification to one or more AR devices 66 (FIG. 10).
  • FIG. 28 shows a surgical procedure 18350 employing three end effectors 18352, 18354, 18356 configured to grasp and transect tissue 18360.
  • the tissue 18360 is initially grasped at two points by a first end effector 18352 and a second end effector 18356.
  • An AR device 66 (FIG. 10) may indicate the initial positions that the end effectors 18352, 18356 should be located.
  • the initial distance 18358 between the first end effector 18352 and the second end effector 18356 is determined based on a force applied to the tissue 18360.
  • the second end effector 18356 is configured to transect the tissue 18360 and requires a third end effector 18354 to compensate for an increase in tissue tension.
  • FIG. 29 shows the third end effector 18354 sliding along the tissue from a first position 18354a to a second position 18354b.
  • the surgical staff members monitor the position of the third end effector 18354 so that it is properly positioned to compensate for increase in tension.
  • an AR device 66 (FIG. 10) may highlight the tissue 18360 when it exceeds a predetermine tension threshold.
  • the surgeon may reposition third end effector 18354 so that the tissue 18360 is no longer highlighted and indicates the tissue tension is back within the predetermine threshold.
  • the surgeon may receive feedback in a handle or joy stick to indicate when need to reposition and when the new position is satisfactory.
  • FIG. 30 shows the third end effector 18354 positioned adjacent to the second end effector 18356. Upon transection of the tissue 18360 by the second end effector, the tissue tension will be within a predetermined threshold, based on the initial distance 18358 (FIG. 28).
  • FIG. 31 shows a surgical procedure 18370 employing three static clamps 18372, 18274, 18378 and a dynamic clamp 18376 configured to transfer tissue between stationary clamps 18372, 18274, 18378.
  • a first clamp 18372 is a stationary clamp configured to hold the end of the tissue 18386 and prevent tension or pulling from beyond a region of interest 18382.
  • a second clamp 18374 and a third clamp 18378 are positioned according to a predetermined distance such that the tissue maintains a predetermined tension.
  • the second clamp 18374 and the third clamp 18378 are stationary but may open and close to pull new tissue 18386 between the stationary distance 18380.
  • a fourth clamp 18376 is a dynamic clamp and is configured to pull the tissue 18386 between the second 18374 and third clamp 18378, and reduce the tension between the first 18372 and second clamp 18374.
  • the fourth clamp 18376 repositions the tissue 18386 to reduce excess tension at 18384, indicated by a graphical highlight.
  • An AR device 66 (FIG. 10) may provide a similar highlight to indicate excess tissue tension.
  • FIG. 32 shows a logic flow diagram of a method 18400 for displaying a surgical location inside of a patient.
  • a surgical hub 56 receives 18402 a video feed from one or more cameras located inside of a patient.
  • the surgical hub 56 identifies 18404 one or more physical markers inside the patient.
  • the surgical hub 56 determines 18406 a target location based on the relationship to the one or more physical markers.
  • the surgical hub 56 generates 18408 a virtual element corresponding to the target location.
  • An AR device 66 (FIG. 10) coupled to the surgical hub 56 (FIG. 6) displays 18410 the virtual element overlaid on the video feed on an augmented reality (AR) display 79 (FIG. 10).
  • AR augmented reality
  • the video feed is a wide angle view stitched together from at least two video feeds.
  • one or more physical markers is visible under the illumination of a light source in outside of the visible spectrum.
  • the one or more physical markers is a fiducial marker assigned in a pre-operative computerized tomography (CT) scan.
  • CT computerized tomography
  • the target location is continuously updated on an augmented reality (AR) device 66 (FIG. 10) in real-time.
  • AR augmented reality
  • Example 1 A surgical system comprising: a surgical device comprising: an axial passage defining an outer diameter and an inner diameter; a proximal end; a distal end configured to penetrate tissue; a camera array comprising individual cameras connected in a ring configuration with an elastic connection; a removable installation trigger configured to extend the camera array from a first recessed position from the inner diameter of the distal end of the axial passage to a second deployed position with the camera array circumferentially positioned around the outer diameter of the distal end of the axial passage; an augmented reality (AR) device; and a surgical hub communicatively coupled to the camera array and the AR device, wherein the surgical hub comprises a control circuit coupled to a memory, and wherein the control circuit is configured to: receive a plurality of video feed from the camera array; identify a physical marker on the video feed; and display the physical marker on the AR display.
  • AR augmented reality
  • Example 2 The surgical system of Example 1 , wherein the video feed is a wide angle view stitched together from each of the individual cameras.
  • Example 3 The surgical system of any one of Examples 1-2, wherein the physical marker is visible under the illumination of a light source in outside of the visible spectrum.
  • Example 4 The surgical system of any one of Examples 1-3, wherein the physical marker is a dye.
  • Example 5 The surgical system of any one of Examples 1-4, wherein the physical marker is a fiducial marker assigned in a pre-operative computerized tomography (CT) scan.
  • CT computerized tomography
  • Example 6 The surgical system of any one of Examples 1-5, wherein the physical marker is configured to indicate a target location of a surgical procedure.
  • Example 7 The surgical system of Example 6, wherein the target location is continuously updated on the AR device in real-time.
  • Example 8 The surgical system of Example 7, wherein the target location is updated in based on the relationship of a surgical instrument and the physical marker.
  • Example 9 A surgical device comprising: a camera array comprising individual cameras connected in a ring configuration with an elastic connection, wherein the camera array is communicatively couplable to a surgical hub; an elongated penetration member having a proximal end and a distal end, wherein the distal end further comprises a tissue penetrating tip; an axial passage through the elongated penetration member and the tissue penetrating tip, and wherein an inner diameter of the axial passage is sized to house the camera array in a first recessed position; and a removable installation trigger configured to extend the camera array from a first recessed position from the inner diameter of the distal end of the elongated penetration member to a second deployed position with the camera array circumferentially positioned around an outer diameter of the distal end of the elongated penetration member.
  • Example 10 The surgical device of Example 9, wherein the camera array is communicatively couplable with the surgical hub though a wireless communication protocol.
  • Example 11 The surgical device of Example 10, wherein the camera array is powered by a rechargeable batteries.
  • Example 12 The surgical device of any one of Examples 9-11 , wherein the camera array is communicatively couplable to the surgical hub through a wired communication protocol.
  • Example 13 The surgical device of Example 12, wherein the camera array is powered by a wired external power source comprising a wire extending along the outer diameter of the elongated penetration member.
  • Example 14 The surgical device of any one of Examples 9-13, wherein the camera array is attached to the outer diameter of the distal end of the elongated penetration member in a squeeze and friction configuration.
  • Example 15 The surgical device of any one of Examples 9-14, wherein the camera array does not occupy space of the inner diameter of the axial passage in the second deployed position.
  • Example 16 A method for displaying a surgical location inside of a patient, the method comprising: receiving, by a surgical hub, a video feed from a camera located inside a patient; identifying, by the surgical hub, a physical marker inside of the patient; determining, by the surgical hub, a target location based on the relationship to the physical marker; generating, by the surgical hub, a virtual element corresponding to the target location; and displaying, by an augmented reality (AR) device coupled to the surgical hub, the virtual element overlaid on the video feed on an AR display.
  • AR augmented reality
  • Example 17 The method of Example 16, wherein the video feed is a wide angle view stitched together from at least two video feeds.
  • Example 18 The method of any one of Examples 16-17, wherein the physical marker is visible under the illumination of a light source in outside of the visible spectrum.
  • Example 19 The method of any one of Examples 16-18, wherein the physical marker is a fiducial marker assigned in a pre-operative computerized tomography (CT) scan.
  • Example 20 The method of any one of Examples 16-19, wherein the target location is continuously updated on an augmented reality (AR) device in real-time.
  • CT computerized tomography
  • Instructions used to program logic to perform various disclosed aspects can be stored within a memory in the system, such as dynamic random access memory (DRAM), cache, flash memory, or other storage. Furthermore, the instructions can be distributed via a network or by way of other computer readable media.
  • DRAM dynamic random access memory
  • cache cache
  • flash memory or other storage.
  • the instructions can be distributed via a network or by way of other computer readable media.
  • a machine-readable medium may include any mechanism for storing or transmitting information in a form readable by a machine (e.g., a computer), but is not limited to, floppy diskettes, optical disks, compact disc, read-only memory (CD-ROMs), and magneto-optical disks, read-only memory (ROMs), random access memory (RAM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), magnetic or optical cards, flash memory, or a tangible, machine-readable storage used in the transmission of information over the Internet via electrical, optical, acoustical or other forms of propagated signals (e.g., carrier waves, infrared signals, digital signals, etc.).
  • the non-transitory computer-readable medium includes any type of tangible machine-readable medium suitable for storing or transmitting electronic instructions or information in a form readable by a machine (e.g., a computer).
  • control circuit may refer to, for example, hardwired circuitry, programmable circuitry (e.g., a computer processor including one or more individual instruction processing cores, processing unit, processor, microcontroller, microcontroller unit, controller, digital signal processor (DSP), programmable logic device (PLD), programmable logic array (PLA), or field programmable gate array (FPGA)), state machine circuitry, firmware that stores instructions executed by programmable circuitry, and any combination thereof.
  • programmable circuitry e.g., a computer processor including one or more individual instruction processing cores, processing unit, processor, microcontroller, microcontroller unit, controller, digital signal processor (DSP), programmable logic device (PLD), programmable logic array (PLA), or field programmable gate array (FPGA)
  • state machine circuitry firmware that stores instructions executed by programmable circuitry, and any combination thereof.
  • the control circuit may, collectively or individually, be embodied as circuitry that forms part of a larger system, for example, an integrated circuit (IC), an application-specific integrated circuit (ASIC), a system on-chip (SoC), desktop computers, laptop computers, tablet computers, servers, smart phones, etc.
  • IC integrated circuit
  • ASIC application-specific integrated circuit
  • SoC system on-chip
  • control circuit includes, but is not limited to, electrical circuitry having at least one discrete electrical circuit, electrical circuitry having at least one integrated circuit, electrical circuitry having at least one application specific integrated circuit, electrical circuitry forming a general purpose computing device configured by a computer program (e.g., a general purpose computer configured by a computer program which at least partially carries out processes and/or devices described herein, or a microprocessor configured by a computer program which at least partially carries out processes and/or devices described herein), electrical circuitry forming a memory device (e.g., forms of random access memory), and/or electrical circuitry forming a communications device (e.g., a modem, communications switch, or optical-electrical equipment).
  • a computer program e.g., a general purpose computer configured by a computer program which at least partially carries out processes and/or devices described herein, or a microprocessor configured by a computer program which at least partially carries out processes and/or devices described herein
  • electrical circuitry forming a memory device
  • logic may refer to an app, software, firmware and/or circuitry configured to perform any of the aforementioned operations.
  • Software may be embodied as a software package, code, instructions, instruction sets and/or data recorded on non-transitory computer readable storage medium.
  • Firmware may be embodied as code, instructions or instruction sets and/or data that are hard-coded (e.g., nonvolatile) in memory devices.
  • the terms “component,” “system,” “module” and the like can refer to a control circuit, computer-related entity, either hardware, a combination of hardware and software, software, or software in execution.
  • an “algorithm” refers to a self-consistent sequence of steps leading to a desired result, where a “step” refers to a manipulation of physical quantities and/or logic states which may, though need not necessarily, take the form of electrical or magnetic signals capable of being stored, transferred, combined, compared, and otherwise manipulated. It is common usage to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, or the like. These and similar terms may be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities and/or states.
  • a network may include a packet switched network.
  • the communication devices may be capable of communicating with each other using a selected packet switched network communications protocol.
  • One example communications protocol may include an Ethernet communications protocol which may be capable permitting communication using a Transmission Control Protocol/Internet Protocol (TCP/IP).
  • TCP/IP Transmission Control Protocol/Internet Protocol
  • the Ethernet protocol may comply or be compatible with the Ethernet standard published by the Institute of Electrical and Electronics Engineers (IEEE) titled “IEEE 802.3 Standard”, published in December, 2008 and/or later versions of this standard.
  • the communication devices may be capable of communicating with each other using an X.25 communications protocol.
  • the X.25 communications protocol may comply or be compatible with a standard promulgated by the International Telecommunication Union-Telecommunication Standardization Sector (ITU-T).
  • the communication devices may be capable of communicating with each other using a frame relay communications protocol.
  • the frame relay communications protocol may comply or be compatible with a standard promulgated by Consultative Committee for International Circuit and Telephone (CCITT) and/or the American National Standards Institute (ANSI).
  • the transceivers may be capable of communicating with each other using an Asynchronous Transfer Mode (ATM) communications protocol.
  • ATM Asynchronous Transfer Mode
  • the ATM communications protocol may comply or be compatible with an ATM standard published by the ATM Forum titled “ATM-MPLS Network Interworking 2.0” published August 2001 , and/or later versions of this standard.
  • ATM-MPLS Network Interworking 2.0 published August 2001
  • One or more components may be referred to herein as “configured to,” “configurable to,” “operable/operative to,” “adapted/adaptable,” “able to,” “conformable/conformed to,” etc.
  • “configured to” can generally encompass active-state components and/or inactive-state components and/or standby-state components, unless context requires otherwise.
  • proximal and distal are used herein with reference to a clinician manipulating the handle portion of the surgical instrument.
  • proximal refers to the portion closest to the clinician and the term “distal” refers to the portion located away from the clinician.
  • distal refers to the portion located away from the clinician.
  • spatial terms such as “vertical”, “horizontal”, “up”, and “down” may be used herein with respect to the drawings.
  • surgical instruments are used in many orientations and positions, and these terms are not intended to be limiting and/or absolute.
  • any reference to “one aspect,” “an aspect,” “an exemplification,” “one exemplification,” and the like means that a particular feature, structure, or characteristic described in connection with the aspect is included in at least one aspect.
  • appearances of the phrases “in one aspect,” “in an aspect,” “in an exemplification,” and “in one exemplification” in various places throughout the specification are not necessarily all referring to the same aspect.
  • the particular features, structures or characteristics may be combined in any suitable manner in one or more aspects.

Landscapes

  • Health & Medical Sciences (AREA)
  • Surgery (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Medical Informatics (AREA)
  • Animal Behavior & Ethology (AREA)
  • Engineering & Computer Science (AREA)
  • Biomedical Technology (AREA)
  • Heart & Thoracic Surgery (AREA)
  • Pathology (AREA)
  • Molecular Biology (AREA)
  • Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
  • General Health & Medical Sciences (AREA)
  • Public Health (AREA)
  • Veterinary Medicine (AREA)
  • Oral & Maxillofacial Surgery (AREA)
  • Surgical Instruments (AREA)
  • Endoscopes (AREA)

Abstract

L'invention concerne un système de caméra intégré dans un trocart. Le système de caméra permet un large champ de vision d'un site chirurgical interne et un mappage 3D de marqueurs de repère pendant une procédure laparoscopique. Lors de son introduction à l'intérieur du patient, le système de caméra est configuré pour se déployer depuis une position en retrait au niveau de l'extrémité distale du trocart. Dans divers aspects, le système de caméra interne est configuré pour maintenir les orifices de trocart libres pour des instruments chirurgicaux et fournir aux membres du personnel chirurgical une vue améliorée de l'environnement chirurgical.
EP22718802.6A 2021-04-14 2022-04-11 Système comprenant une matrice de caméras déployables hors d'un canal d'un dispositif chirurgical pénétrant un tissu Pending EP4178473A1 (fr)

Applications Claiming Priority (4)

Application Number Priority Date Filing Date Title
US202163174674P 2021-04-14 2021-04-14
US202163284326P 2021-11-30 2021-11-30
US17/688,660 US20220331052A1 (en) 2021-04-14 2022-03-07 Cooperation among multiple display systems to provide a healthcare user customized information
PCT/IB2022/053375 WO2022219501A1 (fr) 2021-04-14 2022-04-11 Système comprenant une matrice de caméras déployables hors d'un canal d'un dispositif chirurgical pénétrant un tissu

Publications (1)

Publication Number Publication Date
EP4178473A1 true EP4178473A1 (fr) 2023-05-17

Family

ID=81386558

Family Applications (1)

Application Number Title Priority Date Filing Date
EP22718802.6A Pending EP4178473A1 (fr) 2021-04-14 2022-04-11 Système comprenant une matrice de caméras déployables hors d'un canal d'un dispositif chirurgical pénétrant un tissu

Country Status (4)

Country Link
EP (1) EP4178473A1 (fr)
JP (1) JP2024513992A (fr)
BR (1) BR112023021416A2 (fr)
WO (1) WO2022219501A1 (fr)

Family Cites Families (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8504136B1 (en) * 2009-10-06 2013-08-06 University Of South Florida See-through abdomen display for minimally invasive surgery
US9427255B2 (en) * 2012-05-14 2016-08-30 Ethicon Endo-Surgery, Inc. Apparatus for introducing a steerable camera assembly into a patient
JP2014132979A (ja) * 2013-01-10 2014-07-24 Advanced Healthcare Kk トロカールおよび手術支援システム
USRE49930E1 (en) * 2015-03-26 2024-04-23 Universidade De Coimbra Methods and systems for computer-aided surgery using intra-operative video acquired by a free moving camera
EP3387982B1 (fr) * 2015-12-07 2020-09-23 Kyocera Corporation Trocart
US20190069957A1 (en) * 2017-09-06 2019-03-07 Verily Life Sciences Llc Surgical recognition system
US11969142B2 (en) 2017-12-28 2024-04-30 Cilag Gmbh International Method of compressing tissue within a stapling device and simultaneously displaying the location of the tissue within the jaws
US11589888B2 (en) 2017-12-28 2023-02-28 Cilag Gmbh International Method for controlling smart energy devices
WO2019215354A1 (fr) * 2018-05-09 2019-11-14 Andres Del Amor Arroyo Tristan Tête de vidéocaméra endoscopique annexable à un protecteur de plaie chirurgicale, sans tube rigide ni fixation manuelle
CN113038886B (zh) * 2018-09-18 2023-11-21 布莱阿姆青年大学 可展和可折叠的轴展开机构
US11937885B2 (en) * 2019-03-04 2024-03-26 Smith & Nephew, Inc. Co-registration for augmented reality and surgical navigation

Also Published As

Publication number Publication date
JP2024513992A (ja) 2024-03-27
BR112023021416A2 (pt) 2023-12-19
WO2022219501A1 (fr) 2022-10-20

Similar Documents

Publication Publication Date Title
US20220331052A1 (en) Cooperation among multiple display systems to provide a healthcare user customized information
WO2022219501A1 (fr) Système comprenant une matrice de caméras déployables hors d'un canal d'un dispositif chirurgical pénétrant un tissu
US20230404572A1 (en) Smart circular staplers
US20230395250A1 (en) Customization, troubleshooting, and wireless pairing techniques for surgical instruments
CN117479896A (zh) 包括可在组织穿透外科装置的通道外展开的相机阵列的系统
WO2022219504A1 (fr) Recouvrements coopératifs d'instruments en interaction qui conduisent à l'exécution des deux recouvrements
EP4136651A1 (fr) Mélange directement visualisé avec des éléments rendus pour afficher des éléments mélangés et des actions se produisant sur écran et hors écran
WO2022219500A1 (fr) Systèmes et procédés pour changer un recouvrement d'affichage d'une vue de champ opératoire sur la base d'événements déclencheurs
WO2022219491A1 (fr) Système et procédé de suivi d'une partie de l'utilisateur en tant que substitut pour instrument non surveillé
WO2023242762A1 (fr) Agrafeuses circulaires intelligentes
WO2022219488A1 (fr) Systèmes et procédés de commande de recouvrement de données chirurgicales
CN117461093A (zh) 基于触发事件来改变外科视野的显示叠加项的系统和方法
JP2024514884A (ja) 外科用システムのためのオーバーレイ器具情報の適合及び調整可能性
EP4356396A1 (fr) Techniques de personnalisation, de dépannage et d'appariement sans fil pour instruments chirurgicaux
CN117441212A (zh) 用渲染元素将混合直接可视化以显示混合元素以及在屏幕上和在屏幕外正在发生的动作
JP2024517603A (ja) 手術視野における選択的かつ調節可能な複合現実オーバーレイ
CN117480562A (zh) 外科视野中选择性和可调节的混合现实叠加
WO2022219494A1 (fr) Affichage peropératoire pour systèmes chirurgicaux

Legal Events

Date Code Title Description
STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: UNKNOWN

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE INTERNATIONAL PUBLICATION HAS BEEN MADE

PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE

17P Request for examination filed

Effective date: 20230213

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: EXAMINATION IS IN PROGRESS

17Q First examination report despatched

Effective date: 20231208