JP2024514884A - Adaptability and tunability of overlay instrument information for surgical systems - Google Patents

Abstract

An augmented reality display system for use during a surgical procedure is disclosed. The system includes an imaging device for capturing real images of a surgical field during a surgical procedure. The augmented reality display presents functional overlay data associated with critical motions of an actively visualized surgical instrument and the surgical instrument's interaction with tissue in the surgical field. The functional data is overlaid on the real image of the surgical field. The functional data overlay is a combination of critical motions of the surgical instrument and aspects of the surgical instrument's interaction with tissue in the surgical field. A processor receives functional data from the surgical instrument, determines overlay data related to the functional aspects of the surgical instrument, and combines aspects of the tissue in the surgical field with the functional data received from the surgical instrument.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS
This application claims the benefit under 35 U.S.C. §119(e) to U.S. Provisional Patent Application No. 63/174,674, entitled "HEADS UP DISPLAY," filed on April 14, 2021, and U.S. Provisional Patent Application No. 63/284,326, entitled "INTRAOPERAIVE DISPLAY FOR SURGICAL SYSTEMS," filed on November 30, 2021, the disclosures of each of which are incorporated by reference in their entireties herein.

The present disclosure relates to devices, systems, and methods for providing augmented reality interactive experiences during surgical procedures. During surgical procedures, objects present in the real world are sometimes enhanced by overlaying computer-generated sensory information across multiple sensory modalities, including visual, auditory, tactile, somatosensory, and olfactory senses. It would be desirable to provide an augmented reality interactive experience of a real world environment. In the context of this disclosure, images of the surgical field and surgical instruments and other objects appearing in the surgical field may include computer-generated visual, auditory, tactile, somatosensory, olfactory, or other sensory information about the surgical field and Enhanced by overlaying onto real-world images of instruments or other objects appearing in the surgical field. Images may be streamed in real time or may be static images.

Real-world surgical instruments include a variety of surgical devices. Energy-based surgical devices include, among others, but not limited to, radio frequency (RF)-based monopolar and bipolar electrosurgical instruments, ultrasound surgical instruments, combinations of RF electrosurgical instruments and ultrasound instruments, Includes a combination of an RF electrosurgical stapler and a mechanical stapler. Surgical stapler devices are instruments used to cut and staple tissue in a variety of surgical procedures, including bariatric, thoracic, colorectal, obstetrics and gynecology, urology, and general surgery.

In various instances, the present disclosure provides an augmented reality display system for use during a surgical procedure. The augmented reality display system includes an imaging device for capturing a real image of a surgical field during a surgical procedure. The augmented reality display presents functional data overlays associated with actively visualized critical motions of a surgical instrument and interactions of the surgical instrument with tissue in the surgical field. The functional data is overlaid on the real image of the surgical field. The functional data overlay is a combination of critical motions of the surgical instrument and aspects of the interaction of the surgical instrument with tissue in the surgical field. A processor receives functional data from the surgical instrument, determines overlay data associated with functional aspects of the surgical instrument, and combines aspects of the tissue in the surgical field with the functional data received from the surgical instrument.

In various instances, the present disclosure provides an augmented reality display system for use during a surgical procedure. The augmented reality display system includes an imaging device for capturing real images of a surgical area during a surgical procedure. The augmented reality display presents functional data overlays associated with the parameters of the surgical instrument being actively visualized and the interaction of the surgical instrument with tissue within the surgical field. Functional data is overlaid on the actual image of the surgical field, and the functional data overlay is a combination of parameters of the surgical instrument and aspects of the surgical instrument's interaction with tissue within the surgical field. A processor receives functional data from the surgical instrument, determines overlay data related to functional aspects of the surgical instrument, and combines aspects of tissue within the surgical region with the functional data received from the surgical instrument.

In various instances, the present disclosure provides a system that includes an augmented reality display system for use during a surgical procedure. The augmented reality display system includes an imaging device for capturing real images of a surgical area during a surgical procedure. The augmented reality display presents functional data overlays associated with the surgical instrument, and the energy generator is coupled to the surgical instrument. Surgical instruments use radio frequency (RF) energy and ultrasound energy during surgical procedures. The surgical hub is coupled to an energy generator and an augmented reality display. The surgical hub provides live images of the surgical area to an augmented reality display for displaying live images of the surgical area. The augmented reality display provides a view of the surgical area, the surgical instrument, and panel overlays for displaying information specific to important operations or parameters of the surgical instrument and the interaction of the surgical instrument with tissue within the surgical area. Display.

The various aspects described herein, both as to organization and method of operation, together with further objects and advantages thereof, may best be understood by reference to the following description taken in conjunction with the accompanying drawings, in which:
FIG. 1 is a block diagram of a computer-implemented interactive surgical system according to one aspect of the present disclosure. FIG. 1 is a diagram of a surgical system used to perform a surgical procedure in an operating room, according to one aspect of the present disclosure. 1 is a surgical hub paired with a visualization system, a robotic system, and an intelligent instrument, according to one aspect of the present disclosure. FIG. 1 illustrates a surgical data network including a modular communications hub configured to connect modular devices located at one or more surgical sites in a medical facility or any room in a medical facility equipped with specialized equipment for surgical procedures to a cloud, according to one aspect of the disclosure. FIG. 1 illustrates a computer-implemented interactive surgical system according to one aspect of the present disclosure. FIG. 1 illustrates a surgical hub including multiple modules coupled to a modular control tower according to one aspect of the present disclosure. FIG. 13 illustrates an augmented reality (AR) system including an intermediate signal combiner disposed in a communication path between an imaging module and a surgical hub display according to one aspect of the present disclosure. FIG. 13 illustrates an augmented reality (AR) system including an intermediate signal combiner disposed in a communication path between an imaging module and a surgical hub display according to one aspect of the present disclosure. FIG. 13 illustrates an augmented reality (AR) device worn by a surgeon to communicate data to a surgical hub, according to one aspect of the present disclosure. FIG. 1 illustrates a system for augmenting surgical instrument information using an augmented reality display, according to one aspect of the present disclosure. 1 is an augmented image of a live feed of a surgical field visualized through a laparoscopic camera during a minimally invasive surgical procedure showing tissue in place captured between the jaws of a surgical instrument end effector as a tissue aspect, according to one aspect of the present disclosure. 1 is an augmented image of a live feed of a surgical field visualized through a laparoscopic camera during a minimally invasive surgical procedure showing a tissue location captured proximally between surgical instrument end effectors as a tissue aspect, according to one aspect of the present disclosure. 1 is an augmented image of a live feed of a surgical area visualized through a laparoscopic camera during a minimally invasive surgical procedure showing tissue poorly captured between the jaws of a surgical instrument end effector as a tissue aspect, according to one aspect of the present disclosure. 13 is another expanded image of a live feed of a surgical area visualized through a laparoscopic camera during a minimally invasive surgical procedure showing tissue poorly captured between the jaws of a surgical instrument end effector as a tissue aspect, according to one aspect of the present disclosure. 13 is another expanded image of a live feed of a surgical area visualized through a laparoscopic camera during a minimally invasive surgical procedure showing tissue poorly captured between the jaws of a surgical instrument end effector as a tissue aspect, according to one aspect of the present disclosure. 1 is an augmented image of a live feed of a surgical field visualized through a laparoscopic camera during a minimally invasive surgical procedure showing tissue tension as a tissue behavior in accordance with at least one aspect of the present disclosure. 13 is another expanded image of a live feed of a surgical field visualized through a laparoscopic camera during a minimally invasive surgical procedure showing tissue tension as a tissue aspect according to one aspect of the present disclosure. 18A-18C are a number of graphic images illustrating the jaw closed position as a mode of operation of the surgical instrument shown in FIGS. 11-17 according to one aspect of the present disclosure. 1 is an augmented image of a live feed of a surgical field visualized through a laparoscopic camera during a minimally invasive surgical procedure showing a jaw closed position as an operational mode of a surgical instrument according to one aspect of the present disclosure. 20 is an expanded image of the live video of the surgical field shown in FIG. 19 showing a fully closed surgical instrument end effector and a graphical alert overlay indicating the jaw closed position superimposed on the end effector, according to one aspect of the present disclosure. 1 is an augmented image of a live feed of a surgical field visualized through a laparoscopic camera during a minimally invasive surgical procedure showing clamping on a metal or foreign body as a mode of operation of a surgical instrument according to one aspect of the present disclosure. 1 is an augmented image of a live feed of a surgical field visualized through a laparoscopic camera during a minimally invasive surgical procedure showing a device preheat warning where overheating is an operational aspect of a surgical instrument, according to one aspect of the present disclosure. 1 is an augmented image of a live feed of a surgical field visualized through a laparoscopic camera during a minimally invasive surgical procedure showing tissue movement and flow according to one aspect of the present disclosure. 1 is an augmented image of a live feed of the surgical field visualized through a laparoscopic camera during a minimally invasive surgical procedure showing the geometric relationship of the transection to the tissue and to other launches, according to one aspect of the present disclosure; 13 is an image showing anvil orientation communication as an operational mode of a surgical instrument according to one aspect of the present disclosure. 13 is an image showing tissue thickness detected at a jaw of a surgical instrument end effector, where the tissue thickness is a tissue aspect, according to one aspect of the present disclosure. 1 is an image of a temperature gradient display graphic of an ultrasonic instrument according to one aspect of the present disclosure. 1 is an image of a temperature icon display graphic for an ultrasonic instrument according to one aspect of the present disclosure. 1 is an image of an ultrasonic blade temperature graphic element mapped to an end effector jaw position according to one aspect of the present disclosure. 1 is an image of an ultrasonic generator power level display graphic according to one aspect of the present disclosure. 13 is an image of an ultrasonic generator power level display graphic with a pop-up warning graphic indicating that the ultrasonic end effector jaws are too clogged, according to one aspect of the present disclosure. 13 is an image of an ultrasonic generator power level display graphic with a pop-up warning graphic indicating ultrasonic end effector jaw heat, according to one aspect of the present disclosure. 13 is an image of an electrosurgical generator display graphic having a pop-up warning graphic indicating an electrosurgical seal quality prediction, according to one aspect of the present disclosure. 13 is an image of a surgical stapler reload feedback according to one aspect of the present disclosure. 1 is an image of a surgical stapler preload countdown according to one aspect of the present disclosure. FIG. 1 is a system diagram of a surgical room including a surgical monitor having intraoperative data display of a surgical area, according to one aspect of the present disclosure. 1 is an augmented image of a live feed of a surgical field visualized through a laparoscopic camera during a minimally invasive surgical procedure displayed on an intraoperative data display according to one aspect of the present disclosure. FIG. 38 is a detailed view of the case information panel overlay shown in FIG. 37 according to one aspect of the present disclosure. FIG. 38 is a detailed view of the system notification panel overlay shown in FIG. 37 according to one aspect of the present disclosure. 11A-11C are images of several examples of system notification panel overlays, according to one aspect of the present disclosure. FIG. 38 is a detailed view of the device panel overlay shown in FIG. 37 according to one aspect of the present disclosure. 1 is an augmented image of a live feed of a surgical field visualized through a laparoscopic camera during a minimally invasive surgical procedure displayed on an intraoperative data display according to one aspect of the present disclosure. FIG. 1 is a schematic diagram of an energy device image panel architecture according to one aspect of the present disclosure. 13 is an image of a stacked configuration supplemental device alert/warning/information according to one aspect of the present disclosure. 13 is an image of a supplemental device alert/warning/information in an expanded configuration according to one aspect of the present disclosure. 13 is an instrument state image panel illustrating how the instrument panel state dynamically changes to indicate a state change such as device activation or power level adjustment, according to one aspect of the disclosure. FIG. 13 is a system diagram for translating generator alerts and warnings into a laparoscopic monitor and displaying them on a local interface according to one aspect of the present disclosure. FIG. 11 is a diagram of a series of screens of existing alerts shown on a current generator that are transmitted to a surgical hub, which then displays these as a series of screens on a local interface, according to one aspect of the present disclosure. FIG. 1 is a schematic diagram of a system including a generator in communication with a digital hub that then displays screen and alert data on a local interface, such as a laparoscope screen, according to one aspect of the disclosure. 1 is an augmented image of a live feed of a surgical field visualized through a laparoscopic camera during a minimally invasive surgical procedure displayed on an intraoperative data display according to one aspect of the present disclosure. 13 is a display screen illustrating an intraoperative data display with a secondary bottom edge configurable panel according to one aspect of the present disclosure. 13 is an alternative bottom edge configurable panel according to one aspect of the present disclosure. 1 is a display screen illustrating an intraoperative data display with a secondary top left corner configurable panel, according to one aspect of the present disclosure. 1 is a display screen illustrating an intraoperative data display with a secondary top-center configurable panel, according to one aspect of the present disclosure. 1 is a display screen illustrating an intraoperative data display with secondary side edge configurable panels according to one aspect of the present disclosure. 1 is a series of image panels displaying device troubleshooting information according to one aspect of the present disclosure. 1 is a series of image panels displaying articulating surgical stapler features according to one aspect of the present disclosure. 13 is an alert/warning/information image panel indicating that a joint limit has been reached, according to one aspect of the present disclosure. 13 is an alert/warning/information image panel that indicates the device is in lockout mode, according to one aspect of the present disclosure. 13 is an alert/warning/information image panel that indicates the device cannot articulate when the jaws are closed, according to one aspect of the present disclosure. 1 is a device image panel illustrating articulating surgical stapler features according to one aspect of the present disclosure. 13 is a stacked alert/warning/information image panel displayed in a stacked configuration with a device alert indicating a joint limit has been reached, according to one aspect of the present disclosure. FIG. 62 is a schematic diagram of a system including a surgical stapler communicating with a digital hub via Bluetooth to execute an algorithm for the countdown timer image panel shown in FIGS. 57 and 61 according to one aspect of the present disclosure. 1 is a series of device image panels/alerts displaying ultrasound instrument features according to one aspect of the present disclosure. 1 is a chart illustrating pairing of a surgical stapler instrument according to one aspect of the present disclosure. 13 is an image of a screen displaying paired device information according to one aspect of the present disclosure. 1 is an image of a wireless surgical device including a unique identifier for pairing wireless devices according to one aspect of the present disclosure. 1 is an image of a screen displaying a link to an Optimal Device Performance (ODP) guide image or other electronic Instructions for Use (e-IFU), according to one aspect of the present disclosure. 1 is a diagram of an augmented reality method using a surgical instrument and an augmented reality display for use during a surgical procedure, according to one aspect of the present disclosure. 1 is a diagram of an augmented reality method using a surgical instrument and an augmented reality display for use during a surgical procedure, according to one aspect of the present disclosure. 1 is a diagram of an augmented reality method using a surgical instrument and an augmented reality display for use during a surgical procedure, according to one aspect of the present disclosure. 13 is an image of a staff view screen displaying customized overlay information, according to one aspect of the present disclosure. 13 is an image of a staff view screen displaying detailed customization pop-up information, according to one aspect of the present disclosure. 13 is an image of a staff view screen displaying staff view troubleshooting pop-up information, according to one aspect of the present disclosure. 13 is an image of a primary surgical display interaction screen displaying primary surgical display interactions according to one aspect of the present disclosure. FIG. 1 illustrates a timeline of a situation-aware surgical procedure, according to one aspect of the present disclosure.

Corresponding reference characters indicate corresponding parts throughout the several views. The illustrations set forth herein illustrate various disclosed embodiments in one form and such illustrations should not be construed as limiting the scope thereof in any way.

The applicant of this application owns the following concurrently filed US patent applications, each of which is incorporated herein by reference in its entirety.
- United States patent application entitled "METHOD FOR INTRAOPERATIVE DISPLAY FOR SURGICAL SYSTEMS"; Attorney docket number END9352USNP1/210120-1M;
・U.S. patent application entitled “UTILIZATION OF SURGICAL DATA VALUES AND SITUATIONAL AWARENESS TO CONTROL THE OVERLAY IN SURGICAL FIELD VIEW”; Attorney docket number E ND9352USNP2/210120-2,
- U.S. patent application entitled "SELECTIVE AND ADJUSTABLE MIXED REALITY OVERLAY IN SURGICAL FIELD VIEW"; Attorney docket number END9352USNP3/210120-3;
- U.S. patent application entitled "RISK BASED PRIORITIZATION OF DISPLAY ASPECTS IN SURGICAL FIELD VIEW"; Attorney docket number END9352USNP4/210120-4;
- United States patent application entitled "SYSTEMS AND METHODS FOR CONTROLLING SURGICAL DATA OVERLAY"; Attorney docket number END9352USNP5/210120-5;
・United States patent application entitled "SYSTEMS AND METHODS FOR CHANGING DISPLAY OVERLAY OF SURGICAL FIELD VIEW BASED ON TRIGGERING EVENTS"; Attorney docket number END9352USNP6 /210120-6,
- United States patent application entitled "CUSTOMIZATION OF OVERLAID DATA AND CONFIGURATION"; Attorney docket number END9352USNP7/210120-7;
- U.S. patent application entitled "INDICATION OF THE COUPLE PAIR OF REMOTE CONTROLS WITH REMOTE DEVICES FUNCTIONS"; attorney docket number END9352USNP8/210120-8;
・U.S. patent application entitled "COOPERATIVE OVERLAYS OF INTERACTING INSTRUMENTS WHICH RESULT IN BOTH OVERLAYS BEING EFFECTED"; Attorney docket number END9352USNP9/21012 0-9,
- United States patent application entitled "ANTICIPATION OF INTERACTIVE UTILIZATION OF COMMON DATA OVERLAYS BY DIFFERENT USERS"; Attorney docket number END9352USNP10/210120-10;
・“MIXING DIRECTLY VISUALIZED WITH RENDERED ELEMENTS TO DISPLAY BLENDED ELEMENTS AND ACTIONS HAPPENING ON-SCREEN AND OFF-S US Patent Application entitled ``CREEN''; Attorney Docket No. END9352USNP11/210120-11;
・United States patent application entitled "SYSTEM AND METHOD FOR TRACKING A PORTION OF THE USER AS A PROXY FOR NON-MONITORED INSTRUMENT"; Attorney docket number END9352USNP12/2101 20-12,
・“UTILIZING CONTEXTUAL PARAMETERS OF ONE OR MORE SURGICAL DEVICES TO PREDICT A FREQUENCY INTERVAL FOR DISPLAYING SURGICAL US Patent Application entitled ``INFORMATION''; Attorney Docket No. END9352USNP13/210120-13;
・U.S. patent application entitled “COOPERATION AMONG MULTIPLE DISPLAY SYSTEMS TO PROVIDE A HEALTHCARE USER CUSTOMIZED INFORMATION”; Attorney docket number END9352USNP14/ 210120-14,
・United States patent application entitled “INTRAOPERATIVE DISPLAY FOR SURGICAL SYSTEMS”; Attorney docket number END9352USNP15/210120-15, and ・“MIXED REALITY FEEDBACK SYSTEMS THAT COO U.S. Patent Application entitled ``PERATE TO INCREASE EFFICIENT PERCEPTION OF COMPLEX DATA FEEDS''; Agent Reference number END9352USNP17/210120-17.

The applicant of this application owns the following US patent applications, each of which is incorporated by reference in its entirety.
・“METHOD OF COMPRESSING TISSUE WITHIN A STAPLING DEVICE AND SIMULTANEOUSLY DISPLAYING THE LOCATION OF THE TISSUE WITHIN THE No. 16/209,423 entitled “JAWS” (now U.S. Patent Application Publication No. 2019/0200981 (A1) ),
- U.S. Patent Application No. 16/209,453 entitled "METHOD FOR CONTROLLING SMART ENERGY DEVICES" (now U.S. Patent Application Publication No. 2019/0201046 (A1)).

Before describing in detail various aspects of surgical devices and generators, the illustrative examples are limited to the details of construction and arrangement of parts in the accompanying drawings and description. Please note that this is not the case. The example embodiments may be implemented in or incorporated into other aspects, variations, and modifications and may be practiced or carried out in various ways. Furthermore, unless otherwise stated, the terms and expressions used herein have been selected for the convenience of the reader to describe the illustrative embodiments and are not intended to be limiting. Additionally, one or more of the aspects, expressions of aspects, and/or examples described below may be combined with any one or more of the other aspects, expressions of aspects, and/or examples described below. It should be understood that it can be combined with

Various aspects are directed to on-screen displays for surgical systems for various energy and surgical stapler based medical devices. Energy based medical devices include, but are not limited to, radio frequency (RF) based monopolar and bipolar electrosurgical instruments, ultrasonic surgical instruments, combinations of RF electrosurgical instruments and ultrasonic instruments, combinations of RF electrosurgical staplers and mechanical staplers, among others. Surgical stapler devices include surgical staplers in combination with electrosurgical devices and/or ultrasonic devices. Aspects of ultrasonic surgical devices can be configured, for example, to transect and/or coagulate tissue during a surgical procedure. Aspects of electrosurgical devices can be configured, for example, to transect, coagulate, seal, weld and/or desiccate tissue during a surgical procedure. Aspects of surgical stapler devices can be configured to transect and staple tissue during a surgical procedure, and in some aspects, surgical stapler devices can be configured to deliver RF energy to tissue during a surgical procedure. Electrosurgical devices are configured to deliver therapeutic and/or non-therapeutic RF energy to tissue. The elements of the surgical stapler, electrosurgical device, and ultrasonic device may be used in combination in a single surgical instrument.

In various aspects, the present disclosure provides an on-screen display of real-time information to the OR team during a surgical procedure. In accordance with various aspects of the present disclosure, many new and unique on-screen displays are provided to display various visual information feedback on-screen to the OR team. In accordance with the present disclosure, the visual information can include one or more of a variety of visual media, with or without sound. In general, the visual information includes still photographs, motion photographs, video or audio recordings, graphic art, visual aids, models, displays, visual representation services, and support processes. The visual information can be communicated on any number of display options, such as, for example, the primary OR screen, the energy or surgical stapler device itself, tablets, augmented reality glasses, among others.

In various aspects, the present disclosure provides a long list of potential options for communicating visual information to the OR team in real time without overwhelming the OR team with too much visual information. For example, in various aspects, the present disclosure provides an on-screen display of visual information that allows the surgeon, or other members of the OR team, to selectively activate on-screen displays, such as icons surrounding the screen options, to manage the wealth of visual information. One or a combination of factors can be used to determine the active display, which may include, among others, the energy-based (e.g., electrosurgery, ultrasound) or mechanical-based (e.g., stapler) surgical device being used, the estimated risk associated with a given display, the surgeon's experience level, and the surgeon's choice. In other aspects, the visual information may include a wealth of data overlaid or superimposed on the surgical field to manage the visual information. In various aspects described below, this includes superimposed images that require video analysis and tracking to properly overlay the data. The visual information data communicated in this manner can provide additional useful visual information to the OR team in a more concise and understandable manner, as opposed to static icons.

In various aspects, the present disclosure provides techniques for selectively activating on-screen displays, such as icons that surround the screen, to manage visual information during a surgical procedure. In other aspects, the present disclosure provides techniques for determining an active display using one or a combination of factors. In various aspects, techniques according to the present disclosure may include, among other things, selecting an energy-based or mechanical-based surgical device in use as the active display, estimating the risk associated with a given display, and utilizing the experience level of the surgeon or OR team to make the selection.

In other aspects, techniques according to the present disclosure may include overlaying or superimposing rich data onto the surgical field to manage visual information. Some display arrangements described by this disclosure involve overlaying various visual representations of surgical data onto a live stream of the surgical field. As used herein, the term overlay includes translucent overlays, partial overlays, and/or moving overlays. Graphical overlays may be in the form of transparent, translucent, or opaque graphics, or a combination of transparent, translucent, and opaque elements or effects. Additionally, the overlay may be placed on or at least partially on or near objects in the surgical field, such as, for example, end effectors and/or critical surgical structures. The particular display arrangement may include changes in color, size, shape, display time, display position, display frequency, highlighting, or combinations thereof of one or more displays of the overlay based on the change in the display priority value. May include changes in elements. A graphical overlay is rendered on top of the active display monitor and conveys critical information to the OR team quickly and efficiently.

In other aspects, techniques according to the present disclosure may include overlaying images that require video analysis and tracking to properly overlay visual information data. In other aspects, techniques according to the present disclosure may include communicating rich visual information, as opposed to simple static icons, to provide additional visual information to the OR team in a more concise and easy to understand manner. In other aspects, visual overlays may be used in combination with auditory and/or somatosensory overlays, e.g., thermal, chemical, and mechanical devices, and combinations thereof.

The following description generally relates to devices, systems, and methods for providing augmented reality (AR) interactive experiences during surgical procedures. In this context, images of the surgical field and surgical instruments and other objects that appear in the surgical field may be used to convey computer-generated visual, auditory, tactile, somatosensory, olfactory, or other sensory information to the surgical field. by overlaying onto real-world images of instruments and/or other objects that appear in the image. Images may be streamed in real time or may be static images. Augmented reality is a technology for rendering and displaying virtual or "augmented" virtual objects, data, or visual effects that are overlaid onto a real environment. The real environment may include a surgical field. Virtual objects overlaid on the real environment may be represented in a fixed or set position relative to one or more aspects of the real environment. In a non-limiting example, when a real world object exits the field of view of the real environment, a virtual object fixed to the real world object also exits the field of view of the augmented reality.

Some display arrangements described by this disclosure involve overlaying various visual representations of surgical data onto a live stream of the surgical field. As used herein, the term overlay includes translucent overlays, partial overlays, and/or moving overlays. Additionally, the overlay may be placed on or at least partially on or near objects in the surgical field, such as, for example, end effectors and/or critical surgical structures. The particular display arrangement may include changes in color, size, shape, display time, display position, display frequency, highlighting, or combinations thereof of one or more displays of the overlay based on the change in the display priority value. May include changes in elements.

As described herein, AR is an augmented version of the real physical world, achieved through the use of digital visual elements, sounds, or other sensory stimuli delivered via technology. Virtual reality (VR) is a computer-generated environment with scenes and objects that appear to be real, making the user feel immersed in their surroundings. This environment is perceived through a device known as a virtual reality headset or helmet. Although mixed reality (MR) and AR are both considered immersive technologies, they are not the same. MR is an extension of mixed reality that allows real and virtual elements to interact in the environment. While AR often adds digital elements to a live view by using a camera, an MR experience combines elements of both AR and VR, where real-world and digital objects interact.

In an AR environment, one or more computer-generated virtual objects may be displayed along with one or more real (i.e., so-called "real-world") elements. For example, a real-time image or video of the surrounding environment may be shown on a computer screen display along with one or more overlaid virtual objects. Such virtual objects may provide supplemental information about the environment, or may generally enhance the user's perception of and engagement with the environment. Conversely, a real-time image or video of the surrounding environment may additionally or alternatively enhance the user's engagement with the virtual objects shown on the display.

The devices, systems, and methods in the context of the present disclosure enhance images received from one or more imaging devices during a surgical procedure. The imaging devices may include various scopes used during non-invasive and minimally invasive surgical procedures, AR devices, and/or cameras that provide images during open surgical procedures. The images may be streamed in real time or may be still images. The devices, systems, and methods provide an augmented reality interactive experience by enhancing images of a real-world surgical environment by overlaying virtual objects or representations of data and/or real objects onto the real surgical environment. The augmented reality experience may be viewed on a display and/or AR device that allows a user to view virtual objects overlaid on the real-world surgical environment. The display may be located in the operating room or may be separate from the operating room. The AR device is worn on the head of the surgeon or other operating room personnel and typically includes two stereoscopic display lenses or screens, one for each eye of the user. Natural light can pass through two transparent or translucent display lenses so that aspects of the real environment are visible, while light is projected to make virtual objects visible to the user of the AR device.

Two or more displays and AR devices may be used in conjunction with, for example, a first display or AR device controlling one or more additional displays or AR devices in the system with a defined role. . For example, when activating a display or AR device, a user may select a role (e.g., surgeon during a surgical procedure, surgical assistant, nurse, etc.) and the display or AR device may select a role associated with that role. Information may be displayed. For example, a surgical assistant may display a virtual representation of instruments that the surgeon needs to perform for the next step in a surgical procedure. The surgeon's focus on the current process may see displayed information differently than the surgical assistant.

While there are many known on-screen displays and alerts, the present disclosure provides many new and unique augmented reality interactive experiences during surgical procedures. Such augmented reality interactive experiences include visual, auditory, tactile, somatosensory, olfactory, or other sensory feedback information to the surgical team inside or outside the operating room. Virtual feedback information overlaid on the real-world surgical environment can, for example, improve personnel within the OR, including, but not limited to, operating surgeons, surgeon's assistants, scrub wearers, anesthesiologists, and outgoing nurses, among others. may be provided to the operating room (OR) team, including: Virtual feedback information can be communicated on any number of display options, such as primary OR screen displays, AR devices, energy or surgical stapler instruments, tablets, augmented reality glasses, devices, etc.

FIG. 1 shows a computer-implemented interactive surgical system 1 that includes one or more surgical systems 2 and a cloud-based system 4 . Cloud-based system 4 may include a remote server 13 coupled to remote storage 5 . Each surgical system 2 comprises at least one surgical hub 6 in communication with the cloud 4. For example, surgical system 2 may include visualization system 8 , robotic system 10 , and hand-held intelligent surgical instrument 12 , each configured to communicate with each other and/or with hub 6 . There is. In some aspects, surgical system 2 may include M hubs 6, N visualization systems 8, O robotic systems 10, and P handheld intelligent surgical instruments 12. Often M, N, O and P are integers greater than or equal to 1. Computer-implemented interactive surgical system 1 may be configured to provide an augmented reality interactive experience during a surgical procedure, as described herein.

FIG. 2 shows an example of a surgical system 2 for performing a surgical procedure on a patient lying on an operating table 14 in a surgical operating room 16. Robotic system 10 is used as part of surgical system 2 in a surgical procedure. Robotic system 10 includes a surgeon's console 18 , a patient cart 20 (surgical robot), and a surgical robot hub 22 . The patient-side cart 20 is removably coupled to at least one patient-side cart 20 through a minimally invasive incision in the patient's body while the surgeon views the surgical site through the surgeon's console 18 or an augmented reality (AR) device 66 worn by the surgeon. surgical tool 17 can be operated. Images of the surgical site during a minimally invasive procedure (eg, still images or live images streamed in real time) may be acquired by medical imaging device 24. Patient-side cart 20 can operate imaging device 24 to orient imaging device 24 . Images of the open surgical procedure may be acquired by medical imaging device 96. The robotic hub 22 processes images of the surgical site for subsequent display on the surgeon's console 18 or an AR device 66 worn by the surgeon or other person within the surgical operating room 16 .

Optical components of imaging device 24, 96 or AR device 66 may include one or more illumination sources and/or one or more lenses. One or more illumination sources may be directed to illuminate a portion of the surgical field. One or more image sensors may receive light reflected or refracted from tissue and instruments within the surgical field.

In various aspects, imaging device 24 is configured for use in minimally invasive surgical procedures. Examples of imaging devices suitable for use with the present disclosure include arthroscopes, angioscopes, bronchoscopes, cholangioscopes, colonoscopes, cystoscopes, duodenoscopes, enteroscopes, esophagogastroduodenoscopes (gastroscopes), endoscopic These include, but are not limited to, endoscopes, laryngoscopes, nasopharyngeal-pyeloscopes, sigmoidoscopes, thoracoscopes, and ureteroscopes. In various aspects, imaging device 96 is configured for use in open (invasive) surgical procedures.

In various aspects, visualization system 8 includes one or more imaging sensors strategically positioned relative to the sterile field, one or more image processing devices, one or more storage arrays, and one or more storage arrays. Equipped with a display. In one aspect, visualization system 8 includes interfaces for HL7, PACS, and EMR. In one aspect, imaging device 24 may employ multispectral monitoring to distinguish between topography and underlying structure. Multispectral images capture image data within specific wavelength ranges in the electromagnetic spectrum. The wavelengths are separated by filters or by instruments with sensitivity to specific wavelengths, including light from frequencies beyond the visible light range, such as the IR and ultraviolet. Spectral imaging can extract information invisible to the human eye. Multispectral monitoring can reposition the surgical field after the surgical task is completed to perform tests on the treated tissue.

2 shows a primary display 19 positioned in the sterile field for viewing by an operator at the operating table 14. The visualization tower 11 includes a first non-sterile display 7 and a second non-sterile display 9 positioned outside the sterile field and facing away from each other. A visualization system 8 guided by the hub 6 is configured to utilize the displays 7, 9, 19 to coordinate information flow to operators inside and outside the sterile field. For example, the hub 6 can cause the visualization system 8 to display AR images of the surgical site recorded by the imaging devices 24, 96 through the non-sterile displays 7, 9 or the AR device 66 while maintaining a live video of the surgical site on the primary display 19 or the AR device 66. The non-sterile displays 7, 9 can, for example, enable a non-sterile operator to perform diagnostic steps related to a surgical procedure.

3 shows a hub 6 in communication with a visualization system 8, a robotic system 10, and a handheld intelligent surgical instrument 12. The hub 6 includes a hub display 35, an imaging module 38, a generator module 40, a communication module 30, a processor module 32, a storage array 34, and an operating room mapping module 33. The hub 6 further includes a smoke evacuation module 26 and/or an aspiration/irrigation module 28. In various aspects, the imaging module 38 includes an AR device 66, and the processor module 32 includes an integrated video processor and an augmented reality modeler (e.g., as shown in FIG. 10). The modular light source can be adapted for use with a variety of imaging devices. In various examples, multiple imaging devices can be positioned at different locations of the surgical field to provide multiple views (e.g., non-invasive, minimally invasive, invasive, or open surgical procedures). The imaging module 38 can be configured to switch between imaging devices to provide an optimal view. In various aspects, the imaging module 38 can be configured to integrate images from different imaging devices to provide an augmented reality interactive experience during a surgical procedure as described herein.

4 shows a surgical data network 51 including a modular communication hub 53 configured to connect modular devices located at one or more surgical sites/rooms of a medical facility to a cloud-based system. The cloud 54 can include a remote server 63 (FIG. 5) coupled to a storage device 55. The modular communication hub 53 includes a network hub 57 and/or a network switch 59 in communication with a network router 61. The modular communication hub 53 is coupled to a local computer system 60 for processing data. Modular devices 1a-1n at the surgical site can be coupled to the modular communication hub 53. The network hub 57 and/or the network switch 59 can be coupled to the network router 61 to connect the devices 1a-1n to the cloud 54 or the local computer system 60. Data associated with the devices 1a-1n can be transferred through the router to a cloud-based computer for remote data processing and manipulation. The surgical site devices 1a-1n can be connected to the modular communication hub 53 via wired or wireless channels. The surgical data network 51 environment may be employed to provide an augmented reality interactive experience during a surgical procedure, as described herein, and in particular to provide an augmented image of the surgical field to one or more remote displays 58.

FIG. 5 shows a computer-implemented interactive surgical system 50. Computer-implemented interactive surgical system 50 is similar to computer-implemented interactive surgical system 1 in many respects. Computer-implemented interactive surgical system 50 includes one or more surgical systems 52 that are similar to surgical system 2 in many respects. Each surgical system 52 includes at least one surgical hub 56 that communicates with a cloud 54 that may include a remote server 63. In one aspect, computer-implemented interactive surgical system 50 includes a modular control tower 23 connected to multiple surgical site devices, such as, for example, intelligent surgical instruments, robots, and other computerized devices located within the surgical site. Be prepared. As shown in FIG. 6, modular control tower 23 includes a modular communication hub 53 coupled to computer system 60. As shown in FIG.

Returning to FIG. 5, the modular control tower 23 includes an imaging module 38 coupled to an endoscope 98, a generator module 27 coupled to an energy device 99, a smoke evacuator module 76, an aspiration/irrigation module 78, and a communication The module 13 is coupled to a processor module 15, a storage array 16, a smart device/appliance 21 optionally coupled to a display 39, and a sensor module 29. The surgical site devices are coupled to cloud computing resources such as servers 63, data storage 55, and displays 58 via modular control tower 23. Robotic hub 72 may also be connected to modular control tower 23 and servers 63, data storage 55, and displays 58. Among other things, devices/appliances 21, visualization systems 58 may be coupled to modular control tower 23 via wired or wireless communication standards or protocols, as described herein. Modular control tower 23 is configured to display the received augmented images, including overlaid virtual objects on the real surgical world, received from imaging module 38, device/instrument display 39, and/or other visualization system 58. The hub display 65 (eg, monitor, screen) may also be coupled to the hub display 65 (eg, monitor, screen). Hub display 65 may also display data received from devices connected to modular control tower 23 along with images and overlay images.

6 shows a surgical hub 56 including multiple modules coupled to a modular control tower 23. The modular control tower 23 includes a modular communication hub 53, e.g., a network-connected device, and a computer system 60, e.g., for local processing, visualization, and imaging of extended surgical information. The modular communication hub 53 may be connected in a hierarchical configuration to expand the number of modules (e.g., devices) that may be connected to the modular communication hub 53 and transfer data associated with the modules to the computer system 60, cloud computing resources, or both. Each of the network hubs/switches 57/59 in the modular communication hub 53 may include three downstream ports and one upstream port. The upstream network hubs/switches 57, 59 are connected to a processor 31 to provide communication connections to cloud computing resources and a local display 67. Communication to the cloud 54 may be via either wired or wireless communication channels.

Computer system 60 includes a processor 31 and a network interface 37. Processor 31 is coupled to communication module 41, storage 45, memory 46, non-volatile memory 47, and input/output interface 48 via a system bus. The system bus may be any of several types of bus structures, including a memory bus or memory controller, a peripheral or external bus, and/or a local bus using a variety of available bus architectures. .

The processor 31 includes an augmented reality modeler (e.g., as shown in FIG. 10) and may be implemented as a single-core or multi-core processor such as those known under the trade name ARM Cortex by Texas Instruments. In one aspect, the processor may be, for example, an LM4F230H5QR ARM Cortex-M4F processor core available from Texas Instruments. The processor core includes on-chip memory of 256 KB of single-cycle flash memory or other non-volatile memory up to 40 MHz, a pre-fetch buffer to improve performance beyond 40 MHz, 32 KB of single-cycle serial random access memory (SRAM), internal read-only memory (ROM) with StellarisWare® software, 2 KB of electrically erasable programmable read-only memory (EEPROM), and/or one or more pulse width modulation (PWM) modules, one or more quadrature encoder input (QEI) analogs, one or more 12-bit analog-to-digital converters (ADCs) with 12 analog input channels, details of which are available in the product data sheet.

System memory includes volatile and nonvolatile memory. The basic input/output system (BIOS), containing the basic routines for transferring information between elements within a computer system, such as during start-up, is stored in nonvolatile memory. For example, nonvolatile memory may include ROM, programmable ROM (PROM), electrically programmable ROM (EPROM), EEPROM or flash memory. Volatile memory includes random access memory (RAM), which acts as external cache memory. In addition, RAM is available in many forms, such as SRAM, dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), SyncLink DRAM (SLDRAM), and direct RAMbus RAM (DRRAM).

The computer system 60 also includes removable/non-removable, volatile/non-volatile computer storage media, such as disk storage. Disk storage devices include, but are not limited to, devices such as magnetic disk drives, floppy disk drives, tape drives, Jaz drives, Zip drives, LS-60 drives, flash memory cards, or memory sticks. In addition, the disk storage devices can include the above storage media, either independently or in combination with other storage media. Other storage media include, but are not limited to, optical disk drives, such as compact disk ROM drives (CD-ROM), compact disk recordable drives (CD-R drives), compact disk rewriteable drives (CD-RW drives), or digital versatile disk ROM drives (DVD-ROM). Removable or non-removable interfaces may be used to facilitate connection of the disk storage devices to the system bus.

In various aspects, the computer system 60 of FIG. 6, the imaging module 38 of FIGS. 4-6, and/or the visualization system 58, and/or the processor module 15 may include an image processor, an image processing engine, a graphics processing unit (GPU), a media processor, or any dedicated digital signal processor (DSP) used to process digital images. The image processor may use parallel computing using single instruction multiple data (SIMD) or multiple instruction multiple data (MIMD) techniques to increase speed and efficiency. The digital image processing engine may perform a variety of tasks. The image processor may be a system on a chip with a multi-core processor architecture.

7 shows an augmented reality system 263 including an intermediate signal combiner 64 disposed in the communication path between the imaging module 38 and the surgical hub display 67. The signal combiner 64 combines audio and/or image data received from the imaging module 38 and/or the AR device 66. The surgical hub 56 receives the combined data from the combiner 64 and overlays the provided data on the display 67, where the overlay data is displayed. The imaging device 68 may be a digital video camera and the audio device 69 may be a microphone. The signal combiner 64 may include a wireless heads-up display adapter for coupling to the AR device 66 disposed in the communication path of the display 67 to the console that allows the surgical hub 56 to overlay the data on the display 67.

FIG. 8 shows an augmented reality (AR) system that includes an intermediate signal combiner positioned in a communication path between an imaging module and a surgical hub display. FIG. 8 shows an AR device 66 worn by a surgeon 73 to communicate data to the surgical hub 56. The peripheral information of the AR device 66 does not include active video. Rather, the peripheral information includes only signals that do not have the same requirements for device settings or refresh rates. The interaction may enhance the surgeon's 73 information based on linking with preoperative computed tomography (CT) or other data linked within the surgical hub 56. The AR device 66 can identify structures, for example, can ask if the instrument is touching a nerve, blood vessel, or adhesion. AR device 66 may include preoperative scan data, optical views, tissue investigation characteristics acquired throughout the procedure, and/or processing at surgical hub 56 that is used to provide answers. Surgeon 73 may write notes on AR device 66 to be saved with patient data in hub storage 45 for later use in reporting or follow-up.

An AR device 66 worn by the surgeon 73 links to the surgical hub 56 with audio and visual information to avoid the need for overlays and allows customization of information displayed around the periphery of the field of view. Make it. AR device 66 provides signals from devices (eg, fixtures) and answers queries regarding device settings or location information linked with video to identify quadrants or locations. AR device 66 has audio control and audio feedback from AR device 66 . The AR device 66 can interact with other systems within the surgical site and can have feedback and interaction available wherever the surgeon 73 looks. For example, AR device 66 may receive audio or gesture initiation commands and queries from a surgeon, and AR device 66 may provide feedback in the form of one or more modalities including audio, visual, or tactile touch. Good too.

FIG. 9 shows a surgeon 73 and a patient 74 wearing an AR device 66 and may include a camera 96 within the operating room 75. AR device 66 worn by surgeon 73 may be used to present virtual objects to surgeon 73 that are overlaid on real-time images of the surgical field, either through augmented reality display 89 or through hub-connected display 67. The real-time image may include a portion of the surgical instrument 77. The virtual object may not be visible to others in the operating room 75 (eg, surgical assistants or nurses), although they may also be wearing the AR device 66. Even if another person is viewing the operating room 75 using an AR device 66, that person may not be able to see the virtual objects or in the augmented reality shared with the surgeon 73. It may be possible to view the virtual object, or it may be possible to view a modified version of the virtual object (eg, according to customization specific to the surgeon 73), or it may be possible to view a different virtual object.

The virtual objects and/or data may be captured on a portion of the surgical instrument 77 or by the imaging module 38, the imaging device 68 during a minimally invasive surgical procedure, and/or the camera 96 during an open surgical procedure. It may be configured to appear within the field of view. In the illustrated example, imaging module 38 is a laparoscopic camera that provides live video of the surgical field during a minimally invasive surgical procedure. The AR system may present virtual objects that are fixed to real objects regardless of the perspective of one or more viewers of the AR system (eg, surgeon 73). For example, the virtual object may be visible to a viewer of the AR system within the operating room 75 and may not be visible to a viewer of the AR system outside the operating room 75. The virtual object may be displayed to a viewer outside the operating room 75 when the viewer enters the operating room 75. The augmented image may be displayed on surgical hub display 67 or augmented reality display 89.

The AR device 66 may include one or more screens or lenses, such as a single screen or two screens (e.g., one for each eye of the user). The screens may allow light to pass through them so that aspects of the real environment are visible while displaying the virtual objects. The virtual objects may be made visible to the surgeon 73 by projecting light onto them. The virtual objects may appear to have a degree of transparency or may be opaque (i.e., blocking aspects of the real environment).

The AR system may be visible to one or more viewers and may include differences between the views available to one or more viewers while retaining some aspects common between the views. For example, a heads-up display may change between two views, but virtual objects and/or data may be fixed to real objects or regions in both views. Aspects such as object color, lighting, or other changes may be made between views without changing the fixed position of at least one virtual object.

A user may view virtual objects and/or data presented in the AR system as opaque or as including a level of transparency. In one example, a user may interact with a virtual object, such as by moving the virtual object from a first position to a second position. For example, a user may move an object with his or her hand. This may be done virtually in the AR system by determining that the hand has been moved to a position coincident with or adjacent to the object (e.g., using one or more cameras, which may be mounted on the AR device 66, such as the AR device camera 79 or a separate 96, and may be static or controlled to move) and moving the object accordingly. The virtual aspects may include a virtual representation of a real-world object or may include visual effects, such as lighting effects. The AR system may include rules for governing the behavior of the virtual object, such as subjecting the virtual object to gravity or friction, or may include other predefined rules that negate real-world physical constraints (e.g., floating objects, perpetual motion, etc.). The AR device 66 may include a camera 79 on the AR device 66 (not to be confused with a camera 96, which is separate from the AR device 66). The AR device camera 79 or camera 96 may include an infrared camera, an infrared filter, a visible light filter, multiple cameras, a depth camera, etc. The AR device 66 may project virtual items onto a representation of the real environment that is visible to the user.

AR device 66 may be used, for example, within operating room 75 during a surgical procedure performed on patient 74 by surgeon 73. AR device 66 may project or display virtual objects, such as virtual objects during a surgical procedure, to augment the surgeon's vision. The surgeon 73 may view the virtual object using the AR device 66 , a remote controller for the AR device 66 , or “interact” with the virtual object or gesture recognized by the camera 79 of the AR device 66 , for example. The hands may be used to interact with virtual objects. The virtual object can extend a surgical tool, such as surgical instrument 77. For example, the virtual object may appear to be coupled to or remain a fixed distance from the surgical instrument 77 (to the surgeon 73 viewing the virtual object through the AR device 66). good. In another example, the virtual object may be used to guide surgical instrument 77 and may appear fixed to patient 74. In certain examples, virtual objects may react to movement of other virtual or real-world objects in the surgical field. For example, a virtual object may be modified when a surgeon is manipulating a surgical instrument in close proximity to the virtual object.

Augmented reality display system imaging device 38 captures real images of the surgical area during the surgical procedure. Augmented reality displays 89, 67 present an overlay of the operational aspects of surgical instrument 77 onto a real image of the surgical field. Surgical instrument 77 includes communication circuitry 231 for communicating operational aspects and functional data from surgical instrument 77 to AR device 66 via communication circuitry 233 on AR device 66 . Although surgical instrument 77 and AR device 66 are shown in RF wireless communication between circuits 231, 233, as indicated by arrows B, C, other communication technologies (e.g., wired, ultrasound, infrared etc.) may be adopted. The overlay relates to the operational aspects of the surgical instrument 77 that are being actively visualized. The overlay combines aspects of tissue interaction in the surgical field with functional data from the surgical instrument 77. The processor portion of the AR device 66 receives operational aspects and functional data from the surgical instrument 77, determines overlays related to the operation of the surgical instrument 77, and determines the overlays associated with the operation of the surgical instrument 77 to determine the aspects of the tissue within the surgical field from the surgical instrument 77. Configured to be combined with functional data. The enhanced image shows alerts regarding device performance considerations, non-conforming usage alerts, and incomplete acquisition alerts. Incompatible use includes out-of-range tissue conditions and tissue that is incorrectly balanced within the jaws of the end effector. Additional enhanced images provide an indication of concomitant events, including an indication of tissue tension and an indication of foreign body detection. Other enhanced images show device status overlays and instrument indications.

10 illustrates a system 83 for augmenting an image of a surgical field with information using an AR display 89, according to at least one aspect of the present disclosure. The system 83 may be used to perform the techniques described below, for example, by using a processor 85. The system 83 includes an aspect of an AR device 66 that can communicate with a database 93. The AR device 66 includes a processor 85, a memory 87, an AR display 89, and a camera 79. The AR device 66 may include a sensor 90, a speaker 91, and/or a haptic controller 92. The database 93 may include an image storage 94 or a preoperative planning storage 95.

Processor 85 of AR device 66 includes an augmented reality modeler 86 . Augmented reality modeler 86 may be used by processor 85 to create an augmented reality environment. For example, augmented reality modeler 86 may receive images of instruments within the surgical field, such as from camera 79 or sensor 90, and create an augmented reality environment to fit within the displayed image of the surgical field of view. In another example, physical objects and/or data may be overlaid on the surgical field of view and/or surgical instrument images, and augmented reality modeler 86 uses the physical objects and data to create virtual objects and /Or an augmented reality display of data may be presented within an augmented reality environment. For example, augmented reality modeler 86 uses or detects instruments at a patient's surgical site and displays virtual objects and/or data on the surgical instruments and/or images of the surgical site within the surgical field of view captured by camera 79. May be presented. The AR display 89 may display an AR environment overlaid on the real environment. Display 89 may show virtual objects and/or data using AR device 66, such as at a fixed location within the AR environment.

AR device 66 may include a sensor 90, such as an infrared sensor. Camera 79 or sensor 90 may be used to detect movements, such as gestures by a surgeon or other user, which are determined by processor 85 to indicate an attempted or intended interaction by the user with a virtual target. may be interpreted as Processor 85 may identify objects in the real environment, such as by processing information received using camera 79. In other aspects, sensors 90 may be tactile, audible, chemical, or thermal sensors to generate corresponding signals that may be combined with various data feeds to create an augmented environment. Sensors 90 may include binaural audio sensors (spatial sounds), inertial measurement (accelerometers, gyroscopes, magnetometers) sensors, environmental sensors, depth camera sensors, hand and eye tracking sensors, and voice command recognition capabilities.

The AR display 89 can display virtual images corresponding to physical features hidden by the patient's anatomy, such as within the surgical field, while allowing the surgical field to be viewed through the AR display 89 during a surgical procedure. Features may also be presented. The virtual feature may have a virtual location or orientation that corresponds to a first physical location or orientation of the physical feature. In one example, the virtual position or orientation of the virtual feature may include an offset from a first physical position or orientation of the physical feature. The offset may include a predetermined distance from the augmented reality display, a relative distance from the augmented reality display to the anatomical feature, and the like.

In one example, the AR device 66 can be an individual AR device. In one aspect, the AR device 66 can be a HoloLens 2 AR device manufactured by Microsoft of Redmond, Washington. This AR device 66 includes a visor with lenses and binaural audio features (spatial sound), inertial measurements (accelerometers, gyroscopes, magnetometers), environmental sensors, a depth camera, a video camera, hand and eye tracking, and voice command recognition. It provides an improved field of view with high resolution by using mirrors to direct the waveguide in front of the wearer's eyes. The image can be magnified by changing the angle of the mirror. It also provides eye tracking to recognize the user and adjust the lens width for the particular user.

In another example, AR device 66 may be a Snapchat Spectacles 3 AR device. This AR device provides the ability to capture paired images, recreate 3D depth mapping, add virtual effects, and play 3D videos. The AR device includes two HD cameras to capture 3D photos and videos at 60fps, while four built-in microphones record immersive high-fidelity audio. Images from both cameras are combined to build a geometric map of the real world around the user, providing a new sense of depth perception. Photos and videos can be wirelessly synced to external display devices.

In yet another example, the AR device 66 can be a Glass 2 AR device by Google, which provides inertial measurement (accelerometer, gyroscope, magnetometer) information overlaid on the lenses (outside the field of view) to supplement the information.

In another example, AR device 66 may be an Echo Frames AR device by Amazon. This AR device has no camera/display. The microphone and speaker are linked to Alexa. This AR device has fewer features than a head-up display.

In yet another example, AR device 66 may be a Focals AR device by North (Google). This AR device offers notification pusher/smartwatch analog, inertial measurements, screen overlays of information (weather, calendar, messages), and voice control (Alexa) integration. This AR device provides basic head-up display functionality.

In another example, AR device 66 may be a Nreal AR device. This AR device includes spatial sound, two environmental cameras, a photo camera, an IMU (accelerometer, gyroscope), an ambient light sensor, and a proximity sensor function. nebula projects application information onto the lens.

In various other examples, the AR device 66 is any one of the following commercially available AR devices: Magic Leap 1, Epson Moverio, Vuzix Blade AR, ZenFone AR, Microsoft AR Glasses Prototype, EyeTap. This creates light that is collinear with the light of the environment directly on the retina. A beam splitter, for example, makes the same visible light available to a computer for processing and overlaying information. AR visualization systems include HUDs, contact lenses, glasses, virtual reality (VR) headsets, virtual retinal displays, operating room displays, and/or smart contact lenses (bionic lenses).

The multi-user interface for the AR device 66 includes virtual retinal displays such as raster displays that are drawn directly on the retina rather than on a screen in front of the eyes, spatial displays such as smart TVs, smartphones, and/or Sony Spatial Display Systems. including.

Other AR technologies may include, for example, AR capture devices and software applications, AR creation devices and software applications, and AR cloud devices and software applications. AR capture devices and software applications include, for example, Apple Polycam app, Ubiquity 6 (Mirrorworld using Display.land app), which allows users to scan and capture 3d images of the real world (to create 3D models). AR creation devices and software applications include, for example, Adobe Aero, Vuforia, ARToolKit, Google ARCore, Apple ARKit, MAXST, Aurasma, Zappar, Blippar. AR cloud devices and software applications include, for example, Facebook, Google (world geometry, object recognition, predictive data), Amazon AR Cloud (commerce), Microsoft Azure, Samsung Project Where, Niantic, and Magic Leap.

One aspect of the following disclosure describes various overlays of surgical instrument operation aspects or functions onto a live video stream of a surgical area visualized through a laparoscopic camera surgical field during a minimally invasive surgical procedure. The overlays relate to the operation of one of the actively visualized surgical instruments or devices. The overlays combine aspects of tissue/organ interaction with functional data received from the surgical instruments used in the surgical procedure. The surgical instruments may include graspers, clamps, staplers, ultrasonic, RF, or combinations of each of these instruments. For graspers and clamps, aspects of the tissue parameters may include incomplete capture of tissue along with clamp status or clamp size. For surgical staplers, aspects of the tissue parameters may include tissue capture position of the surgical stapler, tissue compression, clamping, or firing sufficiency. For advanced energy devices such as ultrasonic or RF devices, aspects of the tissue parameters may include impedance, cauterization status, magnitude of bleeding, and aspects of the instrument functions may include, for example, energy level, timing, clamp pressure, among others. The augmented images shown in Figures 11-35 below may be viewed on a local display, a remote display, and/or an AR device, as described above in connection with Figures 1-10. Although the augmented images are described as being visualized through a laparoscopic camera during a minimally invasive surgical procedure, the images may be captured during non-invasive and invasive (e.g., open) surgical procedures without limiting the scope of the disclosure in this context. These aspects are described below.

11-75 illustrate various augmented images visualized through a laparoscopic camera during a minimally invasive surgical procedure. An augmented reality display system is used during a surgical procedure. The augmented reality display system includes an imaging device for capturing a real image of a surgical field during a surgical procedure, an augmented reality display for presenting an overlay of an operating aspect of a surgical instrument on the real image of the surgical field, and a processor. The overlay is related to an operating aspect of an actively visualized surgical instrument. The overlay combines an aspect of tissue interaction in the surgical field with functional data from the surgical instrument. The processor is configured to receive functional data for the surgical instrument, determine an overlay related to the operation of the surgical instrument, and combine an aspect of tissue in the surgical field with functional data from the surgical instrument. The augmented images indicate alerts for device performance considerations, alerts of incompatible use, and alerts for incomplete capture. Incompatible use includes out of range tissue conditions and tissue improperly balanced within the jaws of the end effector. Additional augmented images provide indications of incidents, including indications of tissue tension and indications of foreign body detection. Other augmented images show device status overlays and instrument indications.

11-75 also illustrate functional overlays of key operations or parameters of the surgical stapler, energy device, or instrument to clearly represent aspects of their interaction. In one aspect, the overlay data is adjusted by aspects detected by the surgical hub to modify the overlay from information simply detected by the source instrument to add context. In another aspect, the display may be adjusted or modified by the user, resulting in modification of the operation of the surgical instrument being monitored.

FIG. 11 shows an expanded image 100 of live footage of a surgical field 118 visualized through a laparoscopic camera during a minimally invasive surgical procedure, showing suitable tissue 112 captured between the jaws 110 of a surgical instrument end effector 108. It is. The mode of operation of the surgical instrument is to clamp tissue 112 between the jaws 110 of end effector 108. An aspect of the tissue 112 is proper capture of the tissue 112 between the jaws 110 of the end effector 108. Laparoscopic view 102 of surgical field 118 shows surgical instrument end effector 108 capturing tissue 112 in its jaws 110. Enhanced image 100 shows virtual graphical alert overlay 106 superimposed on anvil 110 of end effector 108. A virtual superimposed graphical alert overlay 106 indicates the amount of tissue grasped in the jaws 110 of the end effector 108 to inform the OR team of staple cartridge reload selections for stapling, overloading the tissue for energy, etc. . A first overlapping alert 104 signals that tissue 112 grasped by jaws 110 at the proximal end of end effector 108 is out of range. A second superimposed alert 116 signals that tissue 112 grasped by jaws 110 at the intermediate portion of end effector 108 is within reload range. A third superposition alert 114 signals that tissue 112 grasped by the jaws at the distal end of end effector 108 is above the cutting line. The superimposed graphical alert overlay 106 is applied to energy-based surgical instruments, surgical stapler instruments, manipulation tools, etc.

FIG. 12 is a live view of a surgical field 212 visualized through a laparoscopic camera during a minimally invasive surgical procedure, showing the tissue 206 position captured proximally between the jaws 210 of a surgical instrument end effector 204 as a tissue aspect. This is an expanded image 200 of the video. The tissue aspect includes tissue 206 that is accidentally captured or placed between the jaws 210 of the end effector 204. Laparoscopic view 202 of surgical field 212 shows surgical instrument end effector 204 grasping tissue 206 between jaws 210 of end effector 204 . The expanded image 200 includes a graphical alert overlay 208 superimposed on one of the jaws 210 of the end effector 204 to indicate that the tissue 206 is being grasped too proximally 206 by the jaws 210 of the end effector 204. shows. Although the superimposed alert 208 applies primarily to energy-based surgical instruments, a similar alert may be superimposed on a surgical stapler instrument, for example.

FIG. 13 shows a live video of a surgical field 324 visualized through a laparoscopic camera during a minimally invasive surgical procedure, showing poorly captured tissue 322 between the jaws 318 of a surgical instrument end effector 320 as a tissue aspect. This is an expanded image 300 of . Laparoscopic view 302 of surgical field 324 shows surgical instrument end effector 320 grasping tissue 322 with jaws 318 of end effector 320. The expanded image 300 shows a graphical alert overlay 304 superimposed on the image of the surgical field 324 to show insufficiently captured tissue 322 for the termination of the cut in the jaws 318 of the end effector 320.

The augmented image 300 also includes a first sub-image 308 showing a graphic image 306 of general anatomy superimposed on or adjacent to the surgical field 302 and a reference frame 310 of actual anatomy superimposed on or adjacent to the surgical field 302. The augmented image 300 also includes a second sub-image 312 showing the type of surgical instrument being used, the energy level if applicable, and the current surgical procedure. The second sub-image 312 may be superimposed on or located adjacent to the surgical field 302. The augmented image 300 shows an ultrasonic surgical instrument being used in the surgical procedure with the energy level set to a maximum of 5 to achieve advanced hemostasis. The graphic image 316 of the surgical instrument is shown superimposed on the graphic image 314 of the incomplete tissue capture alert overlay 304. Thus, the augmented image 300 provides several virtual objects to inform the OR team of insufficiently captured tissue 322 relative to the end of the cut. The superimposed incomplete tissue capture alert overlay 304 applies to energy-based surgical instruments as well as surgical stapler instruments, etc.

14 is another augmented image 330 of a live feed of a surgical field 359 visualized through a laparoscopic camera during a minimally invasive surgical procedure showing insufficiently captured tissue 348 between the jaws 358 of a surgical instrument end effector 354 as a tissue aspect. The laparoscopic view 332 of the surgical field 359 shows a surgical instrument end effector 354 with insufficiently captured tissue 352 relative to the end of the cut at the jaws 359, 358 of the end effector 354. The augmented image 330 shows a graphical alert overlay 352 superimposed on the tissue 348 to indicate the incomplete tissue capture. In the illustrated example, the jaws 358 of the end effector 354, e.g., anvil, include a series of light emitting diode (LED) indicators 356 to indicate the location of the tissue 352 grasped by the jaws 350, 358 of the end effector 354. The surgical view 332 also shows two rows of staples 334 of a previously stapled portion of the tissue 352.

The augmented image 330 includes a first sub-image 338 showing a graphic image 336 of the general anatomy shown in the laparoscopic view 332 and a reference frame 340 of the actual anatomy shown in the laparoscopic view 332. The augmented image 330 includes a second sub-image 342 showing the type of instrument being used and the surgical procedure. In the illustrated example, a powered vascular surgical stapler is being used in the vascular surgical procedure. The second sub-image 342 also shows a graphic image of a stapler cartridge 346 of the powered surgical stapler, and a graphic 344 superimposed on the graphic image of the stapler cartridge 346 to show the cut line of the powered surgical stapler.

FIG. 15 shows a live video of a surgical field 374 visualized through a laparoscopic camera during a minimally invasive surgical procedure, showing poorly captured tissue 372 between the jaws 370 of a surgical instrument end effector 368 as a tissue aspect. This is another expanded image 360 of . Laparoscopic view 362 of surgical region 374 shows surgical instrument end effector 368 with tissue 372 poorly captured relative to the termination of the cut in jaws 370 of end effector 368 . Enhanced image 360 shows a graphical alert overlay 366 superimposed on the anvil of end effector 368 to indicate incomplete tissue capture. A first overlapping alert 364 signals that the tissue 372 grasped by the proximal end of the jaws 370 of the end effector 368 is out of range. A second superimposed alert 378 signals that the tissue 372 grasped by the intermediate portion of the jaws 370 of the end effector 368 is within reload range. A third superposition alert 376 signals that the tissue 372 grasped by the distal end of the jaws 370 of the end effector 368 is above the cutting line.

FIG. 16 is an expanded image 400 of live footage of a surgical field 422 visualized through a laparoscopic camera during a minimally invasive surgical procedure, showing tissue 420 tension as a tissue aspect. Laparoscopic view 402 shows tissue 420 grasped between jaws 424 of surgical instrument end effector 418. Enhanced image 400 shows a graphical alert overlay 404 superimposed within surgical region 422 and coloring of tissue 420 to warn that grasped tissue 420 is under tension. The superimposed tissue tension alert 404 and the coloring of the tissue 420 grasped by the jaws 424 of the end effector 418 signals that the grasped tissue 420 is under tension. The superimposed tissue tension alert overlay 404 and coloring of tissue under tension 420 are applied to energy-based surgical instruments, surgical stapler instruments, and manipulation tools.

The expanded image 400 includes a first sub-image showing a graphical image 406 of the general anatomy shown in the laparoscopic view 402 and a reference frame 410 of the actual anatomy shown in the laparoscopic view 402. Contains image 408. The expanded image 400 also includes a second sub-image 412 that shows the type of surgical instrument in use, the energy level being applied if applicable, and the current surgical procedure. Enhanced image 400 shows an ultrasonic surgical instrument being used in a surgical procedure at up to five energy levels to achieve advanced hemostasis. A graphic image 416 of a surgical instrument is shown superimposed on the graphic image 414 of the low tension alert overlay 404. Therefore, the enhanced image 400 provides several options for informing the OR team to reduce tension in the captured tissue 420 relative to the end of the cut. The superimposed tension reduction alert overlay 404 is applied to energy-based surgical instruments as well as surgical stapler instruments and the like.

FIG. 17 is another expanded image 430 of a live view of a surgical field 442 visualized through a laparoscopic camera during a minimally invasive surgical procedure, showing tissue 444 tension as a tissue aspect. Laparoscopic view 432 shows surgical instrument end effector 436 grasping tissue 444 between jaws 438 of end effector 436. In the expanded image 430, coloring is added to the grasped tissue 444 to indicate that the grasped tissue 444 is under tension. Tissue 444 grasped by jaws 438 of end effector 436 is colored to indicate that tissue 444 is under tension. Laparoscopic view 432 also shows a row of previously applied staples 434. Also shown is a set of LEDs 440 for indicating the position of tissue 444 grasped by jaws 438 of end effector 436.

FIG. 18 is a plurality of graphic images 500 illustrating the jaw closed position as an operational mode of the surgical instrument shown in FIGS. 11-17. The plurality of graphic images 500 include static graphics and alerts superimposed on images of the surgical field and/or superimposed on the jaws of the end effector itself. The superimposed graphic image 500 is applied to energy-based surgical instruments, stapler-based surgical instruments, manipulation tools, etc.

First image 502 includes a first graphical overlay 504 showing patient information, treatment, and case ID. A second graphical overlay 506 signals the type of surgical stapler instrument being used in the surgical procedure, such as a surgical stapler having a closed staple height of 1.5 mm as shown. A third graphical overlay 508 communicates the degree of articulation of the surgical stapler. The fourth graphical overlay 510 is a pop-up error display. Finally, a fifth graphical overlay 512 informs of energy surgical instruments being used in the surgical procedure, such as ultrasound instruments operating at energy levels from a minimum of 3 to a maximum of 5.

The second image 514 includes the graphical overlays 506, 508, 510 described in the description of the first image 502, with the addition of a sixth graphical overlay 516 that indicates that the jaws of the surgical stapler are closed. Contains all 512.

A third image 518 shows the graphical overlays 506, 508 described in the description of the first image 502 with the addition of a sixth graphical overlay 520 to indicate that the jaws of the surgical stapler are partially closed. , 510, and 512.

A fourth image 522 includes the graphical overlays 506, 508, 510 described in the description of the first image 502, with the addition of a sixth graphical overlay 524 indicating that the jaws of the surgical stapler are open. Contains all 512.

FIG. 19 is an expanded image 530 of a live view of a surgical field 544 visualized through a laparoscopic camera during a minimally invasive surgical procedure, showing the jaws 536, 542 closed position as a mode of operation of the surgical instruments. The laparoscopic view 532 shows a partially closed surgical instrument end effector 540 with tissue 534 grasped between the jaws 536, 542 of the end effector 540 and the tissue used to assist in the tissue grasping process. An operation tool 546 is shown. Enhanced image 530 shows a graphical alert overlay 538 superimposed on end effector 540 indicating partial closure.

FIG. 20 shows a live video of the surgical area 544 shown in FIG. This is an extended image 550. A laparoscopic view of the surgical field 544 shows tissue 534 grasped by the jaws 536, 542 of the end effector 540.

21 is an augmented image 600 of a live feed of a surgical field 610 visualized through a laparoscopic camera during a minimally invasive surgical procedure showing clamping on metal or a foreign body as a surgical instrument's mode of operation. A laparoscopic view 602 of the surgical field 610 shows a surgical instrument end effector 604 clamped on metal or a foreign body embedded in tissue 608. The augmented image 600 shows a graphical alert overlay 614 superimposed on a graphical overlay 612 of the upper jaw 606 of the end effector 604 showing a foreign body clamped between the jaws of the end effector 604.

22 is an augmented image 700 of a live feed of a surgical field 714 visualized through a laparoscopic camera during a minimally invasive surgical procedure showing a device preheat warning where overheating is an operational aspect of the surgical instrument. The laparoscopic view of the surgical field 714 shows a surgical instrument end effector 706 having tissue 710 clamped between an ultrasonic blade 710 and a clamp arm 704. The end effector 706 shows a high temperature ultrasonic blade 710. The augmented image 700 includes a graphical alert overlay 708 superimposed on the high temperature ultrasonic blade 710. The graphical alert overlay 708 alerts the user that the ultrasonic blade 710 is accumulating preheat, which is transferred to the surrounding tissue 712 captured between the ultrasonic blade 710 and the clamp arm 704 pivotally coupled thereto.

FIG. 23 is an expanded image 800 of a live view of a surgical field 822 visualized through a laparoscopic camera during a minimally invasive surgical procedure, showing movement and flow of tissue 828 as a tissue aspect. A laparoscopic view of the surgical field 822 shows movement of the tissue 828 relative to the tissue graspers 804, 824 and the surgical instrument end effector 812, with the tissue 828 grasped by the jaws 824 of the end effector 812 and the jaws of the tissue graspers 804, 824. and flow. Enhanced image 800 shows graphical images 814, 820 superimposed on surgical region 822. The first graphical image 814 includes a surgical stapler icon 816 and text identifying the surgical instrument as a "stapler." Second image 820 includes an icon of Grasper 818 . Sub-image 810 alerts the OR team (eg, surgeon) to potential tissue movement above cutting line 808 displayed above the outline of stapler cartridge 806.

24 is an augmented image 900 of a live feed of a surgical field 918 visualized through a laparoscopic camera during a minimally invasive surgical procedure showing the geometric relationship of the transection to tissue 912 and to other firings. The laparoscopic view of the surgical field 918 shows the surgical instrument end effector 910 and the geometric relationship of the transection to tissue 912 grasped between the jaws 914, 916 of the end effector 910 and to other firings. The augmented image 900 shows a graphical image superimposed on the surgical field 918. The sub-image 908 includes graphics predicting a complete transection with two staple lines shown graphically as an active placement staple line 904 and a complete staple line 906.

FIG. 25 is an image 1000 showing anvil orientation communication as a mode of operation for a surgical instrument. The first image 1002 shows all of the graphical overlays 504, 506, 508, 510, 512 described in the description of the first image 502 shown in FIG. 18 with the addition of an anvil up/down indicator graphic 1004. . In the illustrated example, the up/down indicator graphic 1004 indicates that the anvil is in the up (eg, open) position. Another image 1006 shows anatomical structures. The second image 1008 includes all of the graphical overlays 504, 506, 508, 510, 512 described in the description of the first image 502 shown in FIG. An image 1006 of a scientific structure is shown. In the illustrated example, the up/down indicator graphic 1010 indicates that the anvil is in the down (eg, closed) position.

FIG. 26 is an image 1100 showing detected tissue thickness in the jaws of a surgical instrument end effector where the tissue thickness is a tissue aspect. Image 1100 shows a graphic screen 1102 with text 1114 that says tissue thickness is out of range for reloading, along with a recommendation and warning icon 1112 to consider changing the staple cartridge. Graphic screen 1102 shows detected thickness markers 1104 positioned along a scale 1106 indicating detected tissue thickness. Graphic screen 1102 also indicates the type of surgical instrument being used. The type of surgical instrument in use is a surgical stapler 1110 with a closed staple height 1108 of 1.5 mm.

FIG. 27 is an image 1200 of thermal gradient display graphics 1202, 1204, 1206 of an ultrasonic instrument. The first, second, and third display graphics 1202, 1204, 1206 may be overlaid on the surgical field of view displayed on the display described in FIGS. 11-26 to indicate the instrument type, minimum and maximum output energy levels, and display thermal gradient elements 1208, 1210, 1212 as part of the display graphics 1202, 1204, 1206. In the first display graphic 1202, the ultrasonic instrument is off or operating at normal ultrasonic blade temperature and the "green" gradient bar is not displayed or is barely visible. In the second display graphic 1204, the "heat map bar" of the thermal gradient element 1210 ranging from green to yellow indicates that the ultrasonic blade is approaching an above normal temperature. In the third display graphic 1206, the "heat map bar" of temperature gradient elements 1212 ranging from green to red indicates that the ultrasonic blade temperature is above normal in an overheated condition.

FIG. 28 is an image 1300 of temperature icon display graphics 1302, 1304, 1306 for an ultrasound instrument. First, second, and third display graphics 1302, 1304, 1306 may be overlaid on the surgical field of the surgical field of view displayed on the displays described in FIGS. , minimum and maximum output energy levels, and display temperature green, yellow, and red icons 1308, 1310, 1310 on portions of display graphics 1302, 1304, 1306. In the first display graphic 1302, the ultrasound instrument is operating under normal conditions and the temperature of the ultrasound blade is within normal ranges, as indicated by the green icon 1308 and also indicated by the green half-circle thermometer element 1314. It is within. In the second display graphic 1304, a yellow temperature icon 1310 indicates that the temperature is approaching above normal, as also indicated by a yellow half-circle thermometer element 1316. In the third display graphic 1306, a red temperature icon 1312 indicates that the ultrasonic blade temperature is above normal and is in an overheating condition as indicated by the red half-circle thermometer element 1318.

29 is an image 1400 of ultrasonic blade temperature graphic elements 1410, 1412, 1414 mapped to an end effector 1418 jaw 1420 location. The first, second, and third display graphic elements 1410, 1412, 1414 may be overlaid on the jaw 1420 of the ultrasonic end effector 1418 within a surgical field of view displayed on the display described in FIGS. 11-26. The first temperature graphic element 1410 indicates that the ultrasonic blade is operating within a normal range as indicated by the green temperature graphic element 1410. The second temperature graphic element 1412 indicates that the ultrasonic blade is approaching an overheated temperature condition as indicated by the yellow temperature graphic element 1412. The third temperature graphic element 1414 indicates that the ultrasonic blade is in an overheated temperature condition as indicated by the red temperature graphic element 1414.

FIG. 30 is an image 1500 of an ultrasonic generator power level display graphic 1502. The display graphic 1502 indicates the generator type and the minimum and maximum power levels. One or more graphical elements of the image 1500 may be overlaid on the jaws of an ultrasonic end effector or anywhere within the surgical field of the surgical field displayed on the displays described in FIGS. 11-26.

FIG. 31 is an image 1600 of an ultrasonic generator power level display graphic 1502 with a pop-up warning graphic 1602 indicating that the ultrasonic end effector jaws are overcrowded. Pop-up warning graphic 1602 may include language indicating that too much tissue is in the jaws, potentially resulting in an incomplete transverse incision. One or more graphical elements of image 1600 may be overlaid on the jaws of an ultrasound end effector or anywhere within the surgical field of the surgical field of view displayed on the display described in FIGS. 11-26. Good too.

32 is an image 1700 of an ultrasonic generator power level display graphic 1502 with a pop-up warning graphic 1702 indicating ultrasonic end effector jaw heat. The pop-up warning graphic 1702 may include language indicating that the instrument temperature is hot, and in one aspect, the word hot may be shown in red. One or more graphical elements of the image 1700 may be overlaid on the jaws of the ultrasonic end effector or anywhere within the surgical field of the surgical field displayed on the display described in FIGS. 11-26.

33 is an image 1800 of an electrosurgical generator display graphic 1802 having a pop-up warning graphic 1802 indicating an electrosurgical seal quality prediction. The pop-up warning graphic 1804 may include the language WARNING: INCOMPLETE SEAL and the electrosurgical generator display graphic 1802 may include the language ERROR to indicate that an incomplete seal is predicted. One or more graphical elements of image 1800 may be overlaid on the jaws of an electrosurgical end effector or anywhere within the surgical field of a surgical view displayed on the display described in FIGS. 11-26.

FIG. 34 is an image 1900 of a surgical stapler reload feedback. The image 1900 includes a first display graphic 1902 showing a suitable tissue thickness range for the surgical stapler in use. In the illustrated example, the suitable surgical stapler tissue thickness range is 1.5 mm to 2.4 mm. A second display graphic 1904 shows the detected tissue thickness. In the illustrated example, the detected tissue thickness is 2.6 mm. A third pop-up display graphic 1906 shows that the tissue thickness is 2.6 mm, which is thicker than the selected reload, and provides a suggestion to consider a reload in the range of 2.0 mm to 3.3 mm. The image 1800 may be overlaid on the jaws of an electrosurgical end effector or anywhere within the surgical field of a surgical field displayed on a monitor or screen. One or more graphic elements of the image 1900 may be overlaid on the jaws/anvil of a surgical stapler end effector or anywhere within the surgical field of a surgical field displayed on a display described in FIGS. 11-26.

35 is an image 2000 of a surgical stapler preload countdown. Image 2000 includes the first display graphic 1902 shown in FIG. 34, which shows a surgical stapler tissue thickness range of 1.5 mm to 2.4 mm. A second pop-up display graphic 2002 shows a surgical stapler preload countdown element to show the time remaining in the preload phase of the surgical stapler operation. One or more graphic elements of image 2000 may be overlaid on the jaws/anvil of a surgical stapler end effector or anywhere within the surgical field of the surgical field displayed on the display described in FIGS. 11-26.

The following description provides an intraoperative display for a surgical system to provide adaptability and adjustability or overlay instrument information. One aspect provides a functional overlay of key instrument operation or parameters to clearly represent each aspect of the surgical stapler or energy device or its interaction with tissue during a surgical procedure. The overlay data can be adjusted by aspects detected by the surgical hub to modify the overlay from information simply detected by the source instrument to add context. The display may be adjusted or modified by the user, resulting in a modification of the instrument operation being monitored as well.

36-68 illustrate an intraoperative display system for use during a surgical procedure. The system includes a surgical monitor having an intraoperative data display for the surgical area. The high energy generator is coupled to the high energy surgical instrument. High energy surgical instruments use radio frequency (RF) energy and ultrasound energy during surgical procedures on patients. The surgical hub is coupled to an advanced energy generator and a surgical monitor. The surgical hub provides live video of the surgical field for displaying live video of the surgical field via an intraoperative data display. The intraoperative data display displays a view of the surgical field including a high energy surgical instrument grasping tissue and a panel overlay displaying information specific to the high energy surgical instrument.

In one aspect, the intraoperative data display includes an end effector of a surgical instrument grasping tissue and a panel overlay displaying case information, system notifications, or device panels, or any combination thereof, overlaid on live surgical footage. and The position, opacity, size, and placement of panel overlays are customized. Panel overlays are configured to be turned on and off individually or as a group. The panel overlay is further configured to dynamically change to indicate a change in status, such as device activation or power level adjustment. The panel overlay shows Optimum Device Performance (ODP) guide images or other Instructions for Use (IFU)/information sources.

In various aspects, the panel overlay comprises at least one of data entry information from capital equipment, generators, ventilators, smoke extractors, electronic health records, laparoscopes, computers, surgical devices, wired and wireless connected devices, surgeon profile preferences that may be saved, recalled, or edited, or any combination thereof. The panel overlay may include case information including at least one of patient name, surgeon name, case time, or instrument activation, or any combination thereof. The panel overlay may include system notifications including at least one of connected instrument status, minor error alerts, medium error alerts, or major error alerts, or any combination thereof. The panel overlay may include information associated with surgical instruments connected to the system to provide advanced hemostasis. The panel overlay may include a visible patient panel overlay. The panel overlay may include a device panel overlay including at least one of device name, device settings, or device supplemental features, or any combination thereof. The panel overlay may include multiple panel overlays in a stacked configuration. The panel overlay may include multiple panel overlays in an extended configuration. The panel overlay may display device troubleshooting information. The panel overlay may display at least one of alerts, warnings, device information, or device features, or any combination thereof.

In another aspect, the intraoperative data display includes a secondary configurable panel. The secondary configurable panel changes dynamically based on the selected and customized laparoscopic overlay field displayed in the surgical field of view of the live surgical video area of the intraoperative data display. The customized laparoscopic overlay field comprises at least one of a bottom edge panel, a top left corner panel, a top center panel, or a side edge panel, or any combination thereof.

FIG. 36 is a system diagram 3000 of an operating room with a surgical monitor having an intraoperative data display 3002 for a surgical area. High energy generator 3004 is coupled to surgical hub 3006 and high energy surgical instrument 3008. High energy surgical instrument 3008 uses RF and ultrasound energy during surgical procedures on patient 3010. Surgical hub 3006 provides live video 3014 of the surgical field displayed by intraoperative data display 3002. Intraoperative data display 3002 displays a view of the surgical field that includes a high energy surgical instrument 3008 grasping tissue and a panel overlay 3012 that displays information specific to high energy surgical instrument 3008.

FIG. 37 is an expanded image 3100 of live video of a surgical field visualized through a laparoscopic camera during a minimally invasive surgical procedure, displayed on an intraoperative data display 3002. The laparoscopic view shows end effector 3108 of surgical instrument 3008 grasping tissue 3110. Enhanced image 3100 shows a three panel overlay displaying case information 3102, system notifications 3104, and device panel 3106 overlaid on the live surgical video.

Panel overlays 3102, 3104, 3106 are displayed over the live surgical video. The position, opacity, size, and placement of panel overlays 3102, 3104, 3106 can be customized. Panel overlays 3102, 3104, 3106 may be turned on/off individually or as a group. Panel overlays 3102, 3104, 3106 may be opaque or have varying levels of transparency. Panel overlays 3102, 3104, 3106 may include data input information from capital equipment, generators, ventilators, smoke extractors, electronic health records, laparoscopes, computers, surgical devices, wired and wireless connectivity devices. Surgeon profile preferences may be saved, recalled, or edited. In another aspect, general screen settings, including overall screen settings fonts for the sizing of panel overlays 3102, 3104, 3106, may be configurable based on surgeon preferences via the staff console. . The features of panel overlays 3102, 3104, 3106 on all screen displays may be enabled/disabled or bypassed, for example, through dedicated physical switches.

FIG. 38 is a detailed view of the case information panel overlay 3102 shown in FIG. 37. Case information panel overlay 3102 may be selectively enabled/disabled via the staff console. Specific details of the case information panel overlay 3102, including patient name, surgeon name, case time, instrument activation, etc., may be selectively enabled/disabled via the staff console. Other aspects of the case information panel overlay 3102 include, for example, activating the screen at the beginning of a surgery and disabling the screen after certain criteria are met (elapsed surgical time, number of device activations, etc.). obtain.

FIG. 39 is a detailed view of the system notification panel overlay 3104 shown in FIG. 37. System notification panel display overlay 3104 may include variable warning symbols based on the severity of warnings/alerts/information. Messages may potentially be based on any combination of connected capital equipment, connected devices (medical, wired, wireless, etc.), or surgical hub devices (this device). Notifications disappear based on notification dwell time or alert condition resolution.

FIG. 40 is an image of some example system notification panel overlays 3104. The system notification panel overlays 3104 may include, for example, connected instruments 3112, minor error alerts 3114, moderate error alerts 3116, and major error alerts 3118.

FIG. 41 is a detailed view of the device panel overlay 3106 shown in FIG. 37. The exemplary device panel overlay 3106 shown in FIG. 41 displays information associated with an ultrasound/RF coupled instrument connected to the system 3000 (FIG. 36) to provide advanced hemostasis. The device panel overlay 3106 provides, for example, heat indications and ready indications.

In various aspects, the device panel overlay 3106 provides an Ottawa Framework-inspired visual concept and can be selectively enabled/disabled via the staff console. The staff console may also selectively enable/disable individual panels, such as energy and surgical staplers. In one aspect, device panel overlay 3106 appears only when the associated appliance is connected to system 3000 (FIG. 36). Device panel overlay 3106 is not visible when nothing is connected to system 3000 (FIG. 36). The space occupied by device panel overlay 3106 is configurable (eg, left, centered, right aligned). Device panel overlay 3106 may also be configured to display connectivity to handheld instruments. In another aspect, the device panel overlay 3106 may be configurable within the space along the top/bottom of the display. Device panel overlay 3106 may display multiple instrument devices connected simultaneously. In one aspect, the device panel overlay 3106 may display information only for devices that are actively being used by the surgeon so that connected but inactive device displays are hidden.

In one aspect, the size of device panel overlay 3106 may be configurable based on enabled features, such as the thermal example described herein. These features may be enabled or disabled based on specific device design (eg, future models) and may be enabled/disabled by the surgeon or paid subscription.

42 is an augmented image 3120 of a live feed of a surgical field visualized through a laparoscopic camera during a minimally invasive surgical procedure displayed on an intraoperative data display 3122 showing an end effector 3108 of a surgical instrument 3008 grasping tissue 3110 and a screen with a visible patient panel overlay 3124, in accordance with at least one aspect of the present disclosure. The visible patient panel overlay 3124 may require an additional application to display content. The intraoperative data display 3122 also displays a case information panel overlay 3102 and a device panel overlay 3106. In the illustrated example, the system notification panel overlay 3104 is hidden.

FIG. 43 is a schematic diagram of the energy device image panel 3106 architecture. In one aspect, energy device image panel 3106 includes, for example, device name 3132, device settings 3134, and device supplementary features 3136.

Figure 44 is an image 3140 of supplemental device alert/warning/information in a stacked configuration. Supplemental device alert/warning/information displays may be stacked. Supplemental device alert/warning/information displays may also be stacked vertically to cover the main device panel or may be expanded horizontally. As shown in the example of Figure 44, the energy device panel overlay 3106 is stacked vertically below the system ready 3142 display.

FIG. 45 is a supplementary device alert/warning/information image 3150 in an expanded configuration. Supplemental device alerts/warnings/information displays may be expanded. Magnified display 3144 may be expanded vertically or horizontally. As shown in the example of FIG. 45, enlarged display 3144 provides additional information about the instrument device. As shown in the example of FIG. 45, enlarged display 3144 shows an image of a surgical instrument and instructions to activate the instrument for two seconds to perform a test.

FIG. 46 is an appliance status image panel 3160 that illustrates how appliance panel states dynamically change to indicate status changes such as device activation or power level adjustment. The left side 3162 of the instrument state image panel 3160 shows the default state of the instrument. The right side 3164 of the instrument status image panel 3160 shows the active state of the instrument. In the example shown in FIG. 46, a first panel 3166 on the right side 3164 of the instrument status image panel 3160 shows dynamic changes in instrument activation, and a second panel 3168 on the right side 3164 of the instrument status image panel 3160 shows the Showing dynamic changes in power level.

Figure 47 is a system diagram 3170 that converts generator 3066 alerts and warnings 3174 to the laparoscopic monitor 3172 and displays them on the local interface.

FIG. 48 is a diagram 3180 of a series of screens 3182 of existing alerts shown on the current generator transmitted to the surgical hub, which then displays these as a series of screens 3184 on the local interface. do.

FIG. 49 is a schematic diagram of a system 3200 that includes a generator 3202 in communication with a digital hub 3204, which then displays screen and alert data 3220 on a local interface 3206, such as a laparoscopic screen 3216. The generator 3202 includes a controller 3208 and a screen 3210. The controller 3208 of the generator 3202 transmits data 3218 to a controller 3212 of the digital hub 3204. The digital hub 3204 also includes a screen 3214. The controller 3212 of the digital hub 3204 transmits screen and alert data 3220 to a laparoscopic screen 3216 of the local interface 3206.

50 is an augmented image 3230 of a live view of the surgical field as visualized through a laparoscopic camera during a minimally invasive surgical procedure, displayed on an intraoperative data display 3232. The laparoscopic view of the surgical field shows an overlay panel 3234 showing an optimal device performance (ODP) guide image or other instruction for use (IFU)/information source. The augmented image 3230 shows an on-screen display overlay panel 3234 showing a surgical device 3250 comprising a grip housing 3244, a trigger 3242, a shaft 3238, and a blade with a tissue pad 3236. The shaft 3238 is rotated by a rotation knob 3248. The surgical device 3250 also includes an energy button 3240 and an energy button 3246 with advanced hemostasis.

Overlay panel 3234 displays images of surgical device 3250 and ODP guide images or other IFU/information sources. The image may be presented to the surgeon along with surgical device information. The image may be a still image or a moving image. The image may provide general surgical device 3250 information (as shown) or context-specific surgical device information. For example, the emergency exit door of surgical device 3250 may include a sensor to detect removal. When the emergency exit door is removed, the on-screen display overlay panel 3234 shows an image that provides instructions regarding proper use of the emergency exit mechanism. As another example, a surgeon encounters an alert while using ultrasound energy associated with surgical device technology. On-screen display overlay panel 3234 shows information specific to how to best use surgical device 3250 to avoid that alert.

FIG. 51 is a display screen 3300 showing an intraoperative data display 3302 with a secondary bottom edge configurable panel 3312. Intraoperative data display 3302 includes a surgical view of live surgical video 3303 with customized laparoscopic overlay fields 3318 , 3320 , 3322 and secondary bottom edge configurable panel 3312 . Intraoperative data display 3302 displays a navigation button 3304 for returning to the home screen. Intraoperative data display 3302 also provides selectable surgeon profiles 3306. Surgeon profile preferences may be saved, recalled, or edited. Intraoperative data display 3302 also provides several connectivity status indicators 3308. Overlays associated with intraoperative data display overlay fields 3318, 3320, 3322 may be toggled with display overlay toggle button 3310.

The intraoperative data display 3302 also includes a secondary configurable panel that dynamically changes based on the selected customized laparoscopic overlay fields 3318, 3320, 3322 displayed in the surgical view of the live surgical video 3303 area of the intraoperative data display 3302 when the display overlay toggle button 3310 is toggled to the on position. In the example shown in FIG. 51, the bottom edge configurable panel 3312 is displayed by selecting the bottom edge selection field 3322 in the surgical view of the live surgical video 3303 portion of the intraoperative data display 3302. For example, different portions of the display may be selected, including the top left corner selection field 3318, the top center selection field 3320, and the bottom edge selection field 3322.

Bottom configurable panel 3312 includes a panel alignment bar 3314 for aligning bottom configurable panel 3312 to the left, center, or right position, shown here in the left position. Help indicator 3316 provides context information for associated toggle buttons 3324, 3326, 3328, 3330. The bottom edge selection field 3322 may be configured using configurable panel alignment buttons 3314 that shift the bottom edge alignment to the left, center, and right. In another aspect, the bottom edge selection field 3322 may move up or down, for example. In addition to the configurable panel alignment buttons 3314, the bottom edge configurable panel 3312 enables/disables an energy panel display for ultrasound/RF energy tool devices, including alerts, shown here in the on position. A first toggle button 3324 is provided to disable. A second toggle button 3326 enables/disables the display of alerts for ultrasound/RF energy tool devices only, shown here in the off position. A third toggle button 3328 enables/disables the surgical stapler tool device's display, including alerts, here shown in the off position. A fourth toggle button 3330 enables/disables display alerts for surgical stapler tool devices only, here shown in the on position.

FIG. 52 is an alternative bottom edge configurable panel 3400. Alternative base edge configurable panel 3400 is displayed by selecting base edge selection field 3282 within the surgical field of view of live surgical video 3263 portion of intraoperative data display 3262 of FIG. An alternative bottom edge configurable panel 3400 may be configured for energy status and alert customization. In the illustrated example, alternative bottom edge configurable panel 3400 is displayed by toggling the display on monitor toggle button 3402 to the on position. A first toggle button 3404 enables/disables the energy device panel state, here shown in the on position. A second toggle button 3406 enables/disables energy device panel alerts, here shown in the off position. A third toggle button 3408 enables/disables the surgical stapler device state, here shown in the off position. A fourth toggle button 3410 enables/disables the surgical stapler alert, here shown in the off position. Alternative bottom edge configurable panel 3400 includes a panel alignment bar 3412 for aligning alternative bottom edge configurable panel 3400 in a left, center, or right position, shown here in the center position. 53-55, discussed below, illustrate additional configurable panels that are displayed by selecting other selection fields within the surgical field of view of the live surgical video portion of the intraoperative data display.

FIG. 53 is a display screen 3500 showing an intraoperative data display 3502 with a secondary upper left corner configurable panel 3522. Intraoperative data display 3502 includes a surgical view of live surgical video 3503 with customized laparoscopic overlay fields 3318 , 3320 , 3322 and a secondary upper left corner configurable panel 3522 . Intraoperative data display 3502 includes navigation buttons for returning to the home screen, selectable surgeon profiles, and connectivity status indicators, as described in connection with FIG. 51. Overlays associated with intraoperative data display overlay fields 3318, 3320, 3322 may be toggled with display overlay toggle button 3310.

The intraoperative data display 3502 also includes a secondary configurable panel that dynamically changes based on the selected customized laparoscopic overlay fields 3318, 3320, 3322 displayed in the surgical view of the live surgical video 3503 area of the intraoperative data display 3502 when the display overlay toggle button 3310 is toggled to the on position. In the example shown in FIG. 53, the top left corner configurable panel 3522 is displayed by selecting the top left corner selection field 3318 in the surgical view of the live surgical video 3503 portion of the intraoperative data display 3502.

Selection of the top left corner selection field 3318 dynamically changes the visual display. For example, selection of the top left corner selection field 3318 displays the case information overlay 3504, which in this example is the case information panel overlay screen 3102 shown in FIGS. 37 and 38. Various toggle buttons are shown along the right vertical portion of the secondary top left corner configurable panel 3522. As previously described, the display overlay toggle button 3310 enables/disables the display of overlay screens such as the case information overlay 3504. The first toggle button 3506 enables/disables all case information from the case information overlay 3504, shown here in the on position. The second toggle button 3508 enables/disables patient name information from the case information overlay 3504, shown here in the on position. The third toggle button 3510 enables/disables surgeon name information from the case information overlay 3504, shown here in the on position. A fourth toggle button 3512, shown here in the on position, enables/disables case data information from the case information overlay 3504. A fifth toggle button 3514, shown here in the on position, enables/disables total time of case information from the case information overlay 3504. A sixth toggle button 3516, shown here in the on position, enables/disables case metrics, which may include total number of device activations, usage, or other device metrics, from the case information overlay 3504.

54 is a display screen 3600 showing an intraoperative data display 3602 with a secondary top center configurable panel 3622. The intraoperative data display 3602 includes a surgical view of the live surgical video 3603 with customized laparoscopic overlay fields 3318, 3320, 3322 and a secondary top center configurable panel 3622. The intraoperative data display 3602 includes a navigation button to return to the home screen, a selectable surgeon profile, and a connection status indicator, as described in connection with FIG. 51. The overlays associated with the intraoperative data display overlay fields 3318, 3320, 3322 may be toggled with the display overlay toggle button 3310.

The intraoperative data display 3602 also displays a selected and customized laparoscopic overlay field 3318 displayed in the surgical field of the live surgical video 3603 area of the intraoperative data display 3602 when the display overlay toggle button 3310 is toggled to the on position; Also includes secondary configurable panels that dynamically change based on 3320, 3322. In the example shown in FIG. 54, the top center configurable panel 3622 is displayed by selecting the top center selection field 3320 within the surgical field of view of the live surgical video 3603 portion of the intraoperative data display 3602.

Selecting the top center selection field 3320 dynamically changes the visual display. For example, selecting the top center selection field 3320 displays the system notification panel overlay 3604, which in this example is the system notification panel overlay 3104 shown in FIG. 40. Various toggle buttons are shown along the right vertical portion of the secondary top center configurable panel 3622. As previously described, the display overlay toggle button 3310 enables/disables the display of an overlay screen, such as the system notification panel overlay 3604. The first toggle button 3506 enables/disables all alerts for the system notification panel overlay 3604, shown here in the on position. The second toggle button 3608 enables/disables information alerts for the information notification bar 3616 of the system notification panel overlay 3604, shown here in the on position. The third toggle button 3610 enables/disables minor alerts for the minor error alert bar 3618 of the system notification panel overlay 3604, shown here in the on position. A fourth toggle button 3612, shown here in the on position, enables/disables a medium alert for a medium error alert bar 3620 of the system notification panel overlay 3604. A fifth toggle button 3614, shown here in the on position, enables/disables a critical alert for a critical error alert bar 3622 of the system notification panel overlay 3604. Dynamic shading of the toggle buttons, such as the fifth toggle button 3614 as shown, provides visual cues based on the selection state.

FIG. 55 is a display screen 3700 showing an intraoperative data display 3702 with a secondary lateral configurable panel 3722. FIG. Intraoperative data display 3702 includes a surgical view of live surgical video 3703 with customized laparoscopic overlay fields 3318 , 3320 , 3322 , 3716 and secondary lateral configurable panel 3722 . Intraoperative data display 3702 includes navigation buttons for returning to the home screen, selectable surgeon profiles, and connectivity status indicators, as described in connection with FIG. 51. The overlays associated with intraoperative data display overlay fields 3318, 3320, 3322, 3716 may be toggled with display overlay toggle button 3310.

The intraoperative data display 3702 also displays a selected and customized laparoscopic overlay field 3318 displayed in the surgical field of the live surgical video 3703 area of the intraoperative data display 3702 when the display overlay toggle button 3310 is toggled to the on position; Also includes secondary configurable panels that dynamically change based on 3320, 3322, 3716. In the example shown in FIG. 55, the side edge configurable panel 3722 is displayed by selecting the side edge selection field 3716 within the surgical field of view of the live surgical video 3703 portion of the intraoperative data display 3702.

Selecting the side edge selection field 3716 dynamically changes the visual display. For example, selecting the side edge selection field 3716 displays the visible patient panel overlay 3704, which in this example is the visible patient panel overlay 3124 shown in FIG. 42. Various toggle buttons are shown along the right vertical portion of the secondary side edge configurable panel 3722. As previously described, the display overlay toggle button 3310 enables/disables the display of overlay screens such as the visible patient panel overlay 3704. The first toggle button 3706 enables/disables all alerts for the visible patient panel overlay 3704 information, shown here in the on position. The second toggle button 3708 enables/disables gas information for the visible patient panel overlay 3704, shown here in the on position. The third toggle button 3710 enables/disables visible patient information for the visible patient panel overlay 3704, shown here in the on position. A fourth toggle button 3712, shown here in the on position, enables/disables information from another module (module 3) for the visible patient panel overlay 3704. A fifth toggle button 3714, shown here in the on position, enables/disables information from yet another module (module 4) for the visible patient panel overlay 3704.

FIG. 56 is a series of image panels 3800 displaying device troubleshooting information. First image panel 3802 displays instructions for testing the instrument. In the illustrated example, the instructions to the instrument are "Activate the instrument for 2 seconds to run the test." A second image panel 3804 shows that the test is in progress and a third image panel 3806 shows the test status. A fourth image panel 3808 provides instructions for troubleshooting with still images, animations, GIFs, or videos. In the illustrated example, image panel 3808 displays an animation for "tighten assembly." The fifth image panel 3810 shows the motion for the instrument under test, and in the illustrated example, signals to "open jaws during test."

FIG. 57 is a series of image panels 3900 displaying articulating surgical stapler features. A first image panel 3902 shows surgical stapler joint angles of a surgical stapler device. The angle may be indicated numerically or visually. The second and third image panels 3904, 3906 show updates of joint angles based on device state and also show joint orientation. In one aspect, updated measurements are sent to the digital hub 3204 (FIG. 49), 3006 (FIG. 36) each time a joint button is pressed or the joint angle changes. Digital hubs 3204 (FIG. 49), 3006 (FIG. 36) then use the updated measurements to update the reflection angles, as shown in image panels 3902, 3904, 3906.

The fourth image panel 3908 is a standardized countdown indicator that appears when the surgical stapler jaws are closed. The countdown dynamically changes based on time. The device may be fired at any time during the countdown sequence. The fifth image panel 3910 indicates that the device is ready to fire. The sixth, seventh, and eighth image panels 3912, 3914, 3916 show the knife position along the sled. The knife position is shown in gray over a diagram of the cartridge 3918 and dynamically changes based on the device. The diagram of the cartridge 3918 may be generic or specific to the cartridge installed in the surgical stapler. The knife position/surgical stapler image panels 3912, 3914, 3916 may dynamically change based on the type and size of the wirelessly connected surgical stapler. The knife position algorithm is executed after or when the firing trigger of the surgical stapler is pressed and as the surgical stapler begins to fire, the knife and sled begin to move along the length of the surgical stapler. The ninth image panel 3920 shows that the action is complete.

Each of the supplemental features displayed by a corresponding image panel 3902-3920 of a connected device is dynamically updated based on the current state of the device.

FIG. 58 is an alert/warning/information image panel 3922 that displays that a joint limit has been reached.

FIG. 59 is an alert/warning/information image panel 3924 that displays that the device is in lockout mode.

FIG. 60 is an alert/warning/information image panel 3926 indicating that the device cannot articulate when the jaws are closed.

FIG. 61 is a device image panel 4000 showing articulating surgical stapler features. Device image panel 4000 includes a device panel name 4002, shown here as surgical stapler 3000, and a device panel supplementary feature 4004, shown here as a countdown indicator.

FIG. 62 is a stacked alert/warning/information image panel 4010 displayed in a stacked configuration with a device alert indicating that a joint limit has been reached. The stacked image panel 4010 includes a top image panel 4012 indicating that a joint limit has been reached. The stacked image panel 4010 includes a bottom image panel 4014 indicating the device name and type, the joint angle, and the joint direction.

FIG. 63 is a schematic diagram of a system 4020 that includes a surgical stapler communicating with a digital hub via Bluetooth to execute the algorithms for the countdown timer image panel 4000 shown in FIGS. 57 and 61. Surgical stapler 4022 communicates with digital hub 4024 via a wireless network, such as Bluetooth. A surgical stapler controller 4028 is provided, and the digital hub 4024 is provided with a controller 4030. Surgical stapler controller 4028 transmits wireless data 4026 to digital hub controller 4030. When surgical stapler 4022 clamps and sends a clamp status message to digital hub 4024, digital hub 4024 uses the clamp status message as a start command to start a countdown timer. If an unclamp message is received, the timer restarts to 0 and stops incrementing. If the timer reaches the set time, the timer is stopped and a flag is set.

FIG. 64 is a series of device image panels/alerts 4030 displaying ultrasound instrument characteristics. The device image panel is expandable or collapsible depending on the features enabled or disabled by the user and/or by the connected device. As an example, the first device image panel 4032 can be expanded into a second device image panel 4034 that displays a default thermal image panel 4036 and a high thermal image panel 4038. The high thermal image panel 4038 displays the state of the device temperature either by direct means (temperature sensing) or indirectly through other methods (algorithms based on time or other methods). Referring to default thermal image panel 4036, the thermal algorithm may be based on device activation status and total elapsed time, as well as time elapsed since last activation. In other implementations, the thermal algorithm may be based on device characteristics and/or conditions, such as resonant frequency, estimated conditions of the device, or direct measurements. Thermal status is only displayed while the device is not currently booting up. The display is grayed out while the device is inactive. The thermal status is always displayed on the high thermal image panel 4038 and corresponds to the current status of the device. In the illustrated example, the device is an ultrasound instrument.

The third image panel 4040 changes color when the device is being used to indicate the mode of device operation, as shown by internal image panels 4042, 4044, 4046. A fourth device image panel 4048 displays the instrument alert and associated images, as well as text for the alert, as shown in alert image panel 4050. Alert image panel 4050 may provide visual indicators for alerts. An alert may consist of only text, only images, or a combination of text and images.

FIG. 65 is a chart 4060 illustrating pairing of a surgical stapler instrument. Surgical stapler pairing may be implemented as user selectable pairing, power-on connect, power-on connect with a button press, or as an RFID token. With user selectable pairing, the user initiates pairing mode on the capital equipment via a touch screen. The capital equipment then attempts to identify all valid devices of that type in the area, and the user simply selects the device they wish to connect to. The broadcast continues even after a period of time has elapsed. Power-on connect pairing can enable the Bluetooth radio on power-up and can turn it off if it is not connected to a client within a predetermined time. The capital equipment constantly scans for new devices and connects them. Power-on connect with a button press requires the Bluetooth radio to be enabled on power-up and turns off the radio if it is not connected to a client within a predetermined time. The capital equipment has a pairing button that scans/connects devices once pressed. With RFID token pairing, users utilize an RFID card packaged with each device as a method for scanning and pairing new devices with capital equipment.

Figure 66 is an image of screen 4070 displaying paired device information. Screen 4070 provides a system-wide ability to connect/disconnect various wired and wireless devices/appliances.

FIG. 67 is an image of a wireless surgical device including a unique identifier for pairing wireless device 4080.

FIG. 68 is an image of a screen 3090 displaying a link to an Optimal Device Performance (ODP) guide image or other electronic instruction manual (e-IFU). User manual 4092 may be linked to the e-IFU or ODP.

Referring also to FIGS. 1-68, FIG. 69 is an illustration of an augmented reality method 5000 using a surgical instrument 77 and an augmented reality display 89 for use during a surgical procedure, according to one aspect of the present disclosure. The method 5000 may be employed in conjunction with any of the enhanced displays shown in FIGS. 11-48, 50-62, 64-66, and 68; and the systems shown in FIG. 63. In one aspect, method 500 provides an overlay of operational aspects or functionality of surgical instrument 77 onto a surgical laparoscopic video stream. The overlay may relate to the operation of one of the surgical instruments 77 being actively visualized, and the overlay combines aspects of tissue/organ interaction with functional data received from the surgical instrument 77. In another aspect, the surgical instrument 77 may be a grasper or a clamp, and the tissue aspect may be incomplete capture of the tissue along with the clamp condition or the size of the clamp. In another aspect, the surgical instrument 77 may be a surgical stapler, the tissue aspect may be a tissue capture position or tissue compression, the surgical stapler aspect may be a clamp or firing sufficiency, or other may be a parameter. In another aspect, the surgical instrument 77 may be a high energy device, the tissue parameters may be impedance, ablation status, hemorrhage magnitude, and the functionality of the surgical instrument 77 may be, among other parameters. It can be energy level, timing, clamping pressure, among others.

In one aspect, the method 5000 is directed to overlaying data according to the use of the surgical instrument 77. According to the method 5000, the imaging device 38 captures 5002 an actual image of the surgical field during a surgical procedure. The processor 85 receives 5004 functional data from the surgical instrument 77, determines 5006 an overlay associated with the operating behavior of the surgical instrument 77, and combines 5008 the behavior of tissue in the surgical field with the functional data received from the surgical instrument 77. The augmented reality display 89, or the local display 67, presents 5010 an overlay of the operating behavior of the surgical instrument 77 on the actual image of the surgical field. The functional data of the surgical instrument 77 may be received directly from the surgical instrument 77 or from a surgical hub coupled processor or server.

Referring also to FIGS. 1-68, FIG. 70 is an illustration of an augmented reality method 5100 using a surgical instrument 77 and an augmented reality display 89 for use during a surgical procedure, according to one aspect of the present disclosure. The method 5100 may be employed in conjunction with any of the enhanced displays shown in FIGS. 11-48, 50-62, 64-66, and 68; and the systems shown in FIG. 63.

In one aspect, method 5100 is directed to overlaying data according to utilization of surgical instrument 77. Processor 85 monitors 5102 the performance of surgical instrument 77 during the surgical procedure. Processor 85 determines 5104 the use of surgical instrument 77. Augmented reality display 89 displays alerts 5112 regarding performance considerations of surgical instrument 77. Processor 85 determines 5122 contingencies, displays 5132 a status overlay of surgical instrument 77, and displays 5134 an indication of surgical instrument 77 on augmented reality display 89.

According to method 5100, imaging device 38 captures 5002 a real image of a surgical area during a surgical procedure. Processor 85 receives 5004 functional data for surgical instrument 77 , determines 5006 an overlay associated with operational aspects of surgical instrument 77 , and combines 5006 aspects of tissue within the surgical region with the functional data received from surgical instrument 77 . Combine 5008. Augmented reality display 89, or local display 67, presents 5010 an overlay of the operational aspects of surgical instrument 77 onto a real image of the surgical field. Functional data for surgical instrument 77 may be received directly from surgical instrument 77 or from a surgical hub coupled processor or server.

When the processor 85 determines to use the surgical instrument 77 5104 , the processor 85 determines 5106 whether the tissue grasped by the jaws of the surgical instrument 77 is within range of the jaws and the tissue is within the range of the jaws of the surgical instrument 77 . determines 5108 whether the surgical instrument 77 is properly balanced in its jaws and displays 5118 a non-conforming use alert according to the state of use of the surgical instrument 77. If the tissue is out of range, processor 85 displays 5116 a tissue out of range alert on augmented reality display 8. If tissue is incorrectly balanced within the jaws of surgical instrument 77, processor 85 displays 5118 an incorrect balance alert on augmented reality display 89. As part of determining 5104 the use of surgical instrument 77, processor 85 determines whether tissue capture between the jaws of surgical instrument 77 is complete and, if not, incomplete tissue capture. 5110 to display capture alert.

According to method 5100, processor 85 determines 5122 tissue tension and associated events such as foreign object detection. If processor 85 determines 5124 that tissue tension is too high, augmented reality display 89 displays 5126 a tissue tension alert. If processor 85 detects 5128 a foreign object in the jaws of surgical instrument 77, augmented reality display 89 displays 5130 a foreign object detection alert. In either case, augmented reality display 89 displays 5132 a surgical instrument 77 status overlay according to the results of the above determination of tissue tension and foreign object detection. Finally, augmented reality display 89 displays 5134 a surgical instrument 77 indication.

With reference also to FIGS. 1-68, FIG. 71 is an illustration of an augmented reality method 5150 using a surgical instrument 77 and an augmented reality display 89 for use during a surgical procedure, according to one aspect of the present disclosure. The method 5100 may be employed in conjunction with any of the augmented displays shown in FIGS. 11-48, 50-62, 64-66, and 68, and may be implemented with any of the systems shown in FIGS. 1-10, 49, and 63.

In one aspect, the method 5150 is directed to a functional overlay of key operations or parameters of the surgical instrument 77 (e.g., surgical stapler, energy device) to clearly represent any aspect of the interaction between the surgical stapler 77 and tissue within the surgical field. In one aspect, the overlay data may be adjusted by aspects detected by the surgical hub 6 to modify the overlay from information simply detected by the source surgical instrument 77 to add context. In another aspect, the augmented display may be further adjusted or modified by the user, which may result in modification of the surgical instrument 77 being monitored during the surgical procedure.

In one aspect, the method 5150 is directed to overlaying data according to the function of the surgical instrument 77 of critical operations or parameters. According to the method 5150, the imaging device 38 captures 5152 a real image of the surgical field during a surgical procedure. The processor 85 receives 5154 functional data from the surgical instrument 77, determines an overlay related to the functional aspects of the surgical instrument 77 5156, and combines 5158 aspects of tissue in the surgical field with the functional data received from the surgical instrument 77. The augmented reality display 89, or local display 67, presents 5160 an overlay of the functional aspects of the surgical instrument 77 or aspects of the interaction of the surgical instrument 77 with tissue on the real image of the surgical field. The functional data of the surgical instrument 77 may be received directly from the surgical instrument 77 or from a surgical hub coupled processor or server. In one aspect, the processor 85 may modify 5162 the overlay data according to aspects detected by the surgical hub to provide context for the surgical procedure. In another aspect, the processor 85 may modify 5164 the functionality of the surgical instrument based on the user modifications 5162.

In accordance with methods 5000, 5100, 5150 shown in FIGS. 69-71, the processor configures a data overlay to be displayed by the augmented reality display 89, as shown in FIGS. 36-68 above. The augmented reality display 89 also displays the dynamics of the surgical instrument 77 function, as shown in FIGS. 36-68.

Referring to FIGS. 1-71 above, the present disclosure now moves to a description of visual overlays of data specific to the conditions or instruments being used, which require data feeds to the user for context. Adaptability and adjustability of overlay instrument information includes functional overlays of key operations or parameters of the instrument to clearly represent aspects of the surgical stapler, energy device, or interaction thereof. In one aspect, the overlay data is adjusted by aspects detected by the surgical hub to modify the overlay from information simply detected by the source instrument to add context. In another aspect, the display may be adjusted or modified by the user, which may also result in modification of the operation of the instrument being monitored. The visual overlay can control the interaction of the treatment process or plan, the calculated or processed feedback or predictions, the adjustment of the visual overlay based on the detected parameters, the option to control the position, size and placement to combine with moving objects in the field of view. provides control over the overlay of data onto a surgical site visualization stream containing information associated with an overview visual overlay of instruments within the field of view of an overlay device (i.e., an AR device, AR glasses, etc.).

In one aspect, the visual overlay includes AR glasses or Includes interaction of treatment steps or plans according to an aspect of the present disclosure, including bartering of visual overlays onto the AR device 66, such as other augmented screens or local displays 67. A visual overlay provides highlights for the surgeon. The visual overlay also provides highlights for OR nurses or other staff. Use of an AR device 66 such as AR glasses with a virtual instrument template for OR setup and an AR device camera 79 for scanning and introduction and removal of products. The visual overlay also highlights the products needed for the next surgical step and how the instrument was passed to the nurse for reloading. Based on its understanding of the procedure and situational awareness, the AR device 66 sees the current procedure status with the next logical cartridge for use and identifies and highlights this within the user's field of view.

Additional aspects of interaction of treatment steps or plans are described in the Visualization System Quantitative Data Overlaid With Data From At Least One Instrument Function filed December 30, 2019. On Of A Powered Instrument In Communication With The System” No. 16/729,740, incorporated herein by reference in its entirety. In particular, reference is made to, for example, FIGS. 24, 25A-25B, 30, 32A-32D, 34, 35, and 36A-36C, and the related description. Additional examples include U.S. Pat. U.S. Patent Application No. 20190279524 (A1) entitled “Techniques For Virtualized Tool Interaction” filed July 15, 2019 “Methods And Systems For Using Compute” U.S. Patent entitled ``r-Vision To Enhance Surgical Tool Control During Surgery'' No. 1,075,8309 (B1), entitled "Intraoperative Surgical Event Summary," filed February 21, 2019, U.S. Pat. be incorporated into.

The visual overlay includes computed, processed feedback, or predictive techniques according to one aspect of the present disclosure. In one aspect, the visual overlay includes projecting the path of the surgical instrument 77. For example, an endoscope-assisted overlay on a laparoscopic view and an overlay from a laparoscopic view to an endoscopic view. Another aspect includes projecting the position of a robotic arm.

In one aspect, the computed, processed feedback, or predictive techniques provide a separate visual overlay to the bedside assistant of the manual actions required to assist with the robotic case. This may include providing a separate visual overlay to assist with the need for repositioning the liver hook, the up/down or left/right hand handle micromanipulator position, or the up/down or left/right hand handle stapler.

In one aspect, the calculated, processed feedback, or prediction techniques provide a projected overlay of the surgical stapler. For example, the system may provide a projected cut line overlay, a firing delay countdown timer, a knife force, or articulation angle prediction. The projected cut line overlay may include a path overlay, a cut length overlay, or a staple cartridge length.

In one aspect, the calculated, processed feedback, or prediction techniques provide a projected overlay of the energy device. The projected overlay of the energy device may include impedance calculations, straight or curved jaws, or predicted joint angles. Additional aspects include providing an overlay of the surgical stapler countdown timer and impedance calculations.

Additional examples of calculated and processed feedback, or predictive techniques, are provided in U.S. Patent Application No. 20200237452 (A1), 2020, entitled "Timeline Overlay On Surgical Video," filed February 20, 2020. U.S. Patent Application No. 20200268469 (A1) entitled “Image-Based System For Estimating Surgical Contact Force” filed on February 27, 2020, “A Source And Extent” filed on February 27, 2020 Of fluid U.S. Patent Application No. 20200268472 (A1) entitled “Hub Recommendations From Real Time Analysis Of Procedure Varia” filed on November 6, 2018 bles Against A Baseline Highlighting Differences From The Optimal Solution” U.S. Patent Application No. 20190201102 (A1) (in particular, FIGS. 17-19 and related description), and “System And Method For Analysis And Presentation Of Surgical Procedure Vi,” filed on August 13, 2018. titled “deos” No. 1,087,8966 (B2), each of which is incorporated herein by reference in its entirety.

In one aspect, the visual overlay provides a method for adjusting the visual overlay based on detected parameters. One method for adjusting the visual overlay based on detected parameters includes detecting an aspect of the procedure or instrument to trigger an adaptation of the overlay data. One aspect provides an algorithm for aligning the optical axis of the camera to the orientation of the instrument. The algorithm may include automating the angle change based on the surgical task.

Another method for adjusting the visual overlay based on the detected parameters includes an algorithm for AR depth compensation. In one aspect, the algorithm includes adjusting the overlay image by monitoring the focus of the surgeon 73 to automatically adjust the depth of the augmented information. The algorithm may also include adjusting the focus, depth, or zoom by algorithmically adjusting missing elements. The algorithm may also include using a structured light surface 3D model for augmentation.

Another method for adjusting visual overlays based on detected parameters includes displaying base information when the device is active or connected. For all devices, the display may include the device name, device manufacturer, and device status, such as ready or failed. In a surgical stapler device, the display may include a cartridge installed state or a cartridge fired state. For energy devices, the display may include energy settings such as power level settings, operating modes such as advanced hemostasis, minimum power, maximum power, and the current operating mode being used.

Another method for adjusting the visual overlay based on detected parameters includes an algorithm for handling alerts that appear on portions of the display screen that are out of focus. In one aspect, the algorithm includes adjusting the focus or resolution of the portion of the display where the alert is occurring, even if it is outside of the immediate in-situ portion. For example, in the case of tissue tension problems detected during colon mobilization, macro tissue tension is due to pulling on the colon, causing the detected tension to occur away from laparoscopic in situ interaction visualization. Tension indications may make adjustments to the focus, sharpness, or width of the view, or may indicate in which direction events are occurring outside of the currently viewed field of view.

Another method for adjusting visual overlays based on detected parameters includes algorithms for overlaying unadjusted or unmodified information. The information may include device name and serial number. The overlaid information may be in the form of static or dynamic data.

Another method for adjusting the visual overlay based on detected parameters includes algorithms to improve instrument performance by requiring more focus for a short period of time to ensure a perfect task. For example, while a surgical stapler is cutting thick tissue. Cutting the thick tissue causes the surgical stapler to slow down. The system detects this slowdown and adjusts the overlay to highlight the surgical stapler, specifically the knife position and knife speed. This highlighting is to draw the surgeon's focus to the surgical stapler. The surgeon may determine that a cut pause is the best course of action in the current situation.

Another method for adjusting the visual overlay based on detected parameters includes an algorithm that improves the view when smoke fills the abdominal cavity and obscures the view from the laparoscopic camera. The algorithm may include overlaying an infrared view while smoke clouds the image.

In one aspect, visual overlay provides a method for controlling the visual overlay of data onto a surgical site visualization stream. The method includes options to control position, size, placement, and association with moving objects within the field of view. The visual display also provides adaptability of aspects of the overlay data to enable customization of the overlay.

One method for controlling the visual overlay of data onto the surgical site visualization stream includes simultaneous localization and mapping (SLAM) techniques. SLAM technology provides a framework for building maps of unknown locations and unknown environments and determining actual locations within the maps. SLAM technology positions the sensor with its surroundings (sensor signal processing) and maps the environmental structure (pose graph optimization). The sensor provides a digital map of the unknown environment and optimizes the digital map based on continuous input of data and optimization of the data as the wearer moves around the space.

Visual SLAM acquires images from a camera/image sensor using a coarse-to-fine method. The coarse method matches feature points. The fine method controls the brightness of the image.

Light Detection and Ranging (LiDAR) SLAM uses laser/distance sensors and is more accurate, faster, but less detailed than visual SLAM. The matching point cloud provides iterative nearest neighbors and a normal distribution transformation. For example, Google's driverless cars use LiDAR to obtain information about their local environment and make driving functional decisions based on the mapping of their surroundings (in combination with Google Maps information).

The fused method includes other data sources such as inertial measurement units (IMUs), global positioning systems (GPS), etc. According to the fused method, additional known information and mappings can be overlaid with the created mappings to provide additional information to the wearer.

One method for controlling the visual overlay of data onto the surgical site visualization stream includes controlling the overlay through a dedicated device. A single functional device may be configured to provide dedicated control of the overlay (and only the overlay). As shown in FIG. 36, a touch screen interface may be used to control the display features.

One method for controlling the visual overlay of data onto the surgical site visualization stream includes controlling the overlay through a multifunction device. The multifunction device may be configured to support multiple different applications or functions, allowing additional control of the display.

One method for controlling the visual overlay of data onto the surgical site visualization stream includes controlling certain features within the overlay. One example includes controlling overlay transparency. Another example includes controlling overlay size to allow a clinical user to resize images or elements that may be overlaid on the screen to accommodate user preferences. Another example includes controlling font size to allow a clinical user to resize any text that may be overlaid on the display to accommodate user preferences. Another example includes context control functionality. This method includes using configurable panels that dynamically change based on a selected area of a window on the display. Another example includes controlling alerts and warnings. This method uses buttons to select the alignment and position of where alerts and warnings may be presented, as shown in FIG. 50. Additional features include information and case metrics that may be controlled within the overlay.

One method for controlling the visual overlay of data onto the surgical site visualization stream includes controlling elements of the overlay to provide contextualized help. The overlay includes user-selected buttons and automatic prompts based on user actions. This can be useful when a user tries to select a feature by mistake.

One method for controlling the visual overlay of data onto the surgical site visualization stream includes controlling and interacting with the overlay from surgeon audio commands. The surgeon may invoke the desired visual overlay to be overlaid on the primary display monitor. For example, the surgeon may invoke commands such as "patient vitals overlay," "surgical stapler overlay," or "ultrasound overlay" to have these overlays overlaid on the primary display. In one aspect, the system may use a person tracking aspect to distinguish between different users within the operating room.

In one aspect, the visual overlay provides a method for providing an overview visual overlay of an instrument in a field of an overlay device, such as, for example, an AR device 66. The visual overlay may be configured to provide a user with an overview status of the device, the primary configuration or user, and an identification of the device. The visual overlay may also be configured to provide unwieldy control of the overview data to enable interactive setup or reconfiguration of the device.

FIG. 72 is an image of a staff view screen 5600 displaying customized overlay information. The staff view screen 5600 displays a main surgical display 5602 that allows customization of the surgical overlay and includes three main screen portions. The first screen portion 5604 displays a surgeon preset to allow for input of surgeon presets or default settings that can be saved, modified, or deleted. The first screen portion 5604 also includes a section that allows for turning the display overlay on or off, and an overlay size portion that allows for adjustment of the main surgical display 5602 from small to large in three settings: small, medium, and large.

The second screen portion 5606 of the primary surgical display 5602 is displayed to the right of the first screen portion. The second screen portion 5606 displays case information and overall device usage dates and allows editing of the display of case information. The right chevron 5612 can be tapped to access more granular capabilities to turn individual overlays on/off. The virtual switch slider button 5612 is used to turn groups of overlays on/off.

The third screen portion 5608 of the primary surgical display 5602 is displayed below the second screen portion 5606. The third screen portion 5608 displays energy panels and device alerts, and allows editing of the display of device panels. Like the second screen portion 5606, the third screen portion 5608 includes a right chevron 5616 that can be tapped to access more granular capabilities for turning individual overlays on/off, and a virtual switch slider button 5618 for turning groups of overlays on/off.

The fourth screen portion 5610 of the primary surgical display 5602 is displayed below the third screen portion 5608. The fourth screen portion 5610 displays all system notifications and allows editing of system notifications. Similar to the second and third screen portions 5606, 5608, the fourth screen portion 5610 includes a right chevron 5620 that can be tapped to access more granular capabilities for turning individual overlays on/off, and a virtual switch slider button 5622 for turning groups of overlays on/off.

Tapable icons are provided at the bottom of the primary surgical display 5602 to provide additional functionality. For example, one tappable icon 5624 allows navigation of the staff view screen.

FIG. 73 is an image of a staff view screen 5700 displaying detailed customization pop-up information. The staff view screen 5700 displays an edit case overlay screen 5702 on the main surgical display 5602 screen. The edit case information overlay screen 5702 allows the ability to turn individual overlay components on/off. The edit case information overlay screen 5702 includes a series of virtual switch slider buttons 5704 to allow editing of, for example, the patient's first and last name, the surgeon's first and last name, the case (e.g., surgical procedure) date and time, the case duration, the number of instrument activations, or the staple firings.

FIG. 74 is an image of a staff view screen 5800 displaying staff view troubleshooting pop-up information. The staff view screen 5800 displays a staff view troubleshooting pop-up screen 5802 to provide troubleshooting information regarding system devices and components. Tappable icons 5084 provide device connection status and indicate whether the system is properly connected to other capital equipment or devices. The troubleshooting pop-up screen 5802 shows a check ultrasound connection generator error screen with a still image 5806 of an ultrasound generator 5808. Additional troubleshooting screens with still images, gifs, or animations may be provided. The troubleshooting screen may have multiple steps or multiple screens (not shown) that the user can navigate through.

FIG. 75 is an image of a main surgical display interaction screen 5900 displaying main surgical display interactions. The main surgical display interaction screen 5900 shows an instrument panel behavior screen 5902 with an overlay screen 5904 with three sections. The first section of the overlay screen 5904 shows device activation information including an instrument activation default panel 5906 and an instrument minimum active panel 5908. The information includes, for example, a change in the panel background from dark to light with active states and modes shown in dark text and inactive modes shown with reduced opacity.

The second section of the overlay screen 5904 shows instrument override activation information including an instrument override activation panel 5910, an alarm status panel 5912, and a disable instrument panel 5914. When a particular alarm is triggered, the instrument panel is greyed out to indicate that activation has been disabled. This may only be applicable when the user has been locked out of the device due to an alarm condition.

The third section of the overlay screen 5904 shows minimized information including a generic instrument default panel 5916 and a minimized panel 5918. The panels minimize to a predefined size after a predefined period of time. The instrument type remains on the panel and the panel reverts to default when an activation or notification occurs.

Situational awareness is the ability of some aspects of a surgical system to determine or infer information relevant to a surgical procedure from data received from a database and/or instruments. The information may include the type of procedure being performed, the type of tissue being operated on, or the body cavity being treated. According to the context information regarding the surgical procedure, the surgical system may, for example, control modular devices (e.g., robotic arms and/or robotic surgical tools) connected to it, and may be used by the surgeon during the course of the surgical procedure. can be improved in a manner that provides contextualized information or suggestions.

FIG. 76 illustrates a timeline of a situation-aware surgical procedure. FIG. 72 illustrates an example surgical procedure timeline 5200 and the contextual information that the surgical hub 5104 can derive from data received from the data source 5126 at each step of the surgical procedure. The timeline 5200 illustrates the general steps that nurses, surgeons, and other medical personnel would take during the course of a lung segmentectomy procedure, beginning with the setup of the surgical site and ending with the patient being transported to a post-operative recovery room. The situation-aware surgical hub 5104 receives data from the data source 5126 throughout the course of the surgical procedure, including data generated each time a medical personnel uses a modular device 5102 paired with the surgical hub 5104. The surgical hub 5104 receives this data from the paired modular device 5102 and other data sources 5126 to continually derive inferences (i.e., contextual information) regarding the ongoing procedure as new data is received, such as which step of the procedure is occurring at any given time. The situational awareness system of the surgical hub 5104 can, for example, record data regarding the procedure to generate a report, verify steps being taken by medical personnel, provide data or prompts (e.g., via a display screen) that may be relevant to a particular procedure step, adjust the modular device 5102 based on the context (e.g., activate a monitor, adjust the FOV of a medical imaging device, or change the energy level of an ultrasonic surgical instrument or an RF electrosurgical instrument), and any other such actions described above.

First 5202, hospital personnel retrieves the patient's EMR from the hospital's EMR database. Based on the patient data selected in the EMR, surgical hub 5104 determines that the procedure being performed is thoracic surgery.

Second, 5204, personnel scan the incoming medical supplies for the procedure. The surgical hub 5104 cross-references the scanned supplies with a list of supplies utilized in various types of procedures and verifies that the mix of supplies is compatible with a thoracic procedure. Additionally, the surgical hub 5104 may also determine that the procedure is not a wedge resection (either because the incoming supplies do not include the specific supplies required for a thoracic wedge resection or are not otherwise compatible with a thoracic wedge resection).

In a third 5206, the medical personnel scans the patient band via a scanner 5128 communicatively connected to the surgical hub 5104. The surgical hub 5104 can then verify the patient's identity based on the scanned data.

Fourth, 5208, the medical personnel turns on the auxiliary device. The auxiliary equipment utilized may vary according to the type of surgical procedure and the technique used by the surgeon, but in this illustrative case includes a smoke evacuator, a ventilator, and a medical imaging device. Once activated, the ancillary device that is the modular device 5102 can automatically pair with a surgical hub 5104 located within a particular proximity of the modular device 5102 as part of its initialization process. . Surgical hub 5104 can then derive contextual information regarding the surgical procedure by detecting the type of modular device 5102 with which it is paired during this preoperative or initialization phase. In this particular example, surgical hub 5104 determines that the surgical procedure is a VATS procedure based on this particular combination of paired modular devices 5102. Based on a combination of data from the patient's EMR, the list of medical supplies used in the procedure, and the type of modular device 5102 that connects to the hub, the surgical hub 5104 generally estimates the specific procedure that the surgical team will perform. can do. Once the surgical hub 5104 knows what particular procedure is being performed, the surgical hub 5104 then reads the steps for that procedure from memory or from the cloud and then reads the steps of that procedure from the connected data source 5126 (e.g. Data subsequently received from the modular device 5102 and the patient monitoring device 5124) can be cross-referenced to estimate which steps of the surgical procedure the surgical team is performing.

In the fifth 5210, personnel attach EKG electrodes and other patient monitoring devices 5124 to the patient. EKG electrodes and other patient monitoring devices 5124 can be paired with surgical hub 5104. When surgical hub 5104 begins receiving data from patient monitoring device 5124, surgical hub 5104 verifies that the patient is at the surgical site.

Sixth 5212, the medical personnel induces anesthesia in the patient. Surgical hub 5104 determines whether the patient is under anesthesia based on data from modular device 5102 and/or patient monitoring device 5124, including, for example, EKG data, blood pressure data, ventilator data, or a combination thereof. You can guess that. Upon completion of the sixth step 5212, the preoperative portion of the lung segmentectomy surgery is complete and the surgical department begins.

Seventh, 5214, the patient's lung being operated on is collapsed (while ventilation is switched to the contralateral lung). The surgical hub 5104 can infer from the ventilator data that the patient's lung has been collapsed. Because the surgical hub 5104 can compare the detection of the patient's lung being collapsed to the expected steps of the procedure (which may be accessed or read in advance), it can infer that the surgical portion of the procedure has begun, thereby determining that collapsing the lung is the first surgical step in this particular procedure.

In the eighth 5216, the medical imaging device 5108 (e.g., a scope) is inserted and video footage from the medical imaging device is initiated. The surgical hub 5104 receives the medical imaging device data (i.e., still image data or real-time live streaming video) through a connection to the medical imaging device. Upon receiving the medical imaging device data, the surgical hub 5104 can determine that the laparoscopic portion of the surgical procedure has begun. Additionally, the surgical hub 5104 can determine that the particular procedure being performed is a segmentectomy as opposed to a lobectomy (note that a wedge resection has not already been taken into account by the surgical hub 5104 based on the data received in the second step 5204 of the procedure). Data from the medical imaging device 124 (FIG. 2) can be used to determine contextual information regarding the type of procedure being performed in a variety of ways, such as by determining the angle at which the medical imaging device is oriented relative to visualization of the patient's anatomy, by monitoring the number or medical imaging devices being utilized (i.e., activated and paired with the surgical hub 5104), and by monitoring the type of visualization equipment being utilized.

For example, one technique for performing a VATS lobectomy places the camera above the diaphragm in the anteroinferior corner of the patient's thoracic cavity, whereas one technique for performing a VATS segmentectomy involves The camera is placed in an intercostal position anterior to the segmental fissure. The situational awareness system can be trained to recognize the location of the medical imaging device according to the visualization of the patient's anatomy, using, for example, pattern recognition techniques or machine learning techniques. As another example, one technique for performing a VATS lobectomy utilizes a single medical imaging device, whereas another technique for performing a VATS segmentectomy utilizes multiple cameras. . As yet another example, one technique for performing a VATS segmentectomy utilizes an infrared light source (which can be communicatively coupled to a surgical hub as part of the visualization system) to visualize the segmental cleft. However, this is not utilized in VATS lobectomy. By tracking any or all of this data from the medical imaging device 5108, the surgical hub 5104 determines the specific type of surgical procedure being performed and/or used for the specific type of surgical procedure. It is possible to determine the technology being used.

In a ninth step 5218, the surgical team begins the incision step of the procedure. Surgical hub 5104 receives data from an RF or ultrasound generator indicating that an energy instrument is being fired, so it can be inferred that the surgeon is in the process of dissecting and separating the patient's lungs. . The surgical hub 5104 cross-references the received data with the read steps of the surgical procedure to determine whether the energy instrument being fired at this point in the process (i.e., after the steps of the procedure described above are completed) It can be determined that the process corresponds to the incision process.

In the tenth step 5220, the surgical team proceeds to the ligation step of the procedure. Surgical hub 5104 receives data from the surgical stapling and cutting instrument indicating that the instrument is being fired, so it can assume that the surgeon is ligating the artery and vein. Similar to the previous step, the surgical hub 5104 can derive this estimate by cross-referencing the receipt of data from the surgical stapling and cutting instruments with the steps in the process that have been read.

Eleventh 5222, the segmentectomy portion of the procedure is performed. Surgical hub 5104 estimates that the surgeon is transecting parenchymal tissue based on data from the surgical instruments, including data from the staple cartridge. Cartridge data may correspond, for example, to the size or type of staple being fired by the instrument. Cartridge data may indicate the type of tissue being stapled and/or transected for different types of staples utilized on different types of tissue. The type of staple being fired may be applied to parenchymal tissue or other similar tissue type, and the surgical hub 5104 may assume that a segmental ablation procedure is being performed.

Subsequently, in a twelfth step 5224, a nodule dissection step is performed. The surgical hub 5104 may infer that the surgical team is dissecting the nodule and performing a leak test based on data received from the generator indicating that the RF or ultrasound instrument is being fired. . For this particular procedure, the RF or ultrasound instruments utilized after the parenchymal tissue is transected are compatible with the nodule dissection step, which allows the surgical hub 5104 to make this estimate. Because different instruments are better suited to specific tasks, surgeons may choose between surgical stapling/cutting instruments and surgical energy (i.e., RF or ultrasound) instruments depending on the particular step during the procedure. Note that the Thus, the particular sequence in which the stapling/cutting instruments and surgical energy instruments are used can indicate which step of the procedure the surgeon is performing. Upon completion of the twelfth step 5224, the incision is closed and the post-operative portion of the procedure begins.

In a thirteenth step 5226, the patient is deanesthetized. The surgical hub 5104 can estimate that the patient is emerging from anesthesia, for example, based on ventilator data (i.e., the patient's breathing rate begins to increase).

Finally, in a fourteenth 5228, the medical personnel remove the various patient monitoring devices 5124 from the patient. Thus, the surgical hub 5104 can presume that the patient is being transferred to a recovery room when the hub loses EKG, BP, and other data from the patient monitoring devices 5124. According to the data received from the various data sources 5126 communicatively coupled to the surgical hub 5104, the surgical hub 5104 can determine or presume when each step of a given surgical procedure is occurring.

As shown in a first step 5202 of timeline 5200 shown in FIG. 76, patient data from the EMR database(s) is utilized to estimate the type of surgical procedure to be performed. Additionally, patient data can also be utilized by the situational awareness surgical hub 5104 to generate control adjustments for the paired modular device 5102.

Various additional aspects of the subject matter described herein are illustrated in the numbered examples below.

Example 1: An augmented reality display system for use during a surgical procedure, the augmented reality display system comprising: an imaging device for capturing a real image of a surgical field during a surgical procedure; an augmented reality display for presenting functional data overlays associated with actively visualized critical operations of a surgical instrument and interactions of the surgical instrument with tissue in the surgical field, the functional data being overlaid on the real image of the surgical field, the functional data overlay being a combination of critical operations of the surgical instrument and aspects of the interaction of the surgical instrument with tissue in the surgical field; and a processor for receiving functional data from the surgical instrument, determining overlay data associated with the functional aspects of the surgical instrument, and combining aspects of the tissue in the surgical field with the functional data received from the surgical instrument.

Example 2: The augmented reality system of Example 1, further comprising a surgical hub coupled to the augmented reality display, wherein the overlay data is adjusted according to aspects detected by the surgical hub to modify the overlay data from information detected by the surgical instrument to add context.

Example 3: An augmented reality system according to any one of Examples 1 to 2, in which the augmented reality display is modifiable by a user.

Example 4: The augmented reality system of Example 3, wherein the monitored surgical instrument function is modified based on user modifications of the augmented reality display.

Example 5: An augmented reality display system for use during a surgical procedure, the augmented reality display system comprising: an imaging device for capturing a real image of a surgical field during a surgical procedure; an augmented reality display for presenting a functional data overlay associated with parameters of an actively visualized surgical instrument and the interaction of the surgical instrument with tissue in the surgical field, the functional data being overlaid on the real image of the surgical field, the functional data overlay being a combination of the parameters of the surgical instrument and aspects of the interaction of the surgical instrument with tissue in the surgical field; and a processor for receiving functional data from the surgical instrument, determining overlay data associated with the functional aspects of the surgical instrument, and combining aspects of the tissue in the surgical field with the functional data received from the surgical instrument.

Example 6: The augmented reality system of Example 5, further comprising a surgical hub coupled to the augmented reality display, wherein the overlay data is adjusted according to aspects detected by the surgical hub to modify the overlay data from information detected by the surgical instrument to add context.

Example 7: An augmented reality system according to any one of Examples 5 to 6, in which the augmented reality display is modifiable by a user.

Example 8: The augmented reality system of Example 7, in which monitored surgical instrument function is modified based on user modification of the augmented reality display.

Example 9: An augmented reality display system for use during a surgical procedure, the augmented reality display system comprising an imaging device and an imaging device for capturing real images of a surgical area during the surgical procedure. an augmented reality display for presenting functional data overlays associated with a surgical instrument; and an energy generator coupled to the surgical instrument, wherein the surgical instrument emits radio frequency (RF) energy during a surgical procedure. an energy generator using energy and ultrasound energy; and a surgical hub coupled to the energy generator and an augmented reality display, the surgical hub having an augmented reality display for displaying live video of a surgical area. a surgical hub, the augmented reality display system comprising: a surgical hub, the augmented reality display providing live images of the surgical area, the surgical instruments, important operations or parameters of the surgical instruments, and the surgical area; The system displays views with panel overlays to display information specific to the interaction of surgical instruments with tissue within the system.

Example 10: The system of Example 9, in which the augmented reality display shows an end effector of a surgical instrument gripping tissue and a panel overlay displaying case information, system notifications, or device panels, or any combination thereof, overlaid on a live video feed of the surgical field.

Example 11: The system of Example 10, wherein the position, opacity, size, and placement of the panel overlay is customizable.

Example 12: The system of example 11, wherein the panel overlays are configured to be turned on or off individually or as a group.

Example 13: Panel overlays can include capital equipment, generators, ventilators, smoke extractors, electronic health records, laparoscopes, computers, surgical devices, wired and wireless connectivity devices, surgeon profiles that can be saved, recalled, or edited 13. The system as in any one of examples 10-12, comprising at least one of data input information from preferences, or any combination thereof.

Example 14: The system of any one of Examples 10-13, wherein the panel overlay includes case information including at least one of patient name, surgeon name, case time, or instrument activation, or a combination thereof.

Example 15: The system of any one of Examples 10 to 14, wherein the panel overlay includes a system notification including at least one of connected device status, minor error alert, medium error alert, or major error alert, or any combination thereof.

Example 16: The system of any one of Examples 10-15, wherein the panel overlay includes information associated with a surgical instrument connected to the system to provide enhanced hemostasis.

Example 17: A system described in any one of Examples 10 to 16, wherein the panel overlay includes a visible patient panel overlay.

Example 18: The system of any one of Examples 10 to 17, wherein the panel overlay includes a device panel overlay including at least one of a device name, device settings, or device supplemental features, or any combination thereof.

Example 19. The system of any one of Examples 10 to 18, wherein the panel overlay includes a plurality of panel overlays in a stacked configuration.

Example 20: The system as in any one of Examples 10-19, wherein the panel overlay comprises a plurality of panel overlays in an expanded configuration.

Example 21: The system of Example 20, wherein the panel overlay is configured to dynamically change to indicate a change in status, such as device activation or power level adjustment.

Example 22: The system of Example 10, wherein the panel overlay shows an optimal device performance (ODP) guide image or other instruction for use (IFU)/information source.

Example 23: The system described in Example 9, wherein the intraoperative data display includes a secondary configurable panel.

Example 24: As described in Example 23, wherein the secondary configurable panel dynamically changes based on a selected customized laparoscopic overlay field displayed in the surgical field of the live surgical video area of the intraoperative data display. system.

Example 25: As described in Example 24, wherein the customized laparoscopic overlay field includes at least one of a bottom edge panel, a top left corner panel, a top center panel, or a side edge panel, or any combination thereof. system.

Example 26: The system as in any one of Examples 10-25, wherein the panel overlay displays device troubleshooting information.

Example 27: The system of any one of Examples 10 to 26, wherein the panel overlay displays at least one of an alert, a warning, device information, or device features, or any combination thereof.

While several forms have been shown and described, it is not the intention of the applicant to limit or limit the scope of the appended claims to such details. Many modifications, variations, changes, permutations, combinations and equivalents of these forms may be implemented and will occur to those skilled in the art without departing from the scope of the disclosure. Furthermore, the structure of each element associated with the described form may alternatively be described as a means for providing the functionality performed by that element. Also, although materials are disclosed for particular components, other materials may be used. It is therefore intended that the foregoing description and appended claims cover all such modifications, combinations, and variations as falling within the scope of the disclosed forms. I want you to understand. The appended claims are intended to cover all such modifications, variations, changes, substitutions, modifications, and equivalents.

The above detailed description has described various aspects of the apparatus and/or processes using block diagrams, flow diagrams and/or examples. To the extent that such block diagrams, flow diagrams and/or examples include one or more functions and/or operations, it should be understood by one of ordinary skill in the art that each function and/or operation included in such block diagrams, flow diagrams and/or examples can be implemented individually and/or collectively by a variety of hardware, software, firmware, or virtually any combination thereof. It will be understood by one of ordinary skill in the art that some aspects of the embodiments disclosed herein, in whole or in part, can be equivalently implemented on an integrated circuit as one or more computer programs running on one or more computers (e.g., as one or more programs running on one or more computer systems), as one or more programs running on one or more processors (e.g., as one or more programs running on one or more microprocessors), as firmware, or as virtually any combination thereof, and that designing circuitry and/or writing software and/or firmware code is within the skill of one of ordinary skill in the art in view of the present disclosure. In addition, those skilled in the art will appreciate that the subject matter described herein may be distributed as one or more program products in a variety of forms, and that the specific form of the subject matter described herein applies regardless of the particular type of signal-bearing medium used to actually effect the distribution.

The instructions used to program the logic to implement the various disclosed aspects may be stored in a system memory such as dynamic random access memory (DRAM), cache, flash memory, or other storage. Additionally, the instructions may be distributed over a network or by other computer-readable media. Thus, a machine-readable medium may include any mechanism for storing or transmitting information in a form readable by a machine (e.g., a computer), but is not limited to floppy diskettes, optical disks, compact disks, read-only memories (CD-ROMs), as well as magneto-optical disks, read-only memories (ROMs), random access memories (RAMs), erasable programmable read-only memories (EPROMs), electrically erasable programmable read-only memories (EEPROMs), magnetic or optical cards, flash memories, or tangible machine-readable storage used to transmit information over the Internet via electrical, optical, acoustic, or other forms of propagated signals (e.g., carrier waves, infrared signals, digital signals, etc.). Thus, non-transitory computer-readable media includes any type of tangible machine-readable medium suitable for storing or transmitting electronic instructions or information in a form readable by a machine (e.g., a computer).

As used in any aspect of this specification, the term "control circuitry" may refer to, for example, hardwired circuitry, programmable circuitry (e.g., a computer processor, processing unit, processor, microcontroller, microcontroller unit, controller, digital signal processor (DSP), programmable logic device (PLD), programmable logic array (PLA), or field programmable gate array (FPGA) that includes one or more individual instruction processing cores), state machine circuitry, firmware that stores instructions executed by the programmable circuitry, and any combination thereof. The control circuitry may be embodied, collectively or individually, as circuits that form part of a larger system, such as, for example, an integrated circuit (IC), an application specific integrated circuit (ASIC), a system on a chip (SoC), a desktop computer, a laptop computer, a tablet computer, a server, a smartphone, etc. Thus, as used herein, "control circuitry" includes, but is not limited to, electrical circuits having at least one discrete electrical circuit, electrical circuits having at least one integrated circuit, electrical circuits having at least one application specific integrated circuit, electrical circuits forming a general purpose computing device configured by a computer program (e.g., a general purpose computer configured by a computer program to at least partially execute the processes and/or devices described herein, or a microprocessor configured by a computer program to at least partially execute the processes and/or devices described herein), electrical circuits forming a memory device (e.g., a form of random access memory) and/or electrical circuits forming a communication device (e.g., a modem, a communication switch, or an optical-electrical facility). Those skilled in the art will recognize that the subject matter described herein may be implemented in analog or digital form, or some combination thereof.

As used in any aspect of this specification, the term "logic" may refer to an application, software, firmware, and/or circuitry configured to perform any of the operations described above. Software may be embodied as a software package, code, instructions, instruction sets, and/or data recorded on a non-transitory computer-readable storage medium. Firmware may be embodied as code, instructions, or instruction sets, and/or hard-coded (e.g., non-volatile) data in a memory device.

When used in any aspect of this specification, the terms "component," "system," "module," etc. may refer to either a control circuit, a computer-related entity, hardware, a combination of hardware and software, software, or software in execution.

As used in any aspect herein, "algorithm" refers to a self-consistent sequence of steps leading to a desired result; Refers to the manipulation of physical quantities and/or logical states, which can take the form of electrical or magnetic signals, which can be compared, compared, and otherwise manipulated. It is common practice to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, or the like. These and similar terms may be associated with the appropriate physical quantities or are simply convenient labels applied to these quantities and/or conditions.

The network may include a packet switched network. The communication devices can communicate with each other using selected packet-switched network communication protocols. One example communication protocol may include the Ethernet communication protocol, which may enable communication using Transmission Control Protocol/Internet Protocol (TCP/IP). The Ethernet protocol conforms to or is compatible with the Ethernet standard published by the Institute of Electrical and Electronics Engineers (IEEE) entitled "IEEE 802.3 Standard" published December 2008, and/or any later version of this standard. It can be sexual. Alternatively or additionally, the communication device is an X. can communicate with each other using the X.25 communication protocol. X. The T.25 communication protocols may conform to or be compatible with standards promulgated by the International Telecommunication Union-Telecommunication Standardization Sector (ITU-T). Alternatively or additionally, communication devices may communicate with each other using a frame relay communication protocol. The Frame Relay communication protocol is a protocol developed by the Consultative Committee for International Telegraph and Telephone (CCITT) and/or the American National Standards Institute (AN SI) or may be compatible with the standards promulgated by SI. Alternatively or additionally, the transceivers may be capable of communicating with each other using an asynchronous transfer mode (ATM) communication protocol. The ATM communications protocol shall comply with or be compatible with the ATM standard published by the ATM Forum in August 2001 under the title "ATM-MPLS Network Interworking 2.0" and/or with later versions of this standard. obtain. Of course, different and/or later developed connection-oriented network communication protocols are equally contemplated herein.

Unless explicitly stated otherwise, as is clear from the foregoing disclosure, the terms "processing," "computing," and "calculating" are used throughout the foregoing disclosure. , ``determining,'' ``displaying,'' and similar terms refer to the use of terms such as actions and processing of a computer system or similar electronic computing device that manipulate and convert such information into other data similarly represented as physical quantities in memories or registers or other storage, transmission, or display devices. I hope you understand what I'm referring to.

One or more components may be referred to herein as being "configured to," "configurable to," "operable/operative to," "adapted/adaptable," "able to," "conformable/conformed to," and the like. Those skilled in the art will understand that "configured to" may generally encompass active and/or inactive and/or standby components, unless the context requires otherwise.

The terms "proximal" and "distal" are used herein with reference to the clinician manipulating the handle portion of the surgical instrument. The term "proximal" refers to the part closest to the clinician, and the term "distal" refers to the part located away from the clinician. It will be further understood that for convenience and clarity, spatial terms such as "vertical," "horizontal," "above," and "below" may be used herein with respect to the drawings. However, surgical instruments may be used in many orientations and positions, and these terms are not intended to be limiting and/or absolute.

Those skilled in the art will generally understand that the terms used herein, and particularly in the appended claims (e.g., the text of the appended claims), are generally intended as "open" terms. (For example, the term "including" should be interpreted as "including but not limited to," and the term "having" should be interpreted as "including but not limited to.") should be interpreted as “having at least” and the term “includes” should be interpreted as “includes but is not limited to.” should be done, etc.). Further, if a specific number is intended in an introduced claim recitation, such intent will be clearly stated in the claim, and if no such statement is present, no such intent exists. will be understood by those skilled in the art. For example, as an aid to understanding, the following appended claims incorporate the introductory phrases "at least one" and "one or more" into claim statements. may be included in order to However, the use of such phrases does not apply when the claim is introduced by the indefinite article "a" or "an," even if the introductory phrase "one or more" or "at least one" appears within the same claim. and the indefinite article "a" or "an," any particular claim containing such an introduced claim statement is limited to a claim containing only one such statement. (e.g., "a" and/or "an" should generally be interpreted to mean "at least one" or "one or more") ). The same applies when introducing claim statements using definite articles.

In addition, those skilled in the art will recognize that even when a specific number is specified in an introduced claim description, such a description should typically be interpreted to mean at least the number recited (e.g., a description of "two items" without other qualifiers generally means at least two items, or more than two items). Furthermore, when a notation similar to "at least one of A, B, and C, etc." is used, such syntax is generally intended in the sense that a person skilled in the art would understand the notation (e.g., "a system having at least one of A, B, and C" includes, but is not limited to, systems having only A, only B, only C, both A and B, both A and C, both B and C, and/or all of A, B, and C, etc.). When a notation similar to "at least one of A, B, or C, etc." is used, such syntax is generally intended in the sense that one of ordinary skill in the art would understand the notation (e.g., "a system having at least one of A, B, or C" includes, but is not limited to, systems having only A, only B, only C, both A and B, both A and C, both B and C, and/or all of A, B, and C, etc.). Furthermore, one of ordinary skill in the art will understand that any disjunctive word and/or phrase expressing two or more alternative terms should typically be understood to contemplate the possibility of including one of those terms, either of those terms, or both of those terms, unless the context requires otherwise, whether in the specification, claims, or drawings. For example, the phrase "A or B" will typically be understood to include the possibility of "A" or "B" or "A and B."

With reference to the appended claims, those skilled in the art will appreciate that the acts recited herein may generally be performed in any order. Additionally, while flow diagrams of various operations are shown in sequence(s), it is understood that the various operations may be performed in an order other than that shown, or may be performed simultaneously. It should be. Examples of such alternative orderings include overlapping, interleaving, interleaving, reordering, incremental, preliminary, additional, simultaneous, reverse or other different orderings, unless the context requires otherwise. May include. Additionally, terms such as "responsive to," "related to," or other past tense adjectives generally do not mean otherwise, unless the context requires otherwise. It is not intended that such variations be excluded.

It is worth noting that any reference to "one embodiment," "an embodiment," "an example," "an example," or the like means that the particular feature, structure, or characteristic described in connection with that embodiment is included in at least one embodiment. Thus, the appearances of the phrases "in one embodiment," "in an embodiment," "in an example," and "in one example" in various places throughout this specification do not necessarily all refer to the same embodiment. Furthermore, the particular features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

Any patent application, patent, non-patent publication, or other disclosure material referenced herein and/or listed in any application data sheet is incorporated by reference herein to the extent that the incorporated material is not inconsistent with this specification. , incorporated herein by reference. As such, and to the extent necessary, disclosure expressly set forth herein shall supersede any conflicting description incorporated herein by reference. Any material, or portion thereof, that is cited as being incorporated herein by reference but is inconsistent with current definitions, views, or other disclosures set forth herein shall be deemed to be incorporated by reference. They are incorporated only to the extent that they are consistent with existing disclosures.

In summary, many benefits have been described that result from using the concepts described herein. The above description of one or more forms has been presented for purposes of illustration and description. It is not intended to be exhaustive or limited to the precise form disclosed. Modifications or variations are possible in light of the above teachings. The one or more embodiments are intended to be illustrative of the principles and practical application, thereby enabling those skilled in the art to utilize the various embodiments, with various modifications, as appropriate for the particular application contemplated. It has been selected and written in order to It is intended that the claims presented herewith define the overall scope.

[Embodiment]
(1) An augmented reality display system for use during a surgical procedure, the augmented reality display system comprising:
an imaging device for capturing actual images of the surgical field during the surgical procedure;
an augmented reality display for presenting functional data overlays associated with actively visualized critical actions of a surgical instrument and interactions of the surgical instrument with tissue within the surgical field, the functional data being overlaid on the real image of the surgical field, the functional data overlay being a combination of the critical actions of the surgical instrument and aspects of the interaction of the surgical instrument with the tissue within the surgical field;
1. A processor comprising:
receiving functional data from the surgical instrument;
determining the overlay data relating to the functional aspect of the surgical instrument;
a processor that combines the appearance of the tissue in the surgical field with the functional data received from the surgical instrument.
2. The augmented reality display system of claim 1, further comprising a surgical hub coupled to the augmented reality display, wherein the overlay data is adjusted according to aspects detected by the surgical hub to modify the overlay data from information detected by the surgical instrument to add context.
3. The augmented reality display system of claim 1, wherein the augmented reality display is user modifiable.
(4) The augmented reality display system of claim 3, wherein the surgical instrument function being monitored is modified based on a user modification of the augmented reality display.
(5) An augmented reality display system for use during a surgical procedure, the augmented reality display system comprising:
an imaging device for capturing actual images of the surgical field during the surgical procedure;
an augmented reality display for presenting functional data overlays associated with actively visualized parameters of a surgical instrument and its interaction with tissue within the surgical field, the functional data being overlaid on the real image of the surgical field, the functional data overlay being a combination of the parameters of the surgical instrument and aspects of the interaction of the surgical instrument with the tissue within the surgical field;
1. A processor comprising:
receiving functional data from the surgical instrument;
determining the overlay data relating to the functional aspect of the surgical instrument;
a processor that combines the appearance of the tissue in the surgical field with the functional data received from the surgical instrument.

6. The augmented reality display system of claim 5, further comprising a surgical hub coupled to the augmented reality display, wherein the overlay data is adjusted according to aspects detected by the surgical hub to modify the overlay data from information detected by the surgical instrument to add context.
7. The augmented reality display system of claim 5, wherein the augmented reality display is user modifiable.
(8) The augmented reality display system of claim 7, wherein the surgical instrument function being monitored is modified based on a user modification of the augmented reality display.
(9) A system comprising:
1. An augmented reality display system for use during a surgical procedure, the augmented reality display system comprising:
an imaging device for capturing actual images of the surgical field during the surgical procedure;
an augmented reality display for presenting a functional data overlay associated with the surgical instrument;
an energy generator coupled to the surgical instrument, the surgical instrument using radio frequency (RF) energy and ultrasonic energy during a surgical procedure;
a surgical hub coupled to the energy generator and the augmented reality display, the surgical hub providing the live view of the surgical area to the augmented reality display for displaying the live view of the surgical area;
an augmented reality display system,
The system, wherein the augmented reality display displays a view of the surgical field, the surgical instrument, and a panel overlay for displaying critical operations or parameters of the surgical instrument and information specific to the surgical instrument's interaction with tissue within the surgical field.
10. The system of claim 9, wherein the augmented reality display shows an end effector of the surgical instrument grasping tissue and a panel overlay displaying case information, system notifications, or device panels, or any combination thereof, overlaid on the live image of the surgical field.

11. The system of embodiment 10, wherein the position, opacity, size, and placement of the panel overlay is customizable.
12. The system of embodiment 11, wherein the panel overlay is configured to be turned on or off individually or as a group.
(13) The panel overlay may include capital equipment, generators, ventilators, smoke extractors, electronic health records, laparoscopes, computers, surgical devices, wired and wireless connectivity devices, surgeon profiles that may be saved, recalled, or edited. 11. The system of embodiment 10, comprising at least one of data input information from preferences, or any combination thereof.
14. The system of embodiment 10, wherein the panel overlay includes case information including at least one of patient name, surgeon name, case time, or instrument activation, or a combination thereof.
(15) The panel overlay includes a system notification including at least one of a connected appliance status, a minor error alert, a moderate error alert, or a major error alert, or any combination thereof. system.

16. The system of claim 10, wherein the panel overlay includes information associated with the surgical instrument connected to the system to provide advanced hemostasis.
17. The system of claim 10, wherein the panel overlay includes a visible patient panel overlay.
18. The system of claim 10, wherein the panel overlay comprises a device panel overlay including at least one of a device name, device settings, or device supplemental features, or any combination thereof.
19. The system of claim 10, wherein the panel overlay comprises a plurality of panel overlays in a stacked configuration.
20. The system of claim 10, wherein the panel overlay comprises a plurality of panel overlays in an expanded configuration.

21. The system of embodiment 20, wherein the panel overlay is configured to dynamically change to indicate a change in status, such as device activation or power level adjustment.
22. The system of embodiment 10, wherein the panel overlay shows an Optimal Device Performance (ODP) guide image or other Instructions for Use (IFU)/information source.
23. The system of embodiment 9, wherein the intraoperative data display includes a secondary configurable panel.
(24) The secondary configurable panel dynamically changes based on a selected customized laparoscopic overlay field displayed in a surgical field of view of a live surgical video region of the intraoperative data display. The system described.
(25) The customized laparoscopic overlay field includes at least one of a bottom edge panel, a top left corner panel, a top center panel, or a side edge panel, or any combination thereof. system.

26. The system of claim 10, wherein the panel overlay displays device troubleshooting information.
27. The system of claim 10, wherein the panel overlay displays at least one of an alert, a warning, device information, or device features, or any combination thereof.

Claims (27)

1. An augmented reality display system for use during a surgical procedure, the augmented reality display system comprising:
an imaging device for capturing actual images of the surgical field during the surgical procedure;
an augmented reality display for presenting functional data overlays associated with actively visualized critical actions of a surgical instrument and interactions of the surgical instrument with tissue within the surgical field, the functional data being overlaid on the real image of the surgical field, the functional data overlay being a combination of the critical actions of the surgical instrument and aspects of the interaction of the surgical instrument with the tissue within the surgical field;
1. A processor comprising:
receiving functional data from the surgical instrument;
determining the overlay data relating to the functional aspect of the surgical instrument;
a processor that combines the appearance of the tissue in the surgical field with the functional data received from the surgical instrument. further comprising a surgical hub coupled to the augmented reality display, the overlay data modifying the overlay data from information detected by the surgical instrument to add context; 2. The augmented reality display system of claim 1, wherein the augmented reality display system is adjusted by detected aspects. The augmented reality display system of claim 1, wherein the augmented reality display is user modifiable. The augmented reality display system of claim 3, wherein the surgical instrument function being monitored is modified based on user modifications of the augmented reality display. An augmented reality display system for use during a surgical procedure, the augmented reality display system comprising:
an imaging device for capturing real images of the surgical area during the surgical procedure;
an augmented reality display for presenting a functional data overlay associated with actively visualized parameters of a surgical instrument and interaction of the surgical instrument with tissue within the surgical field, the functional data comprising: , overlaid on the actual image of the surgical region, the functional data overlay being a combination of the parameters of the surgical instrument and the aspects of the interaction of the surgical instrument with the tissue within the surgical region. , an augmented reality display, and
A processor,
receiving functional data from the surgical instrument;
determining the overlay data related to the functional aspect of the surgical instrument;
an augmented reality display system comprising: a processor that combines the aspects of the tissue within the surgical field with the functional data received from the surgical instrument. further comprising a surgical hub coupled to the augmented reality display, the overlay data modifying the overlay data from information detected by the surgical instrument to add context; 6. The augmented reality display system of claim 5, wherein the augmented reality display system is adjusted by detected aspects. 6. The augmented reality display system of claim 5, wherein the augmented reality display is user modifiable. The augmented reality display system of claim 7, wherein the surgical instrument function being monitored is modified based on user modifications of the augmented reality display. A system,
An augmented reality display system for use during a surgical procedure, the augmented reality display system comprising:
an imaging device for capturing real images of the surgical area during the surgical procedure;
an augmented reality display for presenting functional data overlays associated with the surgical instrument;
an energy generator coupled to the surgical instrument, wherein the surgical instrument uses radio frequency (RF) energy and ultrasound energy during a surgical procedure;
a surgical hub coupled to the energy generator and the augmented reality display, the surgical hub configured to display the live image of the surgical area on the augmented reality display; a surgical hub that provides
Equipped with an augmented reality display system, including
the augmented reality display displaying information specific to the surgical region, the surgical instrument, important operations or parameters of the surgical instrument, and interactions of the surgical instrument with tissue within the surgical region; The system displays views with panel overlays. The augmented reality display displays an end effector of the surgical instrument grasping tissue and case information, system notifications, or device panels, or any combination thereof, overlaid on the live video of the surgical field. 10. The system of claim 9, wherein the system comprises: a panel overlay; The system of claim 10, wherein the position, opacity, size, and placement of the panel overlay are customizable. 12. The system of claim 11, wherein the panel overlay is configured to be turned on or off individually or as a group. The system of claim 10, wherein the panel overlay includes at least one of data entry information from capital equipment, generators, ventilators, smoke extractors, electronic health records, laparoscopes, computers, surgical devices, wired and wireless connectivity devices, surgeon profile preferences that may be saved, recalled, or edited, or any combination thereof. The system of claim 10, wherein the panel overlay includes case information including at least one of patient name, surgeon name, case time, or instrument activation, or a combination thereof. The system of claim 10, wherein the panel overlay includes system notifications including at least one of connected device status, minor error alerts, medium error alerts, or major error alerts, or any combination thereof. 11. The system of claim 10, wherein the panel overlay includes information associated with the surgical instrument connected to the system to provide enhanced hemostasis. The system of claim 10, wherein the panel overlay includes a visible patient panel overlay. The system of claim 10, wherein the panel overlay comprises a device panel overlay including at least one of a device name, device settings, or device supplemental features, or any combination thereof. 11. The system of claim 10, wherein the panel overlay includes a plurality of panel overlays in a stacked configuration. 11. The system of claim 10, wherein the panel overlay includes a plurality of panel overlays in an expanded configuration. 21. The system of claim 20, wherein the panel overlay is configured to dynamically change to indicate a state change, such as device activation or power level adjustment. The system of claim 10, wherein the panel overlay shows an Optimal Device Performance (ODP) guide image or other Instructions for Use (IFU)/information source. The system of claim 9, wherein the intraoperative data display includes a secondary configurable panel. 24. The system of claim 23, wherein the secondary configurable panel dynamically changes based on a selected customized laparoscopic overlay field displayed in a surgical field of view of a live surgical video region of the intraoperative data display. . 25. The system of claim 24, wherein the customized laparoscopic overlay field includes at least one of a bottom edge panel, a top left corner panel, a top center panel, or a side edge panel, or any combination thereof. The system of claim 10, wherein the panel overlay displays device troubleshooting information. 11. The system of claim 10, wherein the panel overlay displays at least one of alerts, warnings, device information, or device features, or any combination thereof.

Images