CN117652000A - Anticipation of interactive utilization of public data overlays by different users - Google Patents

Abstract

Disclosed herein are devices, systems, and methods for displaying an interactive overlay for a plurality of users. In some aspects, a system for displaying an interactive overlay for a plurality of users may include a surgical hub configured to receive a plurality of data streams related to a surgical procedure and a first augmented reality display device communicatively coupled to the surgical hub. In one aspect, the first augmented reality display device is linked to a first user. In another aspect, the first augmented reality display device displays a first interactive overlay customized for the first user based on at least one of the plurality of data streams.

Description

Anticipation of interactive utilization of public data overlays by different users

Cross Reference to Related Applications

The present application claims the benefit of U.S. 4, 14, 2021, U.S. provisional patent application No. 63/174,674 entitled "head UP DISPLAY" and U.S. provisional patent application No. 63/284,326 entitled "INTRAOPERATIVE DISPLAY FOR SURGICAL SYSTEMS", 2021, 11, 30, each of which is incorporated herein by reference in its entirety.

Background

The present disclosure relates to devices, systems, and methods for providing an augmented reality interactive experience during a surgical procedure. During a surgical procedure, it is desirable to provide an augmented reality interactive experience of a real-world environment in which objects residing in the real world are enhanced by superimposing computer-generated sensory information (sometimes across multiple sensory modalities, including visual, auditory, haptic, somatosensory, and olfactory). In the context of the present disclosure, the surgical field and the images of surgical instruments and other objects present in the surgical field are enhanced by superimposing computer-generated visual, auditory, tactile, somatosensory, olfactory, or other sensory information over the surgical field and the real world images of instruments or other objects present in the surgical field. The image may be streamed in real-time, or may be a still image.

Real world surgical instruments include a variety of surgical devices including energy, staplers, or a combination of energy and staplers. Energy-based medical devices include, but are not limited to, radio Frequency (RF) based monopolar and bipolar electrosurgical instruments, ultrasonic surgical instruments, combined RF electrosurgical and ultrasonic instruments, combined RF electrosurgical and mechanical staplers, and the like. Surgical stapler devices are surgical instruments used to cut and staple tissue in a variety of surgical procedures including weight loss, breast, colorectal, gynecological, urological, and general surgery.

Disclosure of Invention

In various aspects, the present disclosure provides a method for displaying an interactive overlay for a plurality of users of a surgical system. In some aspects, the method comprises: receiving, by a surgical hub, a plurality of data streams related to a surgical procedure; communicatively coupling a first augmented reality display device to a surgical hub; linking, by the surgical hub, the first augmented reality display device to the first user; and displaying, by the first augmented reality display device, a first interactive overlay customized for the first user based on at least one of the plurality of data streams.

In various aspects, the present disclosure provides a surgical system for displaying interactive overlays for a plurality of users. In some aspects, the system comprises: a surgical hub configured to receive a plurality of data streams related to a surgical procedure; and a first augmented reality display device communicatively coupled to the surgical hub. In one aspect, a first augmented reality display device is linked to a first user. In another aspect, the first augmented reality display device displays a first interactive overlay customized for the first user based on at least one of the plurality of data streams.

Drawings

The various aspects (both as to the surgical organization and method) described herein, together with further objects and advantages thereof, may best be understood by reference to the following description taken in connection with the accompanying drawings.

Fig. 1 is a block diagram of a computer-implemented interactive surgical system according to one aspect of the present disclosure.

Fig. 2 is a surgical system for performing a surgical procedure in an operating room according to one aspect of the present disclosure.

Fig. 3 is a surgical hub paired with a visualization system, robotic system, and intelligent instrument, according to one aspect of the present disclosure.

Fig. 4 illustrates a surgical data network including a modular communication hub configured to enable connection of modular devices located in one or more operating rooms of a medical facility or any room specially equipped for surgical procedures in the medical facility to a cloud, according to one aspect of the present disclosure.

Fig. 5 illustrates a computer-implemented interactive surgical system in accordance with an aspect of the present disclosure.

Fig. 6 illustrates a surgical hub including a plurality of modules coupled to a modular control tower according to one aspect of the present disclosure.

Fig. 7 illustrates an Augmented Reality (AR) system including an intermediate signal combiner positioned in a communication path between an imaging module and a surgical hub display, according to one aspect of the present disclosure.

Fig. 8 illustrates an Augmented Reality (AR) system including an intermediate signal combiner positioned in a communication path between an imaging module and a surgical hub display, according to one aspect of the present disclosure.

Fig. 9 illustrates an Augmented Reality (AR) device worn by a surgeon to transmit data to a surgical hub, according to one aspect of the present disclosure.

Fig. 10 illustrates a system for augmenting surgical instrument information using an augmented reality display according to one aspect of the present disclosure.

Fig. 11 illustrates a timeline of a situational awareness surgical procedure in accordance with an aspect of the present disclosure.

Fig. 12 illustrates a surgical system configured to display interactive overlays for multiple users based on multiple data streams in accordance with one aspect of the disclosure.

Fig. 13 illustrates a method of displaying an interactive overlay for multiple users of a surgical system in accordance with an aspect of the disclosure.

FIG. 14 illustrates a method for detecting device-related errors and determining actions to be performed based on the detected errors, according to one aspect of the disclosure.

15A, 15B, 15C, 15D, 15E, and 15F illustrate an exemplary implementation of the method of FIG. 14 during thyroidectomy according to an aspect of the present disclosure.

Fig. 16 illustrates a method for ensuring secure wireless pairing of a smart device with a surgical system in accordance with an aspect of the disclosure.

Fig. 17 illustrates a method for ensuring data authenticity and/or integrity after initial device pairing in accordance with an aspect of the disclosure.

Corresponding reference characters indicate corresponding parts throughout the several views. The exemplifications set out herein illustrate various disclosed embodiments, in one form, and such exemplifications are not to be construed as limiting the scope of the invention in any manner.

Detailed Description

The applicant of the present application owns the following U.S. patent applications filed concurrently herewith, the disclosure of each of these patent applications being incorporated herein by reference in its entirety:

U.S. patent application entitled "METHOD FOR INTRAOPERATIVE DISPLAY FORSURGICAL SYSTEMS", attorney docket END9352USNP 1/210120-1M;

U.S. patent application entitled "UTILIZATION OF SURGICAL DATA VALUES ANDSITUATIONAL AWARENESS TO CONTROL THE OVERLAYIN SURGICAL FIELDVIEW", attorney docket END9352USNP 2/210120-2;

U.S. patent application entitled "SELECTIVE AND ADJUSTABLE MIXED REALITYOVERLAY IN SURGICAL FIELDVIEW", attorney docket END9352USNP 3/210120-3;

U.S. patent application entitled "RISK BASED PRIORITIZATION OF DISPLAYASPECTS IN SURGICAL FIELDVIEW", attorney docket END9352USNP 4/210120-4;

U.S. patent application entitled "SYSTEMS AND METHODS FOR CONTROLLINGSURGICAL DATA OVERLAY", attorney docket END9352USNP 5/210120-5;

U.S. patent application entitled "SYSTEMS AND METHODS FOR CHANGINGDISPLAY OVERLAY OF SURGICAL FIELDVIEW BASED ONTRIGGERING EVENTS", attorney docket END9352USNP 6/210120-6;

U.S. patent application entitled "CUSTOMIZATION OF OVERLAID DATA ANDCONFIGURATION", attorney docket END9352USNP 7/210120-7;

U.S. patent application entitled "INDICATION OF THE COUPLE PAIR OF REMOTECONTROLS WITH REMOTE DEVICES FUNCTIONS", attorney docket END9352USNP 8/210120-8;

U.S. patent application Ser. No. COOPERATIVE OVERLAYS OF INTERACTINGINSTRUMENTS WHICH RESULT IN BOTH OVERLAYSBEING EFFECTED, attorney docket END9352USNP 9/210120-9;

U.S. patent application Ser. No. MIXING DIRECTLY VISUALIZED WITH RENDEREDELEMENTS TO DISPLAY BLENDED ELEMENTS ANDACTIONS HAPPENING ON-SCREEN AND OFF-SCREEN, attorney docket No. END9352USNP 11/210120-11;

U.S. patent application Ser. No. SYSTEM AND METHOD FOR TRACKING APORTION OF THE USER AS A PROXY FORNON-MONITORED INSTRUMENT, attorney docket END9352USNP 12/210120-12;

U.S. patent application entitled "UTILIZING CONTEXTUAL PARAMETERS OF ONEOR MORE SURGICAL DEVICES TO PREDICT AFREQUENCY INTERVAL FOR DISPLAYING SURGICALINFORMATION", attorney docket END9352USNP 13/210120-13;

U.S. patent application entitled "COOPERATION AMONG MULTIPLE DISPLAYSYSTEMS TO PROVIDE A HEALTHCARE USERCUSTOMIZED INFORMATION", attorney docket END9352USNP 14/210120-14;

U.S. patent application entitled "INTRAOPERATIVE DISPLAY FOR SURGICALSYSTEMS", attorney docket END9352USNP 15/210120-15;

U.S. patent application entitled "ADAPTATION AND ADJUSTABILITY OR OVERLAIDINSTRUMENT INFORMATION FOR SURGICAL SYSTEMS", attorney docket END9352USNP 16/210120-16; and

U.S. patent application Ser. No. MIXED REALITY FEEDBACK SYSTEMS THAT COOPERATE TO INCREASE EFFICIENT PERCEPTION OF COMPLEX DATA FEEDS, attorney docket END9352USNP 17/210120-17.

The applicant of the present application owns the following U.S. patent applications, the disclosure of each of which is incorporated herein by reference in its entirety:

U.S. patent application Ser. No. 16/209,423, entitled "METHOD OF COMPRESSING TISSUE WITHIN A STAPLING DEVICE AND SIMULTANNEOUSLY DISPLAYING THE LOCATION OF THE TISSUE WITHIN THE JAWS", now U.S. patent application publication No. US-2019-0200981-A1;

U.S. patent application Ser. No. 16/209,453, entitled "METHOD FOR CONTROLLING SMART ENERGY DEVICES," now U.S. patent application publication No. US-2019-0201046-A1.

Before explaining aspects of the surgical device and generator in detail, it should be noted that the exemplary embodiment is not limited in its application or use to the details of construction and arrangement of parts illustrated in the accompanying drawings and description. The illustrative examples may be implemented alone or in combination with other aspects, variations and modifications and may be practiced or carried out in various ways. Furthermore, unless otherwise indicated, the terms and expressions employed herein have been chosen for the purpose of describing the illustrative examples for the convenience of the reader and are not for the purpose of limitation. Moreover, it is to be understood that the expression of one or more of the aspects, and/or examples described below may be combined with any one or more of the expression of other aspects, and/or examples described below.

Various aspects relate to a screen display for a surgical system for various energy and surgical stapler-based medical devices. Energy-based medical devices include, but are not limited to, radio Frequency (RF) based monopolar and bipolar electrosurgical instruments, ultrasonic surgical instruments, combined RF electrosurgical and ultrasonic instruments, combined RF electrosurgical and mechanical staplers, and the like. The surgical stapler device includes a combination surgical stapler having an electrosurgical device and/or an ultrasonic device. Aspects of the ultrasonic surgical device may be configured to transect and/or coagulate tissue, for example, during a surgical procedure. Aspects of the electrosurgical device may be configured for transecting, coagulating, sealing, welding, and/or desiccating tissue, for example, during a surgical procedure. Aspects of the surgical stapler device can be configured to transect and staple tissue during a surgical procedure, and in some aspects, the surgical stapler device can be configured to deliver RF energy to tissue during a surgical procedure. The electrosurgical device is configured to deliver therapeutic and/or non-therapeutic RF energy to tissue. The elements of the surgical stapler device, electrosurgical device, and ultrasonic device may be used in combination in a single surgical instrument.

In various aspects, the present disclosure provides an on-screen display of real-time information to an OR team during a surgical procedure. In accordance with various aspects of the present disclosure, a number of new and unique screen displays are provided to display various visual information feedback to an OR team on a screen. In accordance with the present disclosure, visual information may include one or more various visual media, whether audible or silent. Generally, visual information includes still photography, movie photography, video or audio recordings, graphic arts, visual aids, models, displays, visual presentation services, and supporting processes. Visual information may be conveyed on any number of display options, such as, for example, a main OR screen, the energy OR surgical stapler device itself, a tablet computer, augmented reality glasses, and the like.

In various aspects, the present disclosure provides a large number of possible lists of options to communicate visual information to an OR team in real-time without providing excessive visual information to the OR team. For example, in various aspects, the present disclosure provides a screen display of visual information to enable a surgeon OR other member of an OR team to selectively activate the screen display, such as an icon surrounding a screen option, to manage a large amount of visual information. The active display may be determined using one or a combination of factors, which may include an energy-based (e.g., electrosurgical, ultrasound) or mechanical-based (e.g., stapler) surgical device in use, estimating the risk associated with a given display, the degree of experience of the surgeon, the surgeon's choice, and so forth. In other aspects, the visual information may include a large amount of data superimposed or overlaid into the surgical field to manage the visual information. In various aspects described below, overlapping images that require video analysis and tracking are included in order to properly overlay data. In contrast to static icons, visual information data transmitted in this manner may provide additional useful visual information to the OR team in a more concise and easily understood manner.

In various aspects, the present disclosure provides techniques for selectively activating a screen display, such as an icon surrounding the screen, to manage visual information during a surgical procedure. In other aspects, the present disclosure provides techniques for determining an active display using one or a combination of various factors. In various aspects, techniques according to the present disclosure may include selecting an energy-based OR mechanical-based surgical device for use as an active display, estimating risk associated with a given display, utilizing the experience level of a surgeon OR OR team making the selection, and so forth.

In other aspects, techniques according to the present disclosure may include overlaying or overlaying a large amount of data onto a surgical field of view to manage visual information. The various display arrangements described in this disclosure relate to superimposing various visual representations of surgical data on a live stream of a surgical field of view. As used herein, the term overlay includes semi-transparent overlays, partial overlays, and/or moving overlays. The graphic overlay may be in the form of a transparent graphic, a translucent graphic, or an opaque graphic, or a combination of transparent, translucent, and opaque elements or effects. Further, the superimposed layers may be positioned on or at least partially on or near objects in the surgical field such as, for example, end effectors and/or critical surgical structures. Some display arrangements may include changes in one or more display elements of the superimposed layers, including changes in color, size, shape, display time, display location, display frequency, highlighting, or combinations thereof, based on changes in display priority values. A graphical overlay is rendered on top of the active display monitor to quickly and efficiently communicate important information to the OR team.

In other aspects, techniques according to the present disclosure may include overlapping images that require analysis of video and tracking in order to properly overlay visual information data. In other aspects, techniques according to the present disclosure may include transmitting rich visual information instead of simple static icons, thereby providing additional visual information to the OR team in a more concise and easily understood manner. In other aspects, the visual overlay may be used in combination with an audible and/or somatosensory overlay (such as thermal, chemical and mechanical devices, and combinations thereof).

The following description relates generally to devices, systems, and methods that provide an Augmented Reality (AR) interactive experience during a surgical procedure. In this context, the surgical field and the images of surgical instruments and other objects present in the surgical field are enhanced by superimposing computer-generated visual, auditory, tactile, somatosensory, olfactory, or other sensory information on the surgical field, the real world images of instruments and other objects present in the surgical field. The image may be streamed in real-time, or may be a still image. Augmented reality is a technology for rendering and displaying virtual or "augmented" virtual objects, data, or visual effects superimposed on a real environment. The real environment may include a surgical field of view. A virtual object superimposed on a real environment may be represented as anchored or in a set position relative to one or more aspects of the real environment. In a non-limiting example, if a real world object leaves the real environment field of view, then a virtual object anchored to the real world object will also leave the augmented reality field of view.

The various display arrangements described in this disclosure relate to superimposing various visual representations of surgical data on a live stream of a surgical field of view. As used herein, the term overlay includes semi-transparent overlays, partial overlays, and/or moving overlays. Further, the superimposed layers may be positioned on or at least partially on or near objects in the surgical field such as, for example, end effectors and/or critical surgical structures. Some display arrangements may include changes in one or more display elements of the superimposed layers, including changes in color, size, shape, display time, display location, display frequency, highlighting, or combinations thereof, based on changes in display priority values.

As described herein, AR is an enhanced version of the real physical world, achieved through the use of digital visual elements, sounds, or other sensory stimuli delivered via technology. Virtual Reality (VR) is a computer-generated environment with scenes and objects that appear to be real, so that users feel themselves immersed in their surroundings. The environment is perceived by a device called a virtual reality headset or helmet. Both Mixed Reality (MR) and AR are considered immersive techniques, but they are not identical. MR is an extension of mixed reality, allowing real and virtual elements to interact in an environment. While AR often adds digital elements to a real-time view through the use of cameras, MR experiences combine elements of both AR and VR, in which real world and digital objects interact.

In an AR environment, one or more computer-generated virtual objects may be displayed with one or more real (i.e., so-called "real world") elements. For example, real-time images or videos of the surrounding environment may be displayed on a computer screen display along with one or more overlaid virtual objects. Such virtual objects may provide supplemental information about the environment or generally enhance the user's perception and participation in the environment. Instead, real-time images or videos of the surrounding environment may additionally or alternatively enhance user engagement with virtual objects shown on the display.

Apparatus, systems, and methods in the context of the present disclosure enhance images received from one or more imaging devices during a surgical procedure. The imaging device may include various endoscopes, AR devices, and/or cameras used during non-invasive and minimally invasive surgical procedures to provide images during open surgical procedures. The image may be streamed in real-time, or may be a still image. The devices, systems, and methods enhance images of a real-world surgical environment by overlaying virtual objects or representations of data and/or real objects on the real-world surgical environment, thereby providing an augmented reality interactive experience. The augmented reality experience may be viewed on a display and/or AR device that allows a user to view the overlaid virtual object on the real world surgical environment. The display may be located in the operating room or remote from the operating room. AR devices are worn on the head of a surgeon or other operating room personnel and typically include two stereoscopic display lenses or screens, one for each eye of the user. Natural light can pass through two transparent or translucent display lenses so that aspects of the real environment are visible, while also projecting light so that the virtual object is visible to the user of the AR device.

Two or more displays and AR devices may be used in a coordinated manner, such as a first display or AR device controlling one or more additional displays or AR devices in a defined character control system. For example, when the display or AR device is activated, the user may select a role (e.g., surgeon, surgical assistant, nurse, etc. during a surgical procedure) and the display or AR device may display information related to the role. For example, the surgical assistant may have a virtual representation of the displayed instrument that the surgeon needs when performing the next step of the surgical procedure. The surgeon's attention to the current step may see different display information than the surgical assistant.

While there are many known screen displays and warnings, the present disclosure provides many new and unique augmented reality interactive experiences during a surgical procedure. Such augmented reality interactive experiences include visual, auditory, tactile, somatosensory, olfactory, or other sensory feedback information to a surgical team inside or outside the operating room. Virtual feedback information superimposed on the real world surgical procedure environment may be provided to an Operating Room (OR) team, including personnel internal to the OR, including, but not limited to, for example, a knife surgeon, a surgeon assistant, a swabbing personnel, an anesthesiologist, and a round nurse, among others. The virtual feedback information may be transmitted over any number of display options, such as a master OR screen display, AR device, energy OR surgical stapler instrument, tablet computer, augmented reality glasses, device, and the like.

Fig. 1 shows a computer-implemented interactive surgical system 1 comprising one or more surgical systems 2 and a cloud-based system 4. The cloud-based system 4 may include a remote server 13 coupled to the storage 5. Each surgical system 2 includes at least one surgical hub 6 in communication with the cloud 4. For example, the surgical system 2 may include a visualization system 8, a robotic system 10, and a hand-held intelligent surgical instrument 12, each configured to communicate with each other and/or with the hub 6. In some aspects, the surgical system 2 may include M hubs 6, N visualization systems 8, O robotic systems 10, and P smart handheld surgical instruments 12, where M, N, O and P are integers greater than or equal to 1. The computer-implemented interactive surgical system 1 may be configured to provide an augmented reality interactive experience during a surgical procedure as described herein.

Fig. 2 shows an example of a surgical system 2 for performing a surgical procedure on a patient lying on an operating table 14 in a surgical operating room 16. The robotic system 10 is used as part of the surgical system 2 in a surgical procedure. The robotic system 10 includes a surgeon's console 18, a patient side cart 20 (surgical robot), and a surgical robotic hub 22. When the surgeon views the surgical site through the surgeon's console 18 or an Augmented Reality (AR) device 66 worn by the surgeon, the patient-side cart 20 may manipulate at least one removably coupled surgical tool 17 through a minimally invasive incision in the patient. An image of the surgical site of the minimally invasive surgical procedure (e.g., a still image or a live image streamed in real time) may be obtained by the medical imaging device 24. The patient side cart 20 may maneuver the imaging device 24 to orient the imaging device 24. An image of the open surgical procedure may be obtained by the medical imaging device 96. The robotic hub 22 processes the image of the surgical site for subsequent display on the surgeon's console 18 or on an AR device 66 worn by the surgeon or to other personnel in the surgical room 16.

The optical components of imaging device 24, 96 or AR device 66 may include one or more illumination sources and/or one or more lenses. One or more illumination sources may be directed to illuminate multiple portions of the surgical field. The one or more image sensors may receive light reflected or refracted from tissue and instruments in the surgical field.

In various aspects, the imaging device 24 is configured for use in minimally invasive surgical procedures. Examples of imaging devices suitable for use in the present disclosure include, but are not limited to, arthroscopes, angioscopes, bronchoscopes, choledochoscopes, colonoscopes, cytoscopes, duodenoscopes, enteroscopes, esophageal-duodenal scopes (gastroscopes), endoscopes, laryngoscopes, nasopharyngeal-renal endoscopes, sigmoidoscopes, thoracoscopes, and hysteroscopes. In various aspects, the imaging device 96 is configured for use in an open (invasive) surgical procedure.

In various aspects, the visualization system 8 includes one or more imaging sensors, one or more image processing units, one or more storage arrays, and one or more displays strategically placed relative to the sterile field. In one aspect, the visualization system 8 includes interfaces for HL7, PACS, and EMR. In one aspect, the imaging device 24 may employ multispectral monitoring to distinguish between topography and underlying structures. Multispectral images capture image data over a specific range of wavelengths across the electromagnetic spectrum. Wavelengths, including light from frequencies outside the visible range, such as IR and ultraviolet, are separated by filters or devices sensitive to the particular wavelengths. Spectral imaging can extract information that is not visible to the human eye. Multispectral monitoring can reposition the surgical field after completion of the surgical task to perform a test on the treated tissue.

Fig. 2 shows the main display 19 positioned in the sterile field to be visible to an operator at the operating table 14. The visualization tower 11 is positioned outside the sterile zone and comprises a first non-sterile display 7 and a second non-sterile display 9 facing away from each other. The visualization system 8 guided by the hub 6 is configured to be able to coordinate the information flow to operators inside and outside the sterile field using the displays 7, 9, 19. For example, hub 6 may cause visualization system 8 to display AR images of the surgical site recorded by imaging devices 24, 96 on non-sterile displays 7, 9 or by AR device 66, while maintaining a real-time feed of the surgical site on primary display 19 or AR device 66. For example, the non-sterile displays 7, 9 may allow a non-sterile operator to perform diagnostic steps related to a surgical procedure.

Fig. 3 shows the surgical hub 6 in communication with the visualization system 8, the robotic system 10, and the hand-held intelligent surgical instrument 12. Hub 6 includes a hub display 35, an imaging module 38, a generator module 40, a communication module 30, a processor module 32, a memory array 34, and an operating room mapping module 33. The hub 6 further comprises a smoke evacuation module 26 and/or a suction/flushing module 28. In various aspects, the imaging module 38 includes an AR device 66 and the processor module 32 includes an integrated video processor and augmented reality modeler (e.g., as shown in fig. 10). The modular light source may be adapted for use with a variety of imaging devices. In various examples, multiple imaging devices may be placed at different locations in the surgical field to provide multiple views (e.g., non-invasive, minimally invasive, or open surgical procedures). The imaging module 38 may be configured to be switchable between imaging devices to provide an optimal view. In various aspects, the imaging module 38 may be configured to integrate images from different imaging devices and provide an augmented reality interactive experience during a surgical procedure as described herein.

Fig. 4 shows a surgical data network 51 including a modular communication hub 53 configured to enable connection of modular devices located in one or more operating rooms/rooms of a medical facility to a cloud-based system. Cloud 54 may include a remote server 63 (fig. 5) coupled to storage 55. Modular communication hub 53 includes a network hub 57 and/or a network switch 59 in communication with a network router 61. Modular communication hub 53 is coupled to local computer system 60 to process data. The modular devices 1a-1n located in the operating room may be coupled to a modular communication hub 53. The network hub 57 and/or the network switch 59 may be coupled to a network router 61 to connect the devices 1a-1n to the cloud 54 or the local computer system 60. The data associated with the devices 1a-1n may be transmitted via routers to cloud-based computers for remote data processing and manipulation. The operating room devices 1a-1n may be connected to the modular communication hub 53 by a wired channel or a wireless channel. The surgical data network 51 environment can be used to provide an augmented reality interactive experience during a surgical procedure as described herein, and in particular to provide an augmented image in a surgical field of view to one or more remote displays 58.

Fig. 5 illustrates a computer-implemented interactive surgical system 50. The computer-implemented interactive surgical system 50 is similar in many respects to the computer-implemented interactive surgical system 1. The computer-implemented interactive surgical system 50 includes one or more surgical systems 52 that are similar in many respects to the surgical system 2. Each surgical system 52 includes at least one surgical hub 56 in communication with a cloud 54, which may include a remote server 63. In one aspect, the computer-implemented interactive surgical system 50 includes a modular control tower 23 that is connected to a plurality of operating room devices, such as intelligent surgical instruments, robots, and other computerized devices located in an operating room. As shown in fig. 6, modular control tower 23 includes a modular communication hub 53 coupled to a computer system 60.

Returning to fig. 5, modular control tower 23 is coupled to imaging module 38 (which is coupled to endoscope 98), generator module 27 (which is coupled to energy device 99), smoke extractor module 76, suction/irrigation module 78, communication module 13, processor module 15, storage array 16, smart device/appliance 21 (which is optionally coupled to display 39), and sensor module 29. The operating room devices are coupled to cloud computing resources, such as servers 63, data storage 55, and display 58, via modular control tower 23. The robotic hub 72 may also be connected to the modular control tower 23 and to the server 63, the data storage 55, and the display 58. The device/instrument 21, visualization system 58, etc. may be coupled to the modular control tower 23 via a wired or wireless communication standard or protocol, as described herein. The modular control tower 23 may be coupled to a hub display 65 (e.g., monitor, screen) to display the received enhanced images, including overlaid virtual objects in the real surgical field received from the imaging module 38, the device/instrument display 39, and/or other visualization system 58. Hub display 65 may also display data received from devices connected to modular control tower 23 in combination with the image and the overlay image.

Fig. 6 shows a surgical hub 56 that includes a plurality of modules coupled to the modular control tower 23. The modular control tower 23 includes a modular communication hub 53 (e.g., a network connectivity device) and a computer system 60 to provide, for example, enhanced local processing, visualization, and imaging of surgical information. Modular communication hub 53 may be hierarchically configured to connect to extend the number of modules (e.g., devices) that may be connected to modular communication hub 53 and to transmit data associated with the modules to computer system 60, cloud computing resources, or both. Each of the hubs 57/switches 59 in the modular communications hub 53 may include three downstream ports and one upstream port. The upstream hub 57/switch 59 is connected to the processor 31 to provide a communication connection with cloud computing resources and a local display 67. Communication with cloud 54 may be through a wired or wireless communication channel.

The computer system 60 includes a processor 31 and a network interface 37. The processor 31 is coupled to a communication module 41, a storage device 45, a memory 46, a non-volatile memory 47 and an input/output interface 48 via a system bus. The system bus may be any of several types of bus structures including a memory bus or memory controller, a peripheral bus or external bus, and/or a local bus using any of a variety of available bus architectures.

The processor 31 includes an augmented reality modeler (e.g., as shown in fig. 10) and may be implemented as a single or multi-core processor, such as those provided by texas instruments (Texas Instruments) under the trade name ARM Cortex. In one aspect, the processor may be an on-chip memory from, for example, texas instruments (Texas Instruments) LM4F230H5QR ARM Cortex-M4F processor core including 256KB of single-cycle flash memory or other non-volatile memory (up to 40 MHz), a prefetch buffer for improving execution above 40MHz, 32KB single-cycle Sequential Random Access Memory (SRAM), loaded withInternal read-only memory (ROM) of software, 2KB electrically erasable programmable read-only memory (EEPROM), and/or one or more Pulse Width Modulation (PWM) modules, one or more Quadrature Encoder Inputs (QEI) analog, one or more 12-bit analog-to-digital converters (ADC) with 12 analog input channels, the details of which can be seen in the product data sheet.

The system memory includes volatile memory and nonvolatile memory. A basic input/output system (BIOS), containing the basic routines to transfer information between elements within the computer system, such as during start-up, is stored in nonvolatile memory. For example, the non-volatile memory may include ROM, programmable ROM (PROM), electrically Programmable ROM (EPROM), EEPROM, or flash memory. Volatile memory includes Random Access Memory (RAM), which acts as external cache memory. Further, the RAM may be available in various forms such as SRAM, dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM) Enhanced SDRAM (ESDRAM), synchronous Link DRAM (SLDRAM), and Direct Rambus RAM (DRRAM).

Computer system 60 also includes removable/non-removable, volatile/nonvolatile computer storage media such as, for example, magnetic disk storage. Disk storage includes, but is not limited to, devices such as magnetic disk drives, floppy disk drives, tape drives, jaz drives, zip drives, LS-60 drives, flash memory cards, or memory sticks. In addition, the disk storage can include storage media separately or in combination with other storage media including, but not limited to, an optical disk drive such as a compact disk ROM device (CD-ROM), compact disk recordable drive (CD-R drive), compact disk rewritable drive (CD-RW drive) or a digital versatile disk ROM drive (DVD-ROM). To facilitate connection of the disk storage devices to the system bus, a removable or non-removable interface may be used.

In various aspects, the computer system 60 of fig. 6, the imaging module 38 and/or the visualization system 58 and/or the processor module 15 of fig. 4-6 may include an image processor, an image processing engine, a Graphics Processing Unit (GPU), a media processor, or any special-purpose Digital Signal Processor (DSP) for processing digital images. The image processor may employ parallel computation with single instruction, multiple data (SIMD) or multiple instruction, multiple data (MIMD) techniques to increase speed and efficiency. The digital image processing engine may perform a series of tasks. The image processor may be a system on a chip having a multi-core processor architecture.

Fig. 7 shows an augmented reality system 263 that includes an intermediate signal combiner 64 positioned in the communication path between the imaging module 38 and the surgical hub display 67. The signal combiner 64 combines audio and/or image data received from the imaging module 38 and/or the AR device 66. The surgical hub 56 receives the combined data from the combiner 64 and superimposes the data provided to the display 67 on which the superimposed data is displayed. Imaging device 68 may be a digital video camera and audio device 69 may be a microphone. The signal combiner 64 may include a wireless heads-up display adapter to couple to an AR device 66 placed in the communication path of the display 67 to the console, allowing the surgical hub 56 to superimpose data on the display 67.

Fig. 8 illustrates an Augmented Reality (AR) system including an intermediate signal combiner positioned in a communication path between an imaging module and a surgical hub display. Fig. 8 shows AR device 66 worn by surgeon 73 to transmit data to surgical hub 56. Peripheral information of AR device 66 does not include active video. Instead, the peripheral information includes only the device settings or signals that do not have the same refresh rate requirements. The interaction may augment the surgeon's 73 information based on linking with preoperative Computed Tomography (CT) or other data linked in the surgical hub 56. The AR device 66 may identify structures, such as querying whether the instrument is contacting a nerve, vessel, or adhesion. The AR device 66 may include preoperative scan data, optical views, tissue interrogation features obtained throughout the procedure, and/or processing in the surgical hub 56 for providing answers. The surgeon 73 may dictate notes to the AR device 66 for storage in the hub storage device 45 with the patient data for later reporting or follow-up.

The AR device 66 worn by the surgeon 73 is linked to the surgical hub 56 with audio and visual information to avoid the need for superposition and to allow the display information to be customized around the perimeter of the field of view. The AR device 66 provides a signal from the device (e.g., instrument), answers queries about device settings or location information linked to the video to identify quadrants or locations. The AR device 66 has audio control and audio feedback from the AR device 66. The AR device 66 is able to interact with all other systems in the operating room and has feedback and interaction available wherever the surgeon 73 looks. For example, the AR device 66 may receive voice or gesture initiated commands and inquiries from the surgeon, and the AR device 66 may provide feedback in the form of one or more modalities including audio, visual, or tactile touches.

Fig. 9 shows a surgeon 73 wearing AR device 66, a patient 74, and a camera 96 may be included in operating room 75. The AR device 66 worn by the surgeon 73 may be used to present virtual objects superimposed on a real-time image of the surgical field to the surgeon 73 via an augmented reality display 89 or via a hub-connected display 67. The real-time image may include a portion of the surgical instrument 77. The virtual object may not be visible to other people (e.g., surgical assistants or nurses) within the operating room 75, although they may also be wearing the AR device 66. Even if another person is viewing operating room 75 using AR device 66, that person may not see the virtual object or be able to see the virtual object in the augmented reality shared with surgeon 73, or be able to see a modified version of the virtual object (e.g., according to a unique customization to surgeon 73) or be able to see a different virtual object.

The virtual objects and/or data may be configured to appear on a portion of the surgical instrument 77 or in a surgical field captured by the imaging module 38, by the imaging device 68 during a minimally invasive surgical procedure, and/or by the camera 96 during an open surgical procedure. In the illustrated example, the imaging module 38 is a laparoscopic camera that provides real-time feeding of the surgical field during a minimally invasive surgical procedure. The AR system may present a virtual object that is fixed to a real object regardless of the perspective of one or more observers (e.g., surgeon 73) of the AR system. For example, the virtual object may be visible to an observer of the AR system inside the operating room 75, and not visible to an observer of the AR system outside the operating room 75. When an observer enters operating room 75, the virtual object may be displayed to the observer outside operating room 75. The augmented image may be displayed on the surgical hub display 67 or the augmented reality display 89.

The AR device 66 may include one or more screens or lenses, such as a single screen or two screens (e.g., one screen for each eye of the user). The screen may allow light to pass through the screen such that aspects of the real environment are visible when the virtual object is displayed. The virtual object may be made visible to the surgeon 73 by projected light. The virtual object may appear to have some degree of transparency or may be opaque (i.e., occlude aspects of the real environment).

The AR system may be viewable to one or more observers and may include differences between views available to one or more observers while maintaining some aspects common between views. For example, the heads-up display may change between two views, while virtual objects and/or data may be fixed to real objects or regions in the two views. Aspects, such as color, illumination, or other changes of the object, may be changed between views without changing the fixed location of the at least one virtual object.

The user may consider virtual objects and/or data presented in the AR system as opaque or as including a degree of transparency. In one example, a user may interact with a virtual object, such as by moving the virtual object from a first location to a second location. For example, the user may move the object with his own hand. This may be accomplished virtually in an AR system by determining that the hand has moved to a position coincident with or adjacent to the object (e.g., using one or more cameras that may be mounted on the AR device 66, such as AR device camera 79 or a separate camera 96, and which may be static or controllable to move) and causing the object to move in response. The virtual aspect may comprise a virtual representation of the real world object or may comprise a visual effect, such as a lighting effect or the like. The AR system may include rules that govern the behavior of the virtual object, such as subjecting the virtual object to gravity or friction, or may include other predefined rules that exclude real world physical constraints (e.g., floating objects, perpetuation, etc.). The AR device 66 may include a camera 79 (not confused with camera 96, separate from the AR device 66) located on the AR device 66. The AR device camera 79 or camera 96 may include an infrared camera, an infrared filter, a visible light filter, multiple cameras, a depth camera, and the like. The AR device 66 may project the virtual item on a representation of the real environment so as to be viewable by the user.

The AR device 66 may be used in an operating room 75 during a surgical procedure, for example, performed on a patient 74 by a surgeon 73. The AR device 66 may project or display virtual objects such as virtual objects during a surgical procedure to enhance the vision of the surgeon. The surgeon 73 may view the virtual object using the AR device 66, a remote control for the AR device 66, or may interact with the virtual object, such as using a hand to "interact" with the virtual object or a gesture recognized by a camera 79 of the AR device 66. The virtual object may augment a surgical tool, such as surgical instrument 77. For example, the virtual object may appear (to the surgeon 73 viewing the virtual object through the AR device 66) to be coupled to or maintained a fixed distance from the surgical instrument 77. In another example, the virtual object may be used to guide a surgical instrument 77 and may appear to be fixed to the patient 74. In some examples, the virtual object may react to movement of other virtual objects or real world objects in the surgical field of view. For example, the virtual object may be changed while the surgeon is manipulating a surgical instrument that is proximate to the virtual object.

The augmented reality display system imaging device 38 captures a real image of the surgical field during the surgical procedure. The augmented reality displays 89, 67 present a superposition of the operational aspects of the surgical instrument 77 over the real image of the surgical field. Surgical instrument 77 includes communication circuitry 231 to communicate operational aspects and functional data from surgical instrument 77 to AR device 66 via communication circuitry 233 on AR device 66. Although surgical instrument 77 and AR device 66 are shown as RF wireless communication between circuits 231, 233 as indicated by arrow B, C, other communication techniques (e.g., wired, ultrasonic, infrared, etc.) may be employed. The overlay is related to the operational aspects of the active visualization of the surgical instrument 77. The overlay combines aspects of tissue interaction in the surgical field with functional data from the surgical instrument 77. The processor portion of AR device 66 is configured to receive operational aspects and functional data from surgical instrument 77, determine overlays relating to the operation of surgical instrument 77, and combine tissue aspects in the surgical field with functional data from surgical instrument 77. The enhanced image indicates warnings regarding device performance considerations, warnings of incompatible usage, warnings regarding incomplete capture. Incompatible uses include tissue out of range conditions and tissue not properly balanced within the jaws of the end effector. The additional enhanced image provides an indication of an incident, including an indication of tissue tension and an indication of foreign object detection. Other enhanced image indicates device status overlay and instrument indication.

Fig. 10 illustrates a system 83 for enhancing a surgical field image with information using an AR display 89 in accordance with at least one aspect of the present disclosure. The system 83 may be used to perform the techniques described below, for example, by using the processor 85. The system 83 includes one aspect of the AR device 66 that may communicate with a database 93. The AR device 66 includes a processor 85, a memory 87, an AR display 89, and a camera 79. The AR device 66 may include a sensor 90, a speaker 91, and/or a haptic controller 92. Database 93 may include an image store 94 or a pre-operative plan store 95.

The processor 85 of the AR device 66 includes an augmented reality modeler 86. The augmented reality modeler 86 may be used by the processor 85 to create an augmented reality environment. For example, the augmented reality modeler 86 may receive an image of an instrument in a surgical field of view, such as from the camera 79 or the sensor 90, and create an augmented reality environment to fit within a displayed image of the surgical field of view. In another example, physical objects and/or data may be superimposed on the surgical field of view and/or the surgical instrument image, and the augmented reality modeler 86 may use the physical objects and data to present an augmented reality display of the virtual objects and/or data in the augmented reality environment. For example, the augmented reality modeler 86 may use or detect an instrument at a surgical site of a patient and present virtual objects and/or data on the surgical instrument and/or images of the surgical site in a surgical field captured by the camera 79. The AR display 89 may display an AR environment superimposed on a real environment. Display 89 may display virtual objects and/or data using AR device 66, such as a fixed location in an AR environment.

The AR device 66 may include a sensor 90, such as an infrared sensor. The camera 79 or sensor 90 may be used to detect movements, such as gestures by a surgeon or other user, which the processor 85 may interpret as user attempts or intended interactions with the virtual target. The processor 85 may identify objects in the real environment, such as by using the processing information received by the camera 79. In other aspects, the sensor 90 may be a tactile, auditory, chemical, or thermal sensor to generate corresponding signals that may be combined with various data feeds to create an enhanced environment. The sensors 90 may include binaural audio sensors (spatial sound), inertial measurement sensors (accelerometers, gyroscopes, magnetometers), environmental sensors, depth camera sensors, hand-eye tracking sensors, and voice command recognition functions.

For example, during a surgical procedure, the AR display 89 may present virtual features corresponding to physical features hidden by anatomical aspects of the patient, such as within the surgical field, while allowing the surgical field to be viewed through the AR display 89. The virtual feature may have a virtual position or orientation corresponding to a first physical position or orientation of the physical feature. In one example, the virtual position or orientation of the virtual feature may include an offset from a first physical position or orientation of the physical feature. The offset may include a predetermined distance from the augmented reality display, a relative distance from the augmented reality display to an anatomical aspect, and the like.

In one example, the AR device 66 may be a single AR device. In one aspect, AR device 66 may be a Hololens 2AR device manufactured by Microsoft corporation of Redmond, wash. The AR device 66 includes goggles with lenses and binaural audio features (spatial sound), inertial measurement devices (accelerometers, gyroscopes, magnetometers), environmental sensors, depth and video cameras, hand-eye tracking, and voice command recognition functions. It provides an improved field of view with high resolution by using a mirror to guide the waveguide in front of the wearer's eye. The image can be magnified by changing the angle of the mirror. It also provides eye tracking to identify the user and adjust the lens width for the particular user.

In another example, the AR device 66 may be a Snapchat Spectacles 3AR device. The AR device is able to capture paired images and recreate a 3D depth map, add virtual effects, and replay 3D video. The AR device includes two HD cameras to capture 3D photos and videos at 60fps while four built-in microphones record immersive high-fidelity audio. The images from the two cameras combine to build a geometric map around the user's real world, providing a new perception of depth. The photos and videos may be synchronized to the external display device wirelessly.

In yet another example, AR device 66 may be a Google Glass 2AR device. The AR device provides inertial measurement (accelerometer, gyroscope, magnetometer) information superimposed on the lens (outside the field of view) to supplement the information.

In another example, the AR device 66 may be Amazon's Echo Frames AR device. The AR device has no camera/display. The microphone and speaker are connected to Alexa. The AR device provides less functionality than a heads-up display.

In yet another example, AR device 66 may be a North (Google) Focals AR device. The AR device provides notification push/smart watch simulation; inertial measurement, screen overlay of information (weather, calendar, messages), voice control (Alexa) integration. The AR device provides a basic heads-up display function.

In another example, the AR device 66 may be an Nreal AR device. The AR device includes spatial sound, two ambient cameras, photo cameras, IMU (accelerometer, gyroscope), ambient light sensor, proximity sensor functions. Nebula projects application information onto the lens.

In various other examples, AR device 66 may be any of the following commercially available AR devices: magic Leap 1, epson Moverio, vuzix Blade AR, zenFone AR, microsoft AR eyeglass prototype, eyeTap to generate light collinear with ambient light directly into the retina. For example, the beam splitter makes the same light seen by the eye available for computer processing and superimposing information. AR visualization systems include HUDs, contact lenses, glasses, virtual Reality (VR) headphones, virtual retinal displays, operating room displays, and/or smart contact lenses (biomimetic lenses).

The multi-user interface for AR device 66 includes a virtual retinal display (such as a raster display drawn directly on the retina rather than on a screen in front of the eye), a smart television, a smart phone, and/or a spatial display (such as a Sony spatial display system).

Other AR technologies may include, for example, AR capture devices and software applications, AR creation devices and software applications, and AR cloud devices and software applications. The AR capture device and software applications include, for example, apple polycyam application u binary 6 (Mirrorworld using display. Land app), the user can scan and obtain 3D images of the real world (to create a 3D model). AR creation devices and software applications include, for example, adobe Aero, vuforia, ARToolKit, google ARCore, apple arcet, MAXST, aurasma, zappar, blippar. AR cloud devices and software applications include, for example, facebook, google (world geometry, object recognition, predictive data), amazon AR cloud (business), microsoft Azure, samsung Project Whare, niantic, magic Leap.

Situational awareness refers to the ability of some aspects of a surgical system to determine or infer information related to a surgical procedure from data received from databases and/or instruments. The information may include the type of surgery being performed, the type of tissue being operated on, or the body cavity being the subject of the surgery. With context information associated with the surgical procedure, the surgical system may, for example, improve the manner in which the surgical system controls the modular devices (e.g., robotic arms and/or robotic surgical tools) connected thereto, and provide context information or advice to the surgeon during the course of the surgical procedure.

Fig. 11 shows a time axis of a situational awareness surgical procedure. Fig. 11 shows a timeline 5200 of an exemplary surgical procedure and context information that the surgical hub 5104 can derive from data received from the data source 5126 at each step of the surgical procedure. The time axis 5200 depicts typical steps that nurses, surgeons and other medical personnel will take during a segmental lung removal procedure, starting from the establishment of an operating room and until the patient is transferred to a post-operative recovery room. The situation aware surgical hub 5104 receives data from the data source 5126 throughout the surgical procedure, including data generated each time a medical professional utilizes the modular device 5102 paired with the surgical hub 5104. The surgical hub 5104 can receive this data from the paired modular device 5102 and other data sources 5126 and continually derive inferences about ongoing surgery (i.e., background information), such as which step of the surgery to perform at any given time, as new data is received. The situational awareness system of the surgical hub 5104 can, for example, record data related to the procedure used to generate the report, verify steps that medical personnel are taking, provide data or cues (e.g., via a display screen) that may be related to a particular procedure, adjust the modular device 5102 based on context (e.g., activate a monitor, adjust the FOV of a medical imaging device, or change the energy level of an ultrasonic surgical instrument or RF electrosurgical instrument), and take any other such action described herein.

First 5202, a hospital staff member retrieves the patient's EMR from the hospital's EMR database. Based on patient data selected in the EMR, the surgical hub 5104 determines that the procedure to be performed is a thoracic procedure.

Second 5204, the staff member scans the incoming medical supplies for the procedure. The surgical hub 5104 cross-compares the scanned supplies with the list of supplies utilized in the various types of protocols and confirms that the combination of supplies corresponds to the chest protocol. In addition, the surgical hub 5104 can also determine that the procedure is not a wedge procedure (because the incoming supplies lack certain supplies required for, or otherwise do not correspond to, a chest wedge procedure).

Third 5206, the medical personnel scans the patient belt via a scanner 5128 communicatively coupled to the surgical hub 5104. The surgical hub 5104 may then confirm the identity of the patient based on the scanned data.

Fourth 5208, the medical staff opens the auxiliary equipment. The auxiliary devices utilized may vary depending on the type of surgery and the technique to be used by the surgeon, but in this exemplary case they include smoke evacuators, insufflators and medical imaging devices. When activated, the ancillary equipment as the modular device 5102 may automatically pair with the surgical hub 5104 located in a specific vicinity of the modular device 5102 as part of its initialization process. The surgical hub 5104 may then derive background information about the surgical procedure by detecting the type of modular device 5102 paired therewith during this pre-operative or initialization phase. In this particular example, the surgical hub 5104 determines that the surgical procedure is a VATS procedure based on this particular combination of paired modular devices 5102. Based on a combination of data from the patient's EMR, a list of medical supplies to be used in the procedure, and the type of modular device 5102 connected to the hub, the surgical hub 5104 can generally infer the particular procedure that the surgical team will perform. Once the surgical hub 5104 knows the particular procedure being performed, the surgical hub 5104 can retrieve the procedure from memory or the cloud and then cross-reference the data it subsequently receives from the connected data sources 5126 (e.g., the modular device 5102 and the patient monitoring device 5124) to infer the procedure being performed by the surgical team.

Fifth 5210, the staff attaches EKG electrodes and other patient monitoring devices 5124 to the patient. The EKG electrode and other patient monitoring device 5124 can be paired with the surgical hub 5104. As the surgical hub 5104 begins to receive data from the patient monitoring device 5124, the surgical hub 5104 thus confirms that the patient is in the operating room.

Sixth 5212, medical personnel induce anesthesia in patients. The surgical hub 5104 may infer that the patient is under anesthesia based on data (including EKG data, blood pressure data, ventilator data, or a combination thereof) from the modular device 5102 and/or the patient monitoring device 5124. At the completion of the sixth step 5212, the preoperative portion of the lung segmental resection procedure is completed and the operative portion begins.

Seventh 5214, collapse of the lungs of the patient being operated on (while ventilation is switched to the contralateral lung). The surgical hub 5104 may infer from the ventilator data that the patient's lungs have collapsed. The surgical hub 5104 can infer that the surgical portion of the procedure has begun, as it can compare the detection of the patient's lung collapse to the expected steps of the procedure (which can be previously accessed or retrieved), thereby determining that collapsing the lung is a surgical step in that particular procedure.

Eighth 5216, a medical imaging device 5108 (e.g., an endoscope) is inserted and video from the medical imaging device is activated. The surgical hub 5104 receives medical imaging device data (i.e., still image data or live streaming video in real time) through its connection with the medical imaging device. After receiving the medical imaging device data, the surgical hub 5104 may determine that the laparoscopic portion of the surgical procedure has begun. In addition, the surgical hub 5104 may determine that the particular procedure being performed is a segmental resection, rather than a pneumonectomy (note that the surgical hub 5104 has excluded a wedge procedure based on data received at the second step 5204 of the procedure). The data from the medical imaging device 124 (fig. 2) may be used to determine background information related to the type of procedure being performed in a number of different ways, including by determining the angle of the visual orientation of the medical imaging device relative to the patient's anatomy, monitoring the number of medical imaging devices utilized (i.e., activated and paired with the surgical hub 5104), and monitoring the type of visualization device utilized.

For example, one technique for performing a vat lobectomy places the camera in the lower anterior corner of the patient's chest over the septum, while one for performing a vat segmented resection places the camera in an anterior intercostal position relative to the segmented slit. For example, using pattern recognition or machine learning techniques, the situational awareness system may be trained to recognize the positioning of the medical imaging device from the visualization of the patient anatomy. As another example, one technique for performing a vat lobectomy utilizes a single medical imaging apparatus, while another technique for performing a vat segmented excision utilizes multiple cameras. As another example, a technique for performing vat segmental resections utilizes an infrared light source (which may be communicatively coupled to a surgical hub as part of a visualization system) to visualize segmental slots that are not used in vat pulmonary resections. By tracking any or all of this data from the medical imaging device 5108, the surgical hub 5104 can thus determine the particular type of surgical procedure being performed and/or the technique for the particular type of surgical procedure.

Ninth 5218, the surgical team begins the anatomic steps of the procedure. The surgical hub 5104 can infer that the surgeon is in the process of dissecting to mobilize the patient's lungs because it receives data from the RF generator or ultrasound generator indicating that the energy instrument is being fired. The surgical hub 5104 can cross-reference the received data with a retrieval step of the surgical procedure to determine that the energy instrument fired at that point in the method (i.e., after completion of the previously discussed surgical step) corresponds to an anatomical step.

Tenth 5220, the surgical team proceeds to the ligation step of the procedure. The surgical hub 5104 can infer that the surgeon is ligating arteries and veins because it receives data from the surgical stapling and severing instrument indicating that the instrument is being fired. Similar to the previous steps, the surgical hub 5104 can derive the inference by cross-referencing the receipt of data from the surgical stapling and severing instrument with the retrieval steps in the method.

Eleventh 5222, a segmental resection portion of the procedure is performed. The surgical hub 5104 infers that the surgeon is transecting soft tissue based on data from the surgical instrument, including data from the staple cartridge. The cartridge data may correspond to the size or type of staples fired by the instrument. The cartridge data may indicate the type of tissue being stapled and/or transected for different types of staples employed in different types of tissue. The type of staples being fired is used for soft tissue or other tissue types to enable the surgical hub 5104 to infer that a segmental resection procedure is being performed.

Twelfth 5224, the node dissection step is performed. The surgical hub 5104 may infer that the surgical team is dissecting a node and performing a leak test based on data received from the generator indicating that an RF or ultrasonic instrument is being fired. For this particular procedure, the use of an RF or ultrasonic instrument after transecting the soft tissue corresponds to a node dissection step, which allows the surgical hub 5104 to make this inference. It should be noted that the surgeon switches back and forth between surgical stapling/cutting instruments and surgical energy (i.e., RF or ultrasonic) instruments periodically, depending on the particular step in the procedure, as the different instruments are better suited for the particular task. Thus, the particular sequence in which the stapling/severing instrument and the surgical energy instrument are used may dictate the steps of the procedure that the surgeon is performing. At the completion of the twelfth step 5224, the incision is closed and the post-operative portion of the procedure begins.

Thirteenth 5226, the patient is reversed from anesthesia. For example, the surgical hub 5104 may infer that the patient is waking from anesthesia based on ventilator data (i.e., the patient's respiration rate begins to increase).

Finally, fourteenth 5228, the medical personnel remove various patient monitoring devices 5124 from the patient. Thus, when the surgical hub 5104 loses EKG, BP and other data from the patient monitoring device 5124, the hub can infer that the patient is being transferred to the recovery room. The surgical hub 5104 can determine or infer when each step of a given surgical procedure occurs from data received from various data sources 5126 communicatively coupled to the surgical hub 5104.

In addition to using patient data from the EMR database to infer the type of surgical procedure to be performed, the situational awareness surgical hub 5104 may also use patient data to generate control adjustments for the paired modular device 5102, as shown in a first step 5202 of the timeline 5200 shown in fig. 11.

Anticipation of interactive utilization of public data overlays by different users

Having described the general implementation of the various surgical systems, surgical hubs, communication systems, augmentation systems, and augmented reality devices disclosed herein, such as surgical systems 1, 2, 50, 52, surgical hubs 6, 56, 5104, communication system 63, visualization system 8, augmentation system 83, imaging devices 24, 96, and AR devices 66, 84, the present disclosure now turns to describing various other implants of systems, hubs, and devices. For the sake of brevity, various details and implementations of systems, hubs, and devices described in the following sections that are similar to the various systems, hubs, and devices described above are not repeated herein. Any aspects of the systems, hubs, and devices described below may be incorporated into and/or implemented by the above systems, hubs, and devices.

Operating Room (OR) personnel typically include a combination of members having different roles, such as a surgeon, anesthesiologist, anesthesia nurse, surgical technician, resident, doctor assistant, and the like. Throughout the surgical procedure, people with different roles may rely on different information to make decisions. Thus, if a person does not receive relevant information about his role, significant errors may be caused in relation to the procedure.

To illustrate the importance of communicating relevant information to various OR people based on their roles, two exemplary scenarios are provided below. The first exemplary scenario involves trans-abdominal perinectomy performed by a surgical team and an anesthesia team. In this procedure, the patient experiences some blood loss and the patient's surgical team learns of the plan for the anesthesia team to administer blood transfusion to the patient with a third unit of packed red blood cells. The surgical team member raised his head to confirm and continue the procedure. Later, blood gas was withdrawn, showing that the patient had a hemoglobin (Hgb) level of 6.6g/dL. In contrast, at the beginning of the protocol, the patient's Hgb level was measured to be 14.8g/dL. When the surgical team learns of the previous 6.6g/dL Hgb measurement, the surgical team indicates that a greater amount of bleeding should be expected. However, the surgical team did not indicate why they expected greater bleeding. Because the anesthesia team is unable to meet the patient's blood, fluid, and resuscitation requirements, the attending anesthesiologist has three secondary surgical team offers up-to-date information about the patient's surgical context. In particular, the amount and source of blood loss (e.g., arterial, venous, or exudation) and the plan by which the surgical team is to continue to cut with continuous blood loss are not clearly communicated to the anesthesia team. The patient's hemodynamics continues to deteriorate. Thus, anesthesia teams initiate a number of transfusion protocols, transesophageal electrocardiography and rapid infusion. More anesthesiologists are also required to provide assistance. At this point, the surgical team eventually reveals to the anesthesia team that the patient's iliac vein was cut earlier in the procedure. Thus, the anesthesia team significantly underestimates how much the patient will suffer from blood loss due to the lack of real-time communication of relevant information from the surgical team.

A second exemplary scenario involves a selective laparoscopic left nephrectomy performed on a 43 year old female to remove renal cell carcinoma. The OR personnel include a surgical team led by a chief surgeon and an anesthesia team led by a chief anesthesiologist. Both the primary surgeon and the primary anesthesiologist agree that the risk of patient perioperative complications is low. The procedure proceeds as expected and the patient wakes up from anesthesia under steady conditions. However, after a few hours, the patient had developed abdominal tightness and swelling. In addition, patients show elevated heart rate and hypotension. The primary surgeon evaluates the patient and considers the patient experiencing internal bleeding. Thus, the primary surgeon returns the patient to the OR.

After returning to OR, a large infusion of blood product is started. After induction of anesthesia, the patient's blood pressure drops significantly. The anesthesia team administers the medication to try and improve the patient's blood pressure. In addition, the surgical team opens the abdomen of the patient and confirms the presence of massive internal bleeding. At this point, the vascular surgeon is called into the room and the surgeon attempts to control the patient's bleeding. The anesthesiologist continues to deliver blood products and larger doses of the boost medication. However, elevated ST of the patient's electrocardiogram indicates heart damage. The chief anesthesiologist communicates this information to the surgical team member. The surgical team continues to try to identify the source of the bleeding.

Finally, the surgical team determines the Inferior Vena Cava (IVC) of the patient as the source of bleeding. However, the patient is experiencing cardiac arrest. In response, the surgical team ceases the procedure because the anesthesia team attempts to resuscitate the patient. After 10 minutes of resuscitation attempts, over 100 units of blood product were administered during this period, achieving spontaneous circulation. Although the surgical team resumes the procedure, the primary anesthesiologist is concerned about the neurological status of the patient. In particular, no anesthetic was administered for a few minutes, indicating that the patient suffered brain damage caused by hypotension. The patient again begins to experience cardiac arrest. Although anesthesia teams try to resuscitate, they eventually stop because they think the patient is dying. However, the surgical team requires the anesthesia team to continue resuscitation as they attempt to control the patient's IVC bleeding.

It can be appreciated from these two exemplary scenarios that different surgical personnel can make decisions based on their roles requiring different information. For example, members of an anesthesia team may need to monitor both neurosurgical-related information and cardiac-related information. Neurosurgical-related information may include, for example, adequacy of oxygenation, deviation from expected hemodynamic parameters, blood loss, hemostatic difficulties, patient-related medical history (e.g., patient risk level), conscious levels, brainstem reflexes, gaze, muscle tone, response to pain, and the presence of occasional movements in the eyes, face, or limbs. The heart related information may include, for example, ultrasound cardiogram data, volume status, electrolyte level, presence of acidosis, related medical history of the patient (e.g., patient risk level), patient location (e.g., during renal cell carcinoma excision), information from a surgical team, and intra-abdominal pressure resulting from insufflation (e.g., related to hypertension risk, increased myocardial load, reduced venous return, reduced preload, reduced final ventricular diastolic volume). Thus, there are a wide variety of information that anesthesia team members may need to access during a surgical procedure. Likewise, surgical team members and other members of the OR personnel may similarly have a large number of information streams specific to their roles that they may need to rely on during a surgical procedure. Accordingly, there is a need for devices, systems, and methods for selectively presenting information to multiple users of a surgical system based on multiple data streams.

In various aspects, disclosed herein are devices, systems, and methods for selectively presenting information to multiple users of a surgical system based on multiple data streams. In some aspects, the devices, systems, and methods may include displaying an interactive overlay to provide information tailored to a particular user.

Fig. 12 illustrates a surgical system 14000 configured to display interactive overlays for multiple users based on multiple data streams in accordance with several non-limiting aspects of the present disclosure. The surgical system 14000 can include a surgical hub 14002 in communication with the patient monitoring device 14004, the surgical device/instrument 14006, the tracking system 14008, and the visualization system 14010. The surgical system 14000, surgical hub 14002, surgical devices/instruments 14006, and visualization system 14010 can be similar in many respects to any of the surgical systems, surgical hubs, surgical devices/instruments, and visualization systems described above (e.g., surgical systems 1, 2, 50, 52; surgical hubs 6, 56, 5104; devices/instruments 21; visualization systems 8, 58), respectively. As described above, the surgical instrument may be communicatively coupled to a surgical hub (e.g., the device/instrument 21 may be coupled to the surgical hub 56 of FIG. 5). Accordingly, the surgical hub 14002 can be configured to receive instrument data from the surgical device/instrument 14006 related to various sensed parameters and operational settings of the device/instrument 14006. Based on the instrument data received by the surgical hub 14002, the surgical hub 14002 can determine the operating parameters of the instrument. For example, based on instrument data received from the surgical device/instrument 14006, the surgical hub 14002 can determine operating parameters such as speed, force, firing speed, firing force, activation status, power level, activation time, energy mode, and instrument settings.

Still referring to fig. 12, the patient monitoring device 14004 can be any type of device configured to monitor various aspects of a patient related to a surgical procedure. For example, referring again to the exemplary surgical scenario discussed above, the patient monitoring device 14004 may be configured to be able to monitor various aspects of the patient that may be used by anesthesia team members (e.g., oxygenation, hemodynamic parameters, blood loss, echocardiographic data, volume status, electrolyte level, intra-abdominal pressure caused by insufflation, etc.). The surgical hub 14002 can be configured to receive patient monitoring data from the patient monitoring device 14004.

The tracking system 14008 and/or the visualization system 14010 may be similar in many respects to the tracking system 15006 and visualization system 15008 discussed in the aforementioned U.S. patent application serial No. currently filed with the present application under the designation "MIXING DIRECTLY VISUALIZED WITH RENDERED ELEMENTS TO DISPLAY BLENDED ELEMENTS AND ACTIONS HAPPENING ON-SCREEN AND OFF-SCREEN," attorney docket No. END9352USNP11/210120-11, the disclosure of which is incorporated herein by reference in its entirety. The tracking system 14008 may be configured to be able to track the position, location, motion, and/OR other attributes of various objects within the OR based on one OR more different types of tracking methods. For example, the tracking system 14008 and/OR the visualization system 1400 may utilize any combination of imaging devices (e.g., cameras, visual and/OR non-visual image sensors, etc.), structured light sensors, LIDAR (light detection and ranging) sensors, ground sensors, acoustic sensors, fiducial markers, user/device sensors, and GPS (global positioning system) sensors to track the position, location, and/OR movement of objects in the OR. The objects tracked by the tracking system may include OR personnel, surgical devices/instruments 14006, patients, AR devices 66, equipment, and the like. In some aspects, the surgical hub 14002 can be configured to receive data from the tracking system 14008 and/or the visualization system 14010 to determine the relative position of tracked objects, interactions between tracked objects, and/or the proximity of tracked objects to each other.

In various aspects, the surgical hub 14002 can be communicatively coupled to a cloud 14012 that can include a remote server 14014 having a data store 14016. Cloud 14012, remote server 14014, and data store 14016 may be similar in many respects to any of the cloud, remote server, and data store described herein (e.g., cloud 54, remote server 63, data store 55 of fig. 5), respectively. In some aspects, the data storage 14016 can store information related to a patient undergoing a surgical procedure. Patient information may include, for example, information related to patient history (e.g., patient risk level). The surgical hub may be configured to retrieve data from the data storage 14016.

Accordingly, in some aspects, the surgical hub 14002 can be configured to receive a plurality of data streams related to a surgical procedure. The plurality of data streams related to the surgical procedure can include any combination of data received from the patient monitoring device 14004, the surgical device/instrument 14006, the tracking system 14008, the visualization system 14010, and/or the server 14014.

Still referring to fig. 12, the surgical hub 14002 is communicatively coupled to the AR device 66. As described above, the AR device 66 may be similar in many respects to the AR device 84 disclosed with reference to fig. 10. The surgical hub 14002 may cause the AR device 66 to display information based on any of a plurality of data streams received from the patient monitoring device 14004, the surgical device/instrument 14006, the tracking system 14008, the visualization system 14010, and/or the server 14014. For example, the surgical hub 14002 may be configured to enable the AR device 66 to display an interactive overlay including information related to monitored aspects of the patient, surgical instrument operating parameters, tracked objects (e.g., device interactions), various captured images of the surgical field of view, and so forth. Thus, a user (e.g., an OR person) using the AR device 66 is able to receive information related to the surgical procedure in real-time based on at least some of the plurality of different data streams. The user may use this information to assist in making decisions during the surgical procedure. Moreover, due to the real-time display of this information, the user is able to make decisions that affect key aspects of the surgical procedure more effectively than other methods of communicating the information in the OR.

In some aspects, the interactive overlay displayed by AR device 66 may be customized based on the user that is using AR device 66. Further, if multiple users are using different AR devices 66, the interactive overlays displayed by each of the different AR devices 66 may be customized on a per user basis. For example, as described in the above exemplary surgical scenario, the OR personnel may generally include a surgical team and an anesthesia team. The surgical team member and the anesthesia team member may each wear the AR device 66. The surgical hub 14002 may be configured to enable the AR device 66 worn by the surgical team to display a first interactive overlay customized based on the information needs of the surgical team. The first interactive overlay may include information such as operating parameters of the device being used by the surgical team member and/or patient information related to the procedure steps being performed by the surgical team. Further, the surgical hub 14002 may be configured to enable the AR device 66 worn by the anesthesia team to display a second interactive overlay that is customized based on the information needs of the anesthesia team. The second interactive overlay may include information such as operating parameters of the device being used by the anesthesia team member and/or patient information related to the procedure steps being performed by the anesthesia team.

In some aspects, control of the surgical device/instrument 14006 can be transferred from a first user to a second user during a surgical procedure. For example, a first user may submit a surgical device/instrument 14006 to a second user. Prior to the handoff, the AR device 66 worn by the first user may be displayed superimposed based on data related to the surgical device/instrument 14006. The tracking system 14008 may be configured to be able to detect that the first user has transferred control of the surgical device/instrument 14006 to the second user. Based on data from the tracking system 14008 related to the detected transfer, the surgical hub 14002 may cause the interactive overlay displayed by the AR device 66 of the first user and the AR device 66 of the second user to be updated. For example, after switching, the AR device 66 worn by the second user may display an overlay based on the data related to the transferred surgical device/instrument 14006. Additionally, the AR device 66 worn by the first user may cease displaying overlays based on the data related to the transferred surgical device/instrument 14006.

In some aspects, the information needs of the user may be superimposed. Thus, the superposition displayed by the various AR devices 66 of the surgical system 14000 may be based on at least some of the same information (i.e., may be based on at least some of the same data streams received by the surgical hub 14002).

In some aspects, the AR device 66 of the surgical system 14000 may be linked to a particular user by a surgical hub 14002. As used herein, the term "link" when referring to an AR device linked to a particular user may mean that the surgical hub 14002 has determined that the particular user is using the AR device. Linking the particular AR device 66 to the particular user may allow the surgical hub 14002 to cause the particular AR 66 to display a customized overlay for the particular user.

In some aspects, the surgical hub 14002 may link the AR device 66 to the user based on information stored by the surgical hub 14002 and/or based on data stored by the data storage device 14016. In one aspect, the data storage 14016 may be a data storage of a manufacturer of the hospital network or surgical system 14000. The data store 14016 may include information identifying OR personnel expected to occur for various types of procedures. In addition, the data store 14016 may include information related to the scheduling system of the hospital, such as which people are scheduled to be present during the procedure. Accordingly, the surgical hub 14002 may be configured to identify a particular AR device 66 that may be linked to an intended OR person. Thus, the surgical hub 14002 may cause the AR device 66 to display a customized overlay based on the role of the intended OR person. For example, referring again to the second exemplary surgical scenario described above, the data storage device 14016 may store information related to an OR person desiring to perform a selective laparoscopic left nephrectomy. Contemplated personnel may include surgical teams and anesthesia teams. The surgical hub 14002 may identify the particular AR device 66 that should be worn by the person based on their respective roles.

In some aspects, the surgical hub 14002 can be configured to provide external notification when an OR person is expected to change during a surgical procedure. In one aspect, these external notifications may be provided via an AR device or other type of display device used by personnel in a different operating room or elsewhere in the hospital (e.g., surgical system 14000 may be similar to surgical data network 51 of fig. 4, with modular communication hub 53 configured to be able to connect to modular devices located in one or more operating rooms/rooms of a medical facility). For example, referring again to the second exemplary surgical scenario described above, an expert such as a vascular surgeon may be called an OR based on a particular event (e.g., unexpected uncontrolled bleeding) that occurs during a surgical procedure. The surgical hub 14002 can be configured to detect the occurrence of an event (e.g., the surgical hub 14002 can be similar to the case aware surgical hub 5104 described above with reference to fig. 11) and cause a relevant person, such as a vascular surgeon in the above example, to receive a notification indicating that the person is or may be required for a procedure. In one aspect, the notification may be bypassed. In another aspect, notifications may be classified based on the risk level and/or severity of the detected event. For example, depending on the level of risk of the detected event, the surgical hub 14002 may deliver an early warning notification to the armed expert indicating that the expert may be needed in the OR. In another aspect, the notification may enable a person not in the OR to view an image of the surgical field and/OR other overlays that the person in the OR is viewing. For example, upon receiving the notification, a standby vascular surgeon not in the OR but wearing the AR device 66 can cause the AR device 66 to display the same interactive overlay displayed on the AR device 66 of the member of the surgical team performing the surgical procedure in the OR (e.g., allowing the vascular surgeon to view vital signs, warnings, etc.).

In some aspects, an OR person can customize the overlay displayed by his AR device 66. For example, at the beginning of a procedure, the AR device 66 may display a menu having a plurality of functional levels that may be selected by the user. In one aspect, the menu may include three levels of functionality, wherein selecting a first level causes a superposition to be displayed that relates to the overall location of the person, wherein selecting a second level causes perioperative data and/or information to be displayed that relates to the patient history, and wherein selecting a third level causes user preferences and/or priorities to be displayed. The user can adjust the settings associated with each selected level.

In some aspects, the surgical hub 14002 may link the AR device 66 to the user based on data from the tracking system 14008. In one aspect, the tracking system 14008 may be configured to be able to detect when a particular user picks up or otherwise controls a particular AR device. For example, as detailed in the aforementioned U.S. patent application serial No. currently filed with the present application under the designation "MIXING DIRECTLY VISUALIZED WITH RENDERED ELEMENTS TO DISPLAY BLENDED ELEMENTS AND ACTIONS HAPPENING ON-SCREEN AND OFF-SCREEN," attorney docket No. END9352USNP11/210120-11, the disclosure of which is incorporated herein by reference in its entirety, a user may wear a user sensor (e.g., a smart glove) that is tracked by tracking system 14008. The tracking system 14008 may detect the proximity of the user sensor to the AR device 66 and, based on the detected proximity, the surgical hub 14002 may link the AR device 66 to the user. In another aspect, the surgical hub 14002 can be configured to enable the linked AR device 66 to automatically display overlays based on the proximity of the tracked user sensor to the surgical device/instrument 14006. For example, the proximity of the user sensor to the surgical device/instrument 14006 (e.g., surgical stapler) may cause the linked AR device 66 to display parameters (e.g., power level, type of staple cartridge installed therein, etc.) of the surgical device/instrument 14006.

In some aspects, after the AR device 66 is linked to the user by the surgical hub 14002, the surgical hub 14002 may cause the settings and/or overlays displayed by the AR device 66 to be updated. For example, the surgical hub 14002 may determine that the AR device 66 is linked to a user with a particular level of experience (e.g., a resident or experienced surgeon). The surgical hub 14002 may cause the AR device 66 to display different levels of information depending on the level of experience of the user. In another aspect, the surgical hub 14002 may select information to display on the AR device 66 based on previous use of the AR device 66 by a particular user (e.g., based on machine learning, based on the user's experience with the AR device 66). In another aspect, the user may override the automatic linking of AR device 66 to the user. In yet another aspect, the user may manually link the AR device to the user (e.g., using controls included on AR device 66).

Fig. 13 illustrates a method 14100 of displaying interactive overlays for multiple users of a surgical system in accordance with several non-limiting aspects of the present disclosure. The method 14100 can be implemented by any combination of a surgical system, a surgical hub, a tracking system, a visualization system, a patient monitoring device, a surgical device/instrument, an AR device, any component thereof, and any other device and system disclosed herein (such as surgical systems 1, 2, 50, 52, 14000, surgical hubs 6, 56, 5104, 14002, tracking system 14008, visualization system 8, 14010, patient monitoring device 14004, surgical device/instrument 14006, and AR device 66, 84).

According to method 14100, a surgical hub can receive 14102 a plurality of data streams related to a surgical procedure. The first augmented reality display device can be communicatively coupled 14104 to a surgical hub. The surgical hub can link 14106 the first augmented reality display device to the first user. The first augmented reality display device may display 14108 a first interactive overlay customized for the first user based on at least one of the plurality of data streams.

According to one aspect of the method 14100, a second augmented reality display device can be communicatively coupled to the surgical hub. Further, the surgical hub may link the second augmented reality display device to the second user. The second augmented reality display device may display a second interactive overlay customized for a second user based on at least one of the plurality of data streams.

According to another aspect of method 14100, the first augmented reality device and the second augmented reality device may simultaneously display the first interactive overlay and the second interactive overlay. In another aspect, the first interactive overlay and the second interactive overlay may be based on the same data stream of the plurality of data streams.

According to another aspect of the method 14100, the first and second augmented reality devices may display the first and second interactive overlays differently for the first and second users, respectively, based on respective preferences of the first and second users.

According to another aspect of the method 14100, the first augmented reality display device may display a first interactive overlay based on a first data stream associated with a first surgical instrument when the first user is using the first surgical instrument, and the second augmented reality display device may display a second interactive overlay based on a second data stream associated with a second surgical instrument when the second user is using the second surgical instrument. In yet another aspect, the first augmented reality display device may update the first interactive overlay to be based on the second data stream when the first user is using the second surgical instrument. In one aspect, the plurality of data streams may include a first data stream and a second data stream.

Decision matrix based on wrong risk level and user notification of surgeon response

As detailed above, communication of information may be critical during performance of a surgical procedure. Thus, augmented reality systems that utilize data streams from multiple sources (such as patient monitoring devices, surgical devices/instruments, tracking systems, and other data storage devices) may rely on seamless integration of the interconnection system to collect, interpret, and transmit these data streams between the devices and the OR personnel throughout the procedure. However, there may be circumstances in which data is erroneously transmitted and/or the transmission of a data stream is interrupted or lost. Accordingly, there is a need for devices, systems, and methods for detecting errors and/or data communication problems and determining appropriate actions to be performed when errors and/or data communication problems occur. Further, there is a need for devices, systems, and methods that are capable of performing actions based on different risk levels associated with the criticality of the detected error and/OR based on the response of the OR person to a previous alert.

Disclosed herein are devices, systems, and methods for detecting errors and/or data communication problems and determining appropriate actions to be performed when errors and/or data communication problems occur. In various aspects, the apparatus, systems, and methods may implement a decision matrix based on the type of error detected. The decision matrix may trigger various actions, such as user notification, system override, and device locking, based on different degrees of risk associated with the detected error. The decision matrix may also trigger various actions based on previous responses by the user to similarly detected errors.

FIG. 14 illustrates a method 14200 for detecting device-related errors and determining actions to implement based on the detected errors, in accordance with several non-limiting aspects of the present disclosure. The method 14200 may be implemented by any combination of a surgical system, surgical hub, tracking system, visualization system, patient monitoring device, surgical device/instrument, AR device, any component thereof, and any other device and system disclosed herein (such as surgical system 1, 2, 50, 52, 14000, surgical hub 6, 56, 5104, 14002, tracking system 14008, visualization system 8, 14010, patient monitoring device 14004, surgical device/instrument 14006, and AR device 66, 84).

Referring primarily to fig. 14 and also to fig. 12, according to method 14200, the surgical hub 14002 can detect 14202 alarms related to operation of the surgical device/instrument 14006. For example, the device/instrument 14006 may indicate that the operating parameters of the instrument are not within the expected and/or suggested operating ranges. Next, the surgical hub 14002 can determine 14204 that the detected alarm is functioning properly. For example, the surgical hub 14002 can determine 14204 that the operating parameters of the device/instrument 14006 are not in fact within the recommended operating range. Upon determining 14204 whether the detected alarm is functioning properly, the surgical hub 14002 may cause the AR device 66 to display 14206 an alarm (e.g., an overlay) related to the detected error. For example, the AR device 66 may display a superposition that instructs the user to take a particular action to resolve the detected error (e.g., waiting for the temperature of the instrument to return to the recommended operating range).

In another aspect of the method 14200, the surgical hub may determine 14204 that the alert detected is not functioning properly. Upon determining 14204 that the alarm is not functioning properly, the surgical hub 14002 can determine 14208 whether the detected error is a low risk level error. For example, the surgical hub 14002 may determine that the device/instrument 14006 is generating erroneous data only at low risk conditions (e.g., the instrument is erroneously generating a high temperature reading only when the instrument is operating at low temperature conditions). Upon determining 14208 that the detected error is a low risk level error, the surgical hub 14002 may determine that no warning is provided 14210 to the user based on the detected error.

In another aspect, upon determining 14204 that the detected alarm is not functioning properly, the surgical hub 14002 can determine 14212 whether the detected alarm is a range-based high risk error. For example, the surgical hub 14002 may determine that the device/instrument 14006 is generating erroneous data under high risk conditions (e.g., the instrument is erroneously generating a temperature reading when the instrument is operating at high temperatures, resulting in a risk of overheating). Upon determining 14212 that the detected alarm is a range-based high risk error, the surgical hub 14002 may determine 14214 a frequency of range-based high risk errors. For example, when the percentage of error data generated by the device/instrument 14006 is above a predetermined threshold (e.g., 10%, 20%, 30%, 40%, 50%, 60%, 70%, 80%, or 90% of the error readings generated by the instrument indicate that the operating parameters of the instrument are not within the expected and/or suggested ranges), the range-based high risk errors may have a high error frequency. In various aspects, the frequency threshold may be determined 14214 based on the sampling rate of the data.

Upon determining 14214 that the range-based high risk error is above the frequency threshold, the surgical hub 14002 may cause the AR device 66 to display an alarm (e.g., an overlay) related to the detected error and deactivate 14216 the surgical instrument/device 14006. For example, the surgical hub 14002 may cause the AR device 66 to display a superposition indicating that the instrument/device 14006 has been locked due to an error associated with the detected operating parameter. Conversely, upon determining 14214 that the range-based high risk error is below the frequency threshold, the surgical hub 14002 may cause the AR device 66 to display 14218 an alert (e.g., an overlay) related to the detected error. For example, the surgical hub 14002 may cause the AR device 66 to display a superposition indicating that the user verified the accuracy of the detected operating parameters via an alternative approach. In this aspect, the surgical hub 14002 may not lock the surgical instrument/device 14006.

In another aspect of the method 14200, when it is determined 14204 that the detected alarm is not functioning properly, the surgical hub 14002 can determine 14220 whether the detected error is a communication-related error. Communication-related errors may be caused by a loss of communication between the surgical hub 14002 and the surgical instrument 14006. In one aspect, upon determining 14220 that the detected alert is a communication-related error, the surgical hub 14002 may cause the AR device 66 to display an alert (e.g., an overlay) related to the detected error. For example, the surgical hub 14002 may cause the AR device 66 to display an overlay indicating a loss of communication with the surgical instrument/device 14006. In another aspect, the surgical hub 14002 can deactivate 14222 the surgical instrument/device 14006, thereby preventing further use of the instrument/device 14006.

15A, 15B, 15C, 15D, 15E, and 15F illustrate an exemplary implementation of the method 14200 of FIG. 14 during thyroidectomy according to several non-limiting aspects of the present disclosure. Referring to fig. 15A, a first step 14300 of a thyroidectomy may include forming an incision 14302 to gain access to the underlying tissue 14306. The first step 14300 may further include using the ultrasonic energy device 14304 to incise the platysma in the underlying tissue 14306. At this point in the procedure, the blade of the ultrasonic energy device 14304 may begin to heat up from repeated use. Referring now to fig. 15A and 12, the ultrasonic energy device 14304 may be a surgical instrument/device 14006 in communication with the surgical hub 14002. The surgical hub 14002 may receive data related to the temperature of the blade from the ultrasonic energy device 14304.

Referring now to fig. 15B, a second step 14310 of the thyroidectomy includes further dissecting the zonal muscle 14312. This second step 14310 may be performed using a blunt device 14314 (shown) or an ultrasonic energy device 14304 (not shown). The use of the ultrasonic energy device 14304 may cause the blade temperature to continue to rise.

Referring now to fig. 15C and 15D, a third step 14320 of the thyroidectomy may include identifying critical structures such as the superior thyroid artery 14324, the inferior thyroid artery 14326, and the recurrent laryngeal nerve 14328. Fig. 15C shows a schematic diagram of an exemplary anatomy 14322 of an suprathyroiditis 14324, a hypothyroid artery 14326, and a recurrent laryngeal nerve 14328. Fig. 15D shows how the anatomical structures 14329a, 14329B, and 14329C vary with different configurations of the recurrent laryngeal nerve 14328 relative to the hypothyroid artery 14326. The third step 14320 may be the introduction step of a thyroidectomy, as it is critical that the recurrent laryngeal nerve 14328 is not damaged in general.

Referring now to fig. 15E, a fourth step 14330 of the thyroectomy may include using an ultrasonic energy device 14304 to segment the hypothyroid artery 14326. However, as described above, the blade temperature of the ultrasonic energy device 14304 may continue to rise as it is used during the procedure. If the blade temperature is too hot, the recurrent laryngeal nerve 14328 may be damaged. For example, as heat accumulates at the blade, the extent of lateral heat diffusion increases. Activation of the ultrasound energy device 14304 in a small space near the laryngeal nerve 14328 can result in damage to the nerve 14328. As another example, between activations, the surgeon may want to grasp and manipulate tissue using the ultrasonic energy device 14304. Doing so with a hot blade may result in damage to the laryngeal nerve 14328. As another example, in small spaces, slight movements of the ultrasonic energy device 14304 may result in contact with the laryngeal nerve 14328 between activations. An inexperienced user may not be able to predict the impact of the ultrasonic energy device 14304 on the tissue when the user is not actively applying energy. Thus, it is important for the ultrasonic energy device 14304 to communicate accurate temperature readings to the surgical hub. In addition, if there is an error associated with the temperature reading, the surgical hub may need to respond appropriately to avoid damaging the laryngeal nerve 14328.

Still referring to fig. 15E and also to fig. 12 and 14, upon receiving a hot blade notification from the ultrasonic energy device 14304, the surgical hub 14002 may implement a method 14200 for detecting device-related errors and determining an action to be performed based on the detected errors. For example, the surgical hub 14002 may detect 14202 a hot blade notification. If the surgical hub 14002 determines 14204 that the detected alarm is functioning properly, the surgical hub 14002 may cause the AR device 66 to display a warning 14206 (e.g., an overlay) indicating that the user is paused in order to cool the blades of the ultrasonic energy device 14304. Once cooled, the AR device 66 may indicate that the user may proceed without risk of damaging the laryngeal nerve 14328 due to the hot blade.

However, the surgical hub 14002 may determine 14208 that the hot blade notification is a low risk level error. For example, the surgical hub 14002 may determine that the ultrasonic energy device 14304 only generates a hot blade notification if the blade is not actually overheated. In this case, the surgical hub 14002 can determine that no alert will be provided 14210 to the user.

Still referring to fig. 15D and also to fig. 12 and 14, the surgical hub 14002 may determine 14212 that the hot blade notification is a range-based high risk error. For example, the ultrasonic energy device 14304 may generate an abnormally high temperature reading. The surgical hub 14002 may determine 14214 whether to deactivate 14216 the ultrasonic energy device 14304 and alert the user or alert 14218 the user based solely on the frequency of the high temperature readings.

Still referring to fig. 15D and also to fig. 12 and 14, the surgical hub 14002 may determine 14220 that the hot blade notification is based on a loss of communication with the ultrasonic energy device 14304. In this case, the surgical hub 14002 may 14222 deactivate (e.g., lock in) the ultrasonic energy device 14304 and cause the AR device 66 to display an alert informing the user of the loss of communication. The locking may be temporary to provide cooling time for the blades of the ultrasonic energy device 14304. Furthermore, the duration of the temporary lock may be based on the time it takes to cool the blade.

Referring now to fig. 15F, after the hot blade notification has been processed according to method 14200, the final step 14340 of the thyroidectomy may include rotating the thyroid leaf 14342 to gain access to the thyroid zonules (Berry ligaments) and associated vasculature. Further, the thyroid may be removed, the wound may be irrigated, the zonal muscles 14312 may be closed, and the skin may be closed.

Superimposed deployment and management power requirements

As detailed above, communication of information may be critical during performance of a surgical procedure. In some aspects, the information may be communicated to a user (e.g., an OR person) via an intraoperative display of a wearable display device (e.g., AR device 66, 84). In some aspects, battery power with limited battery life may be used to power the AR device. Thus, there may be a risk that the AR display device will lose power and/or have low power during a surgical procedure. Accordingly, there is a need for apparatus, systems, and methods for managing power of AR devices.

In various aspects, disclosed herein are apparatuses, systems, and methods for managing power of an AR device. In some aspects, power of an AR device (e.g., AR devices 66, 84) may be managed by ensuring that a second AR device is available to a user in the event that a first AR device used by the user loses power. Referring again to fig. 12, the surgical system 14000 may include a surgical hub 14002 in communication with the AR device 66. In one aspect, each AR device 66 in communication with the surgical hub 14002 may have the same capabilities (i.e., the same function). Further, as described above, various AR devices 66 may display customization information based on the user to which the device is linked. Thus, in the event that a first AR device 66 worn by a first user (e.g., an AR device 66 worn by a surgeon) loses power, a second AR device 66 (e.g., a backup AR device 66, an AR device 66 worn by a different member of the OR staff) may be linked with the first user. For example, the first user may be associated with a first user profile implemented by AR device 66. The user profile may include various settings and preferences of the first user. When the first AR device 66 loses power, the second AR device 66 may link to the first user, causing the second AR device to implement the first user's profile. In some aspects, the linking of the second AR device 66 to the first user may be performed by the surgical hub 14002 using various techniques described herein. In another aspect, the linking of the second AR device 66 to the first user may be performed by selecting a profile of the first user (e.g., via an ID scan) on the second AR device 66.

In some aspects, one OR more of AR devices 66 may display a notification indicating the remaining power available to the various AR devices 66 worn by OR personnel. For example, the tour nurse may receive a notification indicating that the AR device 66 linked to the surgeon is at low power. As another example, the tour nurse may receive a notification indicating the power level of all AR devices 66 linked to the OR person. In one aspect, notifications may be listed based on the priority of each of the OR personnel (e.g., ranked based on which OR personnel have the lowest power level, ranked based on which OR personnel are most critical to performing a surgical procedure, etc.). In another aspect, the notification may include an estimated time of remaining battery life on the AR device 66. Thus, the round nurse may identify which OR personnel may need a new AR device 66 based on the power level notification and prepare the OR personnel for providing an alternate AR device 66.

In other aspects, the power of AR device 66 may be managed by setting a priority of the functions performed by AR device 66. In one aspect, the AR device 66 may be configured to be capable of various levels of functionality. For example, the AR device 66 may be configured to have three levels of functionality, wherein a level 1 functionality causes information related to overall location (e.g., a full imaging and overlay functionality) to be displayed, wherein a level 2 functionality causes information related to perioperative data and/or patient history to be displayed, and wherein a level 3 functionality causes information related to user preferences and/or priorities to be displayed. The level 1 functions may include a level 2 function and a level 3 function. Further, the level 2 functions may include a level 3 function.

In some aspects, the AR device 66 may be configured to enable a user to select a level of functionality of the AR device 66. For example, before the surgical procedure begins, the user may select a level 1 function (e.g., full power mode). As the battery of AR device 66 approaches depletion, the AR device may be configured to be able to adjust the level of functionality, thereby conserving battery life. For example, the AR device 66 may be configured to be able to enter a low power mode that enables only level 2 or level 3 functions. In one aspect, in the low power mode, AR device 66 may display only standard vital signs, emergency alerts, and/or displays related to level 2 and/or level 3 functions. On the other hand, in the low power mode, the user may no longer be able to use the high power consumption function (e.g., slide to view the AR display of other users). Thus, the AR device 66 may be configured to mitigate low power and/or fully depleted power interruption providing information to a user of the device.

Multistage pairing method for ensuring safety of device/system

Network security is often a concern where various devices are wirelessly connected to the system. As detailed above, the various surgical systems described herein may include various smart devices (e.g., smart surgical instruments, patient monitoring devices, tracking devices, AR devices, etc.) that are wirelessly connected to a surgical hub. Thus, unauthorized devices may attempt to take advantage of the wireless capabilities of the surgical system. Furthermore, unauthorized communication between the damaged device and the various components of the surgical system may result in incorrect data being presented to the OR personnel. If the OR personnel rely on this incorrect data, it may ultimately result in the OR personnel making an incorrect decision during the surgical procedure. Accordingly, there is a need for apparatus, systems, and devices for ensuring secure pairing of various intelligent devices of a surgical system. Furthermore, the smart device may be damaged and/or destroyed after initial pairing with the surgical system. Accordingly, there is also a need for apparatus, systems, and methods for verifying secure and/or authenticated connections to paired smart devices.

Disclosed herein are apparatuses, systems, and methods for ensuring secure wireless pairing of a smart device with a surgical system. The smart devices discussed below may include any device disclosed herein that may be configured to wirelessly communicate with a surgical system (e.g., surgical instrument/device 14006, AR devices 66, 84, etc.). Referring again to fig. 12, surgical system 14000 may include a surgical hub 14002, a tracking system 14008, a cloud 14012 (including a server 14014), surgical devices/instruments 14006, and AR devices 66. In some aspects, at least some of the surgical devices/instruments 14006 and/or AR devices 66 (sometimes collectively referred to herein as devices 14006, 66) can be configured to wirelessly mate with the surgical hub 14002. The surgical hub 14002 can be configured to classify a security level of a connection to the paired device 14006, 66 based on the technique used to pair the devices 14006, 66. In one aspect, the classified security levels may include a high security level, a medium security level, and a low security level, wherein the high security level is safer than the medium security level and the medium security level is safer than the low security level.

In various aspects, the devices 14006, 66 may be semi-automatically paired at the inventory level. The inventory level pairing may include the tracking system 14008 identifying devices 14006, 66 as the OR fills (i.e., reserves) for the prepared surgical procedure. For example, various imaging devices and/OR other tracking techniques employed by the tracking system 14008 may automatically detect devices as the devices 14006, 66 enter the OR. The surgical hub 14002 can identify the devices 14006, 66 based on data from the tracking system 14008. The surgical hub 14002 can be configured to cross-reference the identified devices 14006, 66 with data stored by the server 14014 (e.g., hospital web server, device manufacturer server) to retrieve MAC (media access control) addresses associated with the identified devices 14006, 66. Based on the MAC address, the surgical hub 14002 may begin actively searching for the identified device 14006, 66 for wireless pairing. In one aspect, the surgical hub 14002 can automatically mate the devices as the devices 14006, 66 are energized (e.g., an OR person inserts a battery into the devices 14006, 66). In this regard, the surgical hub 14002 can identify the pairing of the devices 14006, 66 as a high security level pairing. In another aspect, the OR person can manually pair the devices 14006, 66 to the surgical hub (e.g., using controls/buttons included on the devices 14006, 66). In this regard, the surgical hub 14002 can identify the pairing of devices 14006, 66 as a medium security level pairing.

In various aspects, the devices 14006, 66 can be semi-automatically paired at the packaging level. The package-level pairing may include a QR (quick response) code included on the package through the surgical hub 14002 or its component scanning devices 14006, 66. For example, an OR person may bring the devices 14006, 66 into the OR to prepare for the surgical procedure. The devices 14006, 66 may still be in their manufacturing packages (e.g., tyvek packages). The package may include a QR code to identify the device. In one aspect, an OR person may scan the QR code using an imaging device associated with the surgical hub 14002. In another aspect, the imaging device of the tracking system 14008 may automatically scan the QR code. The surgical system 14002 may be configured to identify the devices 14006, 66 based on the scanned QR code. Accordingly, the surgical hub 14002 can begin actively searching for the identified devices 14006, 66 for wireless pairing. In one aspect, the surgical hub 14002 can automatically mate the devices as the devices 14006, 66 are energized (e.g., an OR person inserts a battery into the devices 14006, 66). In this regard, the surgical hub 14002 can identify the pairing of the devices 14006, 66 as a high security level pairing. In another aspect, the OR person can manually pair the devices 14006, 66 to the surgical hub (e.g., using controls/buttons included on the devices 14006, 66). In this regard, the surgical hub 14002 can identify the pairing of devices 14006, 66 as a medium security level pairing.

In various aspects, the devices 14006, 66 may be paired semi-automatically at the device level. The semiautomatic device level pairing may include a QR (quick response) code included on the device 14006, 66 through the surgical hub 14002 or its component scanning device. For example, an OR person can bring the devices 14006, 66 into the OR to prepare the surgical procedure and remove it from its packaging. In one aspect, an OR person may scan the QR code using an imaging device associated with the surgical hub 14002. In another aspect, the imaging device of the tracking system 14008 may automatically scan the QR code. The surgical system 14002 may be configured to identify the devices 14006, 66 based on the scanned QR code. Accordingly, the surgical hub 14002 can begin actively searching for the identified devices 14006, 66 for wireless pairing. In one aspect, the surgical hub 14002 can automatically mate the devices as the devices 14006, 66 are energized (e.g., an OR person inserts a battery into the devices 14006, 66). In this regard, the surgical hub 14002 can identify the pairing of the devices 14006, 66 as a high security level pairing. In another aspect, the OR person can manually pair the devices 14006, 66 to the surgical hub (e.g., using controls/buttons included on the devices 14006, 66). In this regard, the surgical hub 14002 can identify the pairing of devices 14006, 66 as a medium security level pairing.

In various aspects, the devices 14006, 66 may be paired manually at the device level. Manual device-level pairing can include an OR person initiating pairing of the devices 14006, 66 with the surgical hub 14002 using controls and/OR buttons included on the devices 14006, 66. For example, as an OR person brings the devices 14006, 66 into the OR to prepare for a surgical procedure, the person may insert a battery into the devices 14006, 66 OR otherwise energize the devices 14006, 66. Further, a person can initiate pairing of the devices 14006, 66 with the surgical hub 14002 using controls and/or buttons included on the devices 14006, 66. Upon recognizing that the devices 14006, 66 are attempting to mate, the surgical hub 14002 can be configured to enable the devices 14006, 66 to require the user to conform to the pairing. In this regard, the surgical hub 14002 can identify the pairing of devices 14006, 66 as a low security level pairing.

Also disclosed herein are apparatuses, systems, and methods for verifying secure and/or authenticated connections to paired smart devices (devices 14006, 66). Various techniques may be implemented to verify and/or authenticate the connected devices 14006, 66 after pairing, such as verifying duplicate data streams, checksum checks, "false" signal checks, data error threshold checks, and pre-procedure checks of the data streams.

In some aspects, the devices 14006, 66 may be authenticated by verifying the repeated data streams sent by the devices 14006, 66 after the initial pairing. For example, the devices 14006, 66 can be configured to wirelessly transmit a repeated data stream (e.g., repeated packets sent multiple times) to the surgical hub 14002. The surgical hub 14002 can be configured to cross-check each of the received data streams to verify the authenticity of the devices 14006, 66.

In some aspects, the devices 14006, 66 may be authenticated based on a "false" signal check. The spurious signal may refer to a signal transmitted by the devices 14006, 66, which the surgical hub 14002 may compare with an expected signal to verify the authenticity of the devices 14006, 66. In one aspect, the dummy signal may be transmitted by the means 14006, 66 at a predefined cadence. If the surgical hub 14002 receives the artifact signal at the expected predefined cadence, it can authenticate the device. In another aspect, the predefined cadence may be adjusted to ensure data authenticity. In yet another aspect, the surgical hub 14002 can be configured to authenticate the devices 14006, 66 based on the timing of the received artifact signals and/or data included in the artifact signals. The surgical hub 14002 can index the signals received by the various devices 14006, 66 so that the security of the various devices 14006, 66 can be verified based on a particular point in time. In yet another aspect, the spurious signals expected by the surgical hub 14002 may be based on signals received by the surgical hub 14002 from the devices 14006, 66 after initial pairing. For example, as described above, the devices 14006, 66 may be initially paired to the surgical hub 14002 based on package-level semi-automatic pairing. After pairing, the devices 14006, 66 can transmit an initial artifact signal that is stored by the surgical hub 14002. The surgical hub 14002 can compare the subsequent artifact signals received from the devices 14006, 66 to the initial artifact signals to authenticate the devices 14006, 66. In another aspect, the artifact may comprise data derived from a manufacturing calibration of the devices 14006, 66. Data derived from manufacturing calibration of the devices 14006, 66 may be stored by the devices (e.g., by EEPROM of the devices). In another aspect, the QR code on the package of the device may include data related to the artifact.

In some aspects, the devices 14006, 66 can be authenticated after initial pairing by monitoring the data error rate of data received by the surgical hub 14002 from the devices 14006, 66. For example, if the error rate of the data received from the devices 14006, 66 exceeds a data error rate threshold, the surgical hub 14002 may be configured to identify the devices 14006, 66 as damaged. In one aspect, an error rate threshold (e.g., six sigma) may be predefined. In another aspect, the data error threshold may be determined based on a risk level associated with the devices 14006, 66. For example, each type of device may have a different error rate threshold. As another example, the error rate threshold may be based on the type of procedure being performed by the device (e.g., based on whether critical structures are present in the surgical field, based on whether arteries are present in the surgical field, based on the location of the device, based on whether the device is performing anatomy and/or manipulating tissue). As another example, the error rate threshold may be combined based on the type of device and the type of procedure the device is performing (e.g., an ultrasound device performing a thyroidectomy may have a different error rate threshold than a device performing PFS firing on a pulmonary artery or using a device that acts as a grasper, etc.).

In some aspects, the devices 14006, 66 can be authenticated through a pre-procedure check of the data stream received by the surgical hub 14002 from the devices 14006, 66. For example, the devices 14006, 66 may be endoscopic cutters. The surgical hub 14002 may require the endoscopic cutter to perform a test firing before the surgical procedure begins. Based on the data received by the surgical hub 14002 relating to the test firing, the surgical hub 14002 can verify the authenticity of the connection between the endoscopic cutter (devices 14006, 66) and the surgical hub 14002.

Fig. 16 illustrates a method 14400 for ensuring secure wireless pairing of a smart device with a surgical system in accordance with several non-limiting aspects of the present disclosure. The method 14400 may be implemented by any combination of a surgical system, surgical hub, tracking system, visualization system, patient monitoring device, surgical device/instrument, AR device, any component thereof, and any other device and system disclosed herein (such as surgical systems 1, 2, 50, 52, 14000, surgical hubs 6, 56, 5104, 14002, tracking system 14008, visualization system 8, 14010, patient monitoring device 14004, surgical device/instrument 14006, and AR device 66, 84).

Referring primarily to fig. 16 and also to fig. 12, the tracking system 14008 may detect 14402 the devices 14006, 66 according to the method 14400. The surgical hub 14002 can be wirelessly paired 14404 with the detected devices 14006, 66. The surgical hub 14002 can determine 14406 a first security level or a second security level of the wireless pairing, wherein the first security level is determined based on the tracking system 14008 detecting 14002 a device and the detected automatic wireless pairing of the device 14006, and wherein the second security level is determined based on the tracking system 14008 detecting 14002 a device and the detected manual pairing of the device. In addition, the surgical hub 14002 can authenticate 14408 the data stream transmitted by the detected device 14006, 66.

According to one aspect of method 14400, detecting 14002 the devices 14006, 66 by the tracking system may include identifying the devices 14006, 66 as OR being filled. In another aspect of the method 14400, detecting 14002 the device 14006, 66 by the tracking system may include scanning a QR code included on a package of the device 14006, 66. In yet another aspect of the method 14400, detecting 14002 the device 14006, 66 by the tracking system may include scanning a QR code included on the device 14006, 66.

QR code system for confirming data transmission accuracy and delay in connected OR devices

As described above, as devices and appliances gain the ability to make connections within a digital OR ecosystem, concerns over data integrity (e.g., corrupted data transmitted by the device) become more necessary. Further, in some aspects, successful pairing of the device with the surgical hub does not preclude the surgical hub from receiving data from the device that may be erroneous, corrupted, and/or delayed. Simple methods such as data checksums may be employed to help ensure data authenticity and/or integrity. However, simple methods may provide false sensations of data authenticity and integrity. For example, different arrangements of bytes in a packet may generate the same checksum value. Thus, even if the checksum value appears to be authentic, the received data may be in fact defective, corrupted, and so on. Accordingly, there is a need for apparatus, systems, and methods for ensuring data authenticity and/or integrity after initial device pairing.

Disclosed herein are apparatuses, systems, and methods for ensuring data authenticity and/or integrity after initial device pairing. The apparatus, systems, and methods may employ a two-part method that includes (i) securely initially pairing the device to the surgical hub based on the QR code, and (ii) subsequently authenticating the pairing based on data transmitted by the device to the surgical hub at the time of the initial pairing.

Fig. 17 illustrates a method 14500 for ensuring data authenticity and/or integrity after initial device pairing in accordance with several non-limiting aspects of the present disclosure. The method 14500 may be implemented by any combination of a surgical system, surgical hub, tracking system, visualization system, patient monitoring device, surgical device/instrument, AR device, any component thereof, and any other device and system disclosed herein (such as surgical system 1, 2, 50, 52, 14000, surgical hub 6, 56, 5104, 14002, tracking system 14008, visualization system 8, 14010, patient monitoring device 14004, surgical device/instrument 14006, and AR device 66, 84).

Referring primarily to fig. 17 and also to fig. 12, according to the method 14500, a 14502QR code may be provided on the device 14006, 66 (e.g., on the device package, on the device itself). The QR code may include information such as a Media Access Control (MAC) address and first unique authentication data that may be used to uniquely identify the device 14006, 66. A component of the surgical system 14000, such as the surgical hub 14002 or tracking system 14008, may scan 14504 the QR code to retrieve the MAC address and the first unique authentication data. Further, the surgical hub 14002 can be wirelessly paired 14506 with the devices 14006, 66 when the devices are powered on. After pairing 14506, the devices 14006, 66 can transmit 14508 second unique authentication data to the surgical hub 14002. The surgical hub 14002 can compare 14510 the first unique authentication data to the second authentication data.

Still referring primarily to fig. 17 and also to fig. 12, the surgical hub 14002 may be configured to determine 14512 various comparison results based on the comparison 14510. Based further on the determination 14512 comparison, the surgical hub 14002 can cause various actions 14514. In one aspect, the surgical hub can determine 14516 that the second unique authentication data is accurate and received within a transmission time threshold. Thus, the surgical hub 14002 can be configured to notify 14518 the user of the device 14006, 66 that the device 14006, 66 has been authenticated (e.g., via superposition on the AR device 66) and to allow the user to continue using the device 14006, 66.

In another aspect, the surgical hub can determine 14520 that the second unique authentication data is accurate but not received within the transmission time threshold. Thus, the surgical hub 14002 can be configured to alert 14522 the user of the device 14006, 66 to a poor transmission rate (e.g., via superposition on the AR device 66).

In another aspect, the surgical hub may determine 14524 that the second unique authentication data is inaccurate. Thus, the surgical hub 14002 can be configured to alert 14526 the user of the device 14006, 66 that the device 14006, 66 has not been authenticated (e.g., via superposition on the AR device 66) and lock the device 14006, 66.

In another aspect, the surgical hub can determine 14528 that the second unique authentication data has not been received. Thus, the surgical hub 14002 can be configured to alert 14530 users of the devices 14006, 66 that the devices 14006, 66 have not been authenticated (e.g., via superposition on the AR device 66) and lock the devices 14006, 66.

Various additional aspects of the subject matter described herein are set forth in the following numbered embodiments:

example 1: a method for displaying an interactive overlay for a plurality of users of a surgical system, the method comprising: receiving, by a surgical hub, a plurality of data streams related to a surgical procedure; communicatively coupling a first augmented reality display device to a surgical hub; linking, by the surgical hub, the first augmented reality display device to a first user; and displaying, by the first augmented reality display device, a first interactive overlay customized for the first user based on at least one of the plurality of data streams.

Example 2: the method of embodiment 1, further comprising: communicatively coupling a second augmented reality display device to the surgical hub; linking, by the surgical hub, the second augmented reality display device to a second user; and displaying, by the second augmented reality display device, a second interactive overlay customized for the second user based on at least one of the plurality of data streams.

Example 3: the method of any of embodiments 1-2, further comprising simultaneously displaying the first interactive overlay by the first augmented reality display device and the second interactive overlay by the second augmented reality display device; wherein the first interactive superposition and the second interactive superposition are based on a same data stream of the plurality of data streams.

Example 4: the method of any one of embodiments 1 to 3, further comprising: the first interactive overlay and the second interactive overlay are displayed differently for the first user and the second user based on respective preferences of the first user and the second user.

Example 5: the method of any of embodiments 1-4, displaying, by the first augmented reality display device, the first interactive overlay based on a first data stream associated with a first surgical instrument while the first user is using the first surgical instrument; displaying, by the second augmented reality display device, the second interactive overlay based on a second data stream associated with a second surgical instrument while the second user is using the second surgical instrument; and updating, by the first augmented reality display device, the first interactive overlay to be based on the second data stream when the first user is using the second surgical instrument; wherein the plurality of data streams includes the first data stream and the second data stream.

Example 6: the method of any of embodiments 1-5, transmitting, by a first surgical instrument, a first surgical instrument data stream to the surgical hub, wherein the first surgical instrument data stream is one of the plurality of data streams related to the surgical procedure; detecting, by the surgical hub, an error associated with the first surgical instrument data stream; and causing, by the surgical hub, the first interactive data overlay to include an alert based on the detected error exceeding a risk level threshold.

Example 7: the method of any of embodiments 1-6, wherein the risk level threshold is based on a frequency of detected errors, detected errors exceeding an expected range, a loss of communication with the first surgical instrument, or a combination thereof.

Example 8: the method of any of embodiments 1-7, further comprising causing, by the surgical hub, locking of the first surgical instrument based on the detected error exceeding the risk level threshold.

Example 9: the method of any one of embodiments 1-8, automatically pairing a surgical device to the surgical hub by the surgical hub to achieve a first level of security, or manually pairing the surgical device to the surgical hub by a user to achieve a second level of security; and authenticating, by the surgical hub, a data stream transmitted between the surgical device and the surgical hub; wherein the data stream is one of the plurality of data streams.

Example 10: the method of any of embodiments 1-9, scanning, by the surgical hub, a Quick Response (QR) code of a surgical device to receive first device authentication data; transmitting, by the surgical device, second device authentication data to the surgical hub, wherein the second device authentication data is one of the plurality of data streams; comparing, by the surgical hub, the first device authentication data with the second device authentication data; and displaying, by the first augmented reality display, a notification based on the comparison of the first device authentication data and the second device authentication data.

Example 11: a surgical system for displaying interactive overlays for a plurality of users, the system comprising: a surgical hub configured to receive a plurality of data streams related to a surgical procedure; and a first augmented reality display device communicatively coupled to the surgical hub, wherein the first augmented reality display device is linked to a first user; wherein the first augmented reality display device displays a first interactive overlay customized for the first user based on at least one of the plurality of data streams.

Example 12: the system of embodiment 11, further comprising a second augmented reality display device communicatively coupled to the surgical hub, wherein the second augmented reality display device is linked to a second user; wherein the second augmented reality display device displays a second interactive overlay customized for the second user based on at least one of the plurality of data streams.

Example 13: the system of any of embodiments 11-12, wherein the first interactive overlay and the second interactive overlay are based on a same data stream of the plurality of data streams; and wherein the first and second augmented reality display devices simultaneously display the first and second interactive overlays.

Example 14: the method of any of embodiments 11-13, wherein the first interactive overlay and the second interactive overlay are displayed differently for the first user and the second user based on respective preferences of the first user and the second user.

Example 15: the system of any of embodiments 11-14, wherein the first interactive overlay is based on a first data stream associated with a first surgical instrument when the first user is using the first surgical instrument; wherein the second interactive overlay is based on a second data stream associated with a second surgical instrument when the second user is using the second surgical instrument; wherein the first interactive superposition is updated to be based on the second data stream when the first user is using the second surgical instrument; and wherein the plurality of data streams includes the first data stream and the second data stream.

Example 16: the system of any of embodiments 11-15, the first surgical instrument configured to transmit a first surgical instrument data stream to the surgical hub, wherein the first surgical instrument data stream is one of the plurality of data streams related to the surgical procedure; wherein the surgical hub is configured to detect errors associated with the first surgical instrument data stream; and wherein the surgical hub is configured to cause the first interactive data overlay to include a notification based on the detected error exceeding a risk level threshold.

Example 17: the system of any of embodiments 11-16, wherein the risk level threshold is based on a frequency of detected errors, detected errors exceeding an expected range, a loss of communication with the first surgical instrument, or a combination thereof.

Example 18: the system of any of embodiments 11-17, wherein the surgical hub is configured to cause locking of the first surgical instrument based on the detected error exceeding the risk level threshold.

Example 19: the system of any one of embodiments 11-18, further comprising a surgical device configured as a surgical device matable with the surgical hub; wherein automatically pairing the surgical device to the surgical hub achieves a first level of security; wherein manually pairing the surgical device to the surgical hub achieves a second level of security; wherein the surgical hub is configured to authenticate a data stream transmitted between the surgical device and the surgical hub; and wherein the data stream is one of the plurality of data streams.

Example 20: the system of any of embodiments 11-19, further comprising a surgical device having a Quick Response (QR) code, the surgical device configured to provide first device authentication data to the surgical hub; wherein the surgical hub is configured to scan the QR code to receive second device authentication data; wherein the surgical hub is configured to compare the first device authentication data with the second device authentication data; wherein the first augmented reality display is configured to be capable of displaying a notification based on the first device authentication data and the second device authentication data; and wherein the first device authentication data is one of the plurality of data streams.

While various forms have been illustrated and described, it is not the intention of the applicant to restrict or limit the scope of the appended claims to such detail. Many modifications, variations, changes, substitutions, combinations, and equivalents of these forms may be made by one skilled in the art without departing from the scope of the disclosure. Furthermore, the structure of each element associated with the described form may alternatively be described as a means for providing the function performed by the element. In addition, where materials for certain components are disclosed, other materials may be used. It is, therefore, to be understood that the foregoing detailed description and the appended claims are intended to cover all such modifications, combinations, and variations as fall within the scope of the disclosed forms of the invention. The appended claims are intended to cover all such modifications, variations, changes, substitutions, modifications and equivalents.

The foregoing detailed description has set forth various forms of the apparatus and/or methods via the use of block diagrams, flowcharts, and/or examples. Insofar as such block diagrams, flowcharts, and/or examples contain one or more functions and/or operations, it will be understood by those within the art that each function and/or operation within such block diagrams, flowcharts, or examples can be implemented, individually and/or collectively, by a wide range of hardware, software, firmware, or virtually any combination thereof. Those skilled in the art will recognize that some aspects of the forms disclosed herein may be equivalently implemented in integrated circuits, as one or more computer programs running on one or more computers (e.g., as one or more programs running on one or more computer systems), as one or more programs running on one or more processors (e.g., as one or more programs running on one or more microprocessors), as firmware, or as virtually any combination thereof, and that designing the circuitry and/or writing the code for the software and/or hardware would be well within the skill of one of skill in the art in light of this disclosure. In addition, those skilled in the art will appreciate that the mechanisms of the subject matter described herein are capable of being distributed as a program product or products in a variety of forms, and that an illustrative form of the subject matter described herein applies regardless of the particular type of signal bearing media used to actually carry out the distribution.

Instructions for programming logic to perform the various disclosed aspects can be stored within a memory in a system, such as Dynamic Random Access Memory (DRAM), cache, flash memory, or other memory. Furthermore, the instructions may be distributed via a network or by other computer readable media. Thus, a machine-readable medium may include any mechanism for storing or transmitting information in a form readable by a machine (e.g., a computer), but is not limited to floppy diskettes, optical disks, compact disk read-only memories (CD-ROMs), and magneto-optical disks, read-only memories (ROMs), random Access Memories (RAMs), erasable programmable read-only memories (EPROMs), electrically erasable programmable read-only memories (EEPROMs), magnetic or optical cards, flash memory, or a tangible, machine-readable storage device for use in transmitting information over the internet via electrical, optical, acoustic, or other form of propagated signals (e.g., carrier waves, infrared signals, digital signals, etc.). Thus, a non-transitory computer-readable medium includes any type of tangible machine-readable medium suitable for storing or transmitting electronic instructions or information in a form readable by a machine (e.g., a computer).

As used in any aspect herein, the term "control circuitry" may refer to, for example, hardwired circuitry, programmable circuitry (e.g., a computer processor including one or more individual instruction processing cores, processing units, processors, microcontrollers, microcontroller units, controllers, digital Signal Processors (DSPs), programmable Logic Devices (PLDs), programmable Logic Arrays (PLAs), field Programmable Gate Arrays (FPGAs)), state machine circuitry, firmware storing instructions executed by the programmable circuitry, and any combination thereof. The control circuitry may be implemented collectively or individually as circuitry forming part of a larger system, such as an Integrated Circuit (IC), an Application Specific Integrated Circuit (ASIC), a system-on-a-chip (SoC), a desktop computer, a laptop computer, a tablet computer, a server, a smart phone, or the like. Thus, as used herein, "control circuitry" includes, but is not limited to, electronic circuitry having at least one discrete circuit, electronic circuitry having at least one integrated circuit, electronic circuitry having at least one application specific integrated circuit, electronic circuitry forming a general purpose computing device configured by a computer program (e.g., a general purpose computer configured by a computer program that at least partially implements the methods and/or apparatus described herein, or a microprocessor configured by a computer program that at least partially implements the methods and/or apparatus described herein), electronic circuitry forming a memory device (e.g., forming a random access memory), and/or electronic circuitry forming a communication device (e.g., a modem, communication switch, or optoelectronic device). Those skilled in the art will recognize that the subject matter described herein may be implemented in analog or digital fashion, or some combination thereof.

As used in any aspect herein, the term "logic" may refer to an application, software, firmware, and/or circuitry configured to be capable of performing any of the foregoing operations. The software may be embodied as software packages, code, instructions, instruction sets, and/or data recorded on a non-transitory computer readable storage medium. The firmware may be embodied as code, instructions or a set of instructions and/or data that are hard-coded (e.g., non-volatile) in a memory device.

As used in any aspect herein, the terms "component," "system," "module," and the like can refer to a control circuit, a computer-related entity, hardware, a combination of hardware and software, or software in execution.

As used in any aspect herein, an "algorithm" refers to an organized sequence of steps leading to a desired result, wherein "step" refers to the manipulation of physical quantities and/or logical states, which may, but need not, take the form of electrical or magnetic signals capable of being stored, transferred, combined, compared, and otherwise manipulated. Are often used to refer to signals such as bits, values, elements, symbols, characters, terms, numbers, or the like. These and similar terms may be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities and/or conditions.

The network may comprise a packet switched network. The communication devices may be capable of communicating with each other using the selected packet switched network communication protocol. One exemplary communication protocol may include an ethernet communication protocol that may be capable of allowing communication using transmission control protocol/internet protocol (TCP/IP). The ethernet protocol may conform to or be compatible with the ethernet Standard and/or a higher version of the Standard, titled "IEEE 802.3Standard" published by the Institute of Electrical and Electronics Engineers (IEEE) at month 12 of 2008. Alternatively or additionally, the communication devices may be capable of communicating with each other using an x.25 communication protocol. The x.25 communication protocol may conform to or be compatible with standards promulgated by the international telecommunications union telecommunication standardization sector (ITU-T). Alternatively or additionally, the communication devices may be capable of communicating with each other using a frame relay communication protocol. The frame relay communication protocol may conform to or be compatible with standards promulgated by the international telegraph and telephone Consultation Committee (CCITT) and/or the American National Standards Institute (ANSI). Alternatively or additionally, the transceivers may be capable of communicating with each other using an Asynchronous Transfer Mode (ATM) communication protocol. The ATM communication protocol may conform to or be compatible with the ATM standard promulgated by the ATM forum at month 8 of 2001 under the name "ATM-MPLS Network Interworking 2.0" and/or a higher version of the standard. Of course, different and/or later developed connection oriented network communication protocols are likewise contemplated herein.

Unless specifically stated otherwise as apparent from the above disclosure, it is appreciated that throughout the above disclosure, discussions utilizing terms such as "processing," "computing," "calculating," "determining," "displaying" or the like, refer to the action and processes of a computer system, or similar electronic computing device, that manipulates and transforms data represented as physical (electronic) quantities within the computer system's registers and memories into other data similarly represented as physical quantities within the computer system memories or registers or other such information storage, transmission or display devices.

One or more components may be referred to herein as "configured to be capable of", "operable/operative", "adapted/adaptable", "capable of", "conformable/conforming" and the like. Those skilled in the art will recognize that "configured to be capable of" may generally encompass active and/or inactive and/or standby components unless the context indicates otherwise.

The terms "proximal" and "distal" are used herein with respect to a clinician manipulating a handle portion of a surgical instrument. The term "proximal" refers to the portion closest to the clinician, and the term "distal" refers to the portion located away from the clinician. It will also be appreciated that for simplicity and clarity, spatial terms such as "vertical," "horizontal," "upper," and "lower" may be used herein with respect to the drawings. However, surgical instruments are used in many orientations and positions, and these terms are not intended to be limiting and/or absolute.

Those skilled in the art will recognize that, in general, terms used herein, and particularly in the appended claims (e.g., bodies of the appended claims) are generally intended to be "open" terms (e.g., the term "including" should be construed as "including but not limited to," the term "having" should be construed as "having at least," the term "comprising" should be construed as "including but not limited to," etc.). It will be further understood by those with skill in the art that if a specific number of an introduced claim recitation is intended, such an intent will be explicitly recited in the claim, and in the absence of such recitation no such intent is present. For example, as an aid to understanding, the following appended claims may contain usage of the introductory phrases "at least one" and "one or more" to introduce claim(s). However, the use of such phrases should not be construed to imply that the introduction of a claim recitation by the indefinite articles "a" or "an" limits any particular claim containing such introduced claim recitation to claims containing only one such recitation, even when the same claim includes the introductory phrases "one or more" or "at least one" and indefinite articles such as "a" or "an" (e.g., "a" and/or "an" should typically be interpreted to mean "at least one" or "one or more"); the same holds true for the use of definite articles used to introduce claim recitations.

In addition, even if a specific number of an introduced claim recitation is explicitly recited, those skilled in the art will recognize that such recitation should typically be interpreted to mean at least the recited number (e.g., the bare recitation of "two recitations," without other modifiers, typically means at least two recitations, or two or more recitations). Moreover, in those instances where a convention analogous to "at least one of A, B and C, etc." is used, in general such a construction is intended in the sense one having skill in the art would understand the convention (e.g., "a system having at least one of A, B and C" would include but not be limited to systems having only a, only B, only C, A and B together, a and C together, B and C together, and/or A, B and C together, etc.). In those instances where a convention analogous to "A, B or at least one of C, etc." is used, in general such a construction is intended in the sense one having skill in the art would understand the convention (e.g., "a system having at least one of A, B or C" shall include but not be limited to systems having only a, only B, only C, A and B together, a and C together, B and C together, and/or A, B and C together, etc.). It will be further understood by those within the art that, in general, unless the context indicates otherwise, disjunctive words and/or phrases presenting two or more alternative terms in the detailed description, claims, or drawings should be understood to encompass the possibility of including one of the terms, either of the terms, or both. For example, the phrase "a or B" will generally be understood to include the possibility of "a" or "B" or "a and B".

For the purposes of the appended claims, those skilled in the art will understand that the operations recited therein can generally be performed in any order. Additionally, while various operational flow diagrams are set forth in one or more sequences, it should be understood that various operations may be performed in other sequences than the illustrated sequences, or may be performed concurrently. Examples of such alternative ordering may include overlapping, staggered, interrupted, reordered, incremental, preparatory, supplemental, simultaneous, reverse, or other altered ordering unless the context dictates otherwise. Moreover, unless the context dictates otherwise, terms such as "responsive to," "related to," or other past-type adjectives are generally not intended to exclude such variants.

It is worth mentioning that any reference to "an aspect", "an example" means that a particular feature, structure or characteristic described in connection with the aspect is included in at least one aspect. Thus, the appearances of the phrases "in one aspect," "in an example," and "in an example" in various places throughout this specification are not necessarily all referring to the same aspect. Furthermore, the particular features, structures, or characteristics may be combined in any suitable manner in one or more aspects.

Any patent application, patent, non-patent publication, or other disclosure material referred to in this specification and/or listed in any application data sheet is incorporated herein by reference, as if the incorporated material was not inconsistent herewith. Accordingly, and to the extent necessary, the disclosure as explicitly set forth herein supersedes any conflicting material incorporated herein by reference. Any material, or portion thereof, that is said to be incorporated by reference herein, but which conflicts with existing definitions, statements, or other disclosure material set forth herein will only be incorporated to the extent that no conflict arises between that incorporated material and the existing disclosure material.

In summary, many of the benefits resulting from employing the concepts described herein have been described. The foregoing detailed description of one or more forms has been presented for purposes of illustration and description. It is not intended to be exhaustive or to limit the invention to the precise form disclosed. Modifications or variations of the present invention are possible in light of the above teachings. One or more of the forms selected and described are chosen to illustrate principles and practical application to thereby enable one of ordinary skill in the art to utilize various forms and various modifications as are suited to the particular use contemplated. The claims filed herewith are intended to define the full scope.

Claims (20)

1. A method for displaying an interactive overlay for a plurality of users of a surgical system, the method comprising:

receiving, by a surgical hub, a plurality of data streams related to a surgical procedure;

communicatively coupling a first augmented reality display device to a surgical hub;

linking, by the surgical hub, the first augmented reality display device to a first user; and

a first interactive overlay customized for the first user is displayed by the first augmented reality display device based on at least one of the plurality of data streams.

2. The method of claim 1, further comprising:

communicatively coupling a second augmented reality display device to the surgical hub;

linking, by the surgical hub, the second augmented reality display device to a second user; and

a second interactive overlay customized for the second user is displayed by the second augmented reality display device based on at least one of the plurality of data streams.

3. The method of claim 2, further comprising simultaneously displaying the first interactive overlay by the first augmented reality display device and the second interactive overlay by the second augmented reality display device;

Wherein the first interactive superposition and the second interactive superposition are based on a same data stream of the plurality of data streams.

4. The method of claim 3, further comprising displaying the first interactive overlay and the second interactive overlay differently for the first user and the second user based on respective preferences of the first user and the second user.

5. The method of claim 2, further comprising:

displaying, by the first augmented reality display device, the first interactive overlay based on a first data stream associated with a first surgical instrument while the first user is using the first surgical instrument;

displaying, by the second augmented reality display device, the second interactive overlay based on a second data stream associated with a second surgical instrument while the second user is using the second surgical instrument; and

updating, by the first augmented reality display device, the first interactive overlay to be based on the second data stream while the first user is using the second surgical instrument;

wherein the plurality of data streams includes the first data stream and the second data stream.

6. The method of claim 1, further comprising:

transmitting, by a first surgical instrument, a first surgical instrument data stream to the surgical hub, wherein the first surgical instrument data stream is one of the plurality of data streams related to the surgical procedure;

detecting, by the surgical hub, an error associated with the first surgical instrument data stream; and

causing, by the surgical hub, the first interactive data overlay to include an alert based on the detected error exceeding a risk level threshold.

7. The method of claim 6, wherein the risk level threshold is based on a frequency of detected errors, detected errors exceeding an expected range, a loss of communication with the first surgical instrument, or a combination thereof.

8. The method of claim 7, further comprising causing, by the surgical hub, locking of the first surgical instrument based on the detected error exceeding the risk level threshold.

9. The method of claim 1, further comprising:

pairing a surgical device to the surgical hub automatically by the surgical hub to achieve a first level of security, or pairing the surgical device to the surgical hub manually by a user to achieve a second level of security; and

Authenticating, by the surgical hub, a data stream transmitted between the surgical device and the surgical hub;

wherein the data stream is one of the plurality of data streams.

10. The method of claim 1, further comprising:

scanning, by the surgical hub, a Quick Response (QR) code of a surgical device to receive first device authentication data;

transmitting, by the surgical device, second device authentication data to the surgical hub, wherein the second device authentication data is one of the plurality of data streams;

comparing, by the surgical hub, the first device authentication data with the second device authentication data; and

a notification is displayed by the first augmented reality display based on the comparison of the first device authentication data and the second device authentication data.

11. A surgical system for displaying interactive overlays for a plurality of users, the system comprising:

a surgical hub configured to receive a plurality of data streams related to a surgical procedure; and

a first augmented reality display device communicatively coupled to the surgical hub, wherein the first augmented reality display device is linked to a first user;

Wherein the first augmented reality display device displays a first interactive overlay customized for the first user based on at least one of the plurality of data streams.

12. The surgical system of claim 11, further comprising a second augmented reality display device communicatively coupled to the surgical hub, wherein the second augmented reality display device is linked to a second user;

wherein the second augmented reality display device displays a second interactive overlay customized for the second user based on at least one of the plurality of data streams.

13. The surgical system of claim 12, wherein the first interactive overlay and the second interactive overlay are based on a same data stream of the plurality of data streams; and is also provided with

Wherein the first and second augmented reality display devices simultaneously display the first and second interactive overlays.

14. The surgical system of claim 13, wherein the first interactive overlay and the second interactive overlay are displayed differently for the first user and the second user based on respective preferences of the first user and the second user.

15. The surgical system of claim 12, wherein the first interactive overlay is based on a first data stream associated with a first surgical instrument when the first user is using the first surgical instrument;

wherein the second interactive overlay is based on a second data stream associated with a second surgical instrument when the second user is using the second surgical instrument;

wherein the first interactive superposition is updated to be based on the second data stream when the first user is using the second surgical instrument; and is also provided with

Wherein the plurality of data streams includes the first data stream and the second data stream.

16. The surgical system of claim 11, further comprising:

a first surgical instrument configured to transmit a first surgical instrument data stream to the surgical hub, wherein the first surgical instrument data stream is one of the plurality of data streams related to the surgical procedure;

wherein the surgical hub is configured to detect errors associated with the first surgical instrument data stream; and is also provided with

Wherein the surgical hub is configured to cause the first interactive data overlay to include a notification based on the detected error exceeding a risk level threshold.

17. The surgical system of claim 16, wherein the risk level threshold is based on a frequency of detected errors, detected errors exceeding an expected range, a loss of communication with the first surgical instrument, or a combination thereof.

18. The surgical system of claim 16, wherein the surgical hub is configured to cause locking of the first surgical instrument based on the detected error exceeding the risk level threshold.

19. The system of claim 11, further comprising a surgical device configured to mate with the surgical hub;

wherein automatically pairing the surgical device to the surgical hub achieves a first level of security;

wherein manually pairing the surgical device to the surgical hub achieves a second level of security;

wherein the surgical hub is configured to authenticate a data stream transmitted between the surgical device and the surgical hub; and is also provided with

Wherein the data stream is one of the plurality of data streams.

20. The system of claim 11, further comprising a surgical device having a Quick Response (QR) code, the surgical device configured to provide first device authentication data to the surgical hub;

Wherein the surgical hub is configured to scan the QR code to receive second device authentication data;

wherein the surgical hub is configured to compare the first device authentication data with the second device authentication data;

wherein the first augmented reality display is configured to be capable of displaying a notification based on the first device authentication data and the second device authentication data; and is also provided with

Wherein the first device authentication data is one of the plurality of data streams.