CN118160044A - Customization of overlay data and configuration - Google Patents

Abstract

The present invention relates to a method of distributing information among members of a surgical team, the method may include: receiving, by a modular control tower, imaging data from a plurality of imaging devices; receiving, by the modular control tower, device-related data from each of a plurality of intelligent surgical instruments; associating, by the modular control tower, a display device with a member of the surgical team; defining, by the modular control tower, a functional role of the member of the surgical team; and displaying, by the modular control tower, an augmented reality display via the display device. The augmented reality display on the display device may include a virtual object based on the imaging data, the device-related data, the functional role of the member of the surgical team, and the surgical activity of the member of the surgical team. An interactive surgical system may implement the method.

Description

Customization of overlay data and configuration

Cross Reference to Related Applications

The present application is in accordance with 35U.S. c. ≡119 (e) claims that U.S. provisional patent application No. 63/174,674 entitled "HEADS UP DISPLAY" filed on day 4, month 14 of 2021 and U.S. provisional patent application No. 63/284,326 entitled "INTRAOPERATIVE DISPLAY FOR SURGICAL SYSTEMS" filed on day 11, month 30 of 2021, the disclosures of each of which are incorporated herein by reference in their entirety.

Background

The present disclosure relates to devices, systems, and methods for providing an augmented reality interactive experience during a surgical procedure. During a surgical procedure, it is desirable to provide an augmented reality interactive experience of a real-world environment in which objects residing in the real world are enhanced by superimposing computer-generated sensory information (sometimes across multiple sensory modalities, including visual, auditory, haptic, somatosensory, and olfactory). In the context of the present disclosure, the surgical field and the images of surgical instruments and other objects present in the surgical field are enhanced by superimposing computer-generated visual, auditory, tactile, somatosensory, olfactory, or other sensory information over the surgical field and the real world images of instruments or other objects present in the surgical field. The image may be streamed in real-time, or may be a still image.

Real world surgical instruments include a variety of surgical devices including energy, staplers, or a combination of energy and staplers. Energy-based medical devices include, but are not limited to, radio Frequency (RF) based monopolar and bipolar electrosurgical instruments, ultrasonic surgical instruments, combined RF electrosurgical and ultrasonic instruments, combined RF electrosurgical and mechanical staplers, and the like. Surgical stapler devices are surgical instruments used to cut and staple tissue in a variety of surgical procedures including weight loss, breast, colorectal, gynecological, urological, and general surgery.

Disclosure of Invention

In various examples, the present disclosure provides a method of distributing information among members of a surgical team, the method comprising: receiving, by a modular control tower, imaging data from a plurality of imaging devices; receiving, by the modular control tower, device-related data from each of a plurality of intelligent surgical instruments; associating, by the modular control tower, a display device with a member of the surgical team; defining, by the modular control tower, a functional role of the member of the surgical team; and displaying, by the modular control tower, an augmented reality display via the display device. The augmented reality display on the display device includes a virtual object based on the imaging data, the device-related data, the functional role of the member of the surgical team, and the surgical activity of the member of the surgical team.

In another example, the present disclosure provides an automated surgical system comprising: a modular control tower; a plurality of imaging devices in data communication with the modular control tower; a plurality of intelligent surgical instruments; and a plurality of display devices in data communication with the modular control tower. Each of the plurality of display devices is associated with one or more members of a surgical team through the modular control tower, and each of the one or more members of the surgical team is defined by a functional role. The modular control tower includes a controller in data communication with one or more memory components configured to be capable of storing instructions that, when executed by the controller, cause the controller to: receiving imaging data from the plurality of imaging devices; receiving device-related data from each of the plurality of intelligent surgical instruments; and displaying the augmented reality display on each of the plurality of display devices. The augmented reality display on a designated display device may include a virtual object based on the imaging data, the device-related data, the functional role of a designated member of the surgical team associated with the designated display device, and a surgical activity of the designated member of the surgical team.

Drawings

The various aspects (both as to the surgical organization and method) described herein, together with further objects and advantages thereof, may best be understood by reference to the following description taken in connection with the accompanying drawings.

Fig. 1 is a block diagram of a computer-implemented interactive surgical system according to one aspect of the present disclosure.

Fig. 2 is a surgical system for performing a surgical procedure in an operating room according to one aspect of the present disclosure.

Fig. 3 is a surgical hub paired with a visualization system, robotic system, and intelligent instrument, according to one aspect of the present disclosure.

Fig. 4 illustrates a surgical data network including a modular communication hub configured to enable connection of modular devices located in one or more operating rooms of a medical facility or any room specially equipped for surgical procedures in the medical facility to a cloud, according to one aspect of the present disclosure.

Fig. 5 illustrates a computer-implemented interactive surgical system in accordance with an aspect of the present disclosure.

Fig. 6 illustrates a surgical hub including a plurality of modules coupled to a modular control tower according to one aspect of the present disclosure.

Fig. 7 illustrates an Augmented Reality (AR) system including an intermediate signal combiner positioned in a communication path between an imaging module and a surgical hub display, according to one aspect of the present disclosure.

Fig. 8 illustrates an Augmented Reality (AR) system including an intermediate signal combiner positioned in a communication path between an imaging module and a surgical hub display, according to one aspect of the present disclosure.

Fig. 9 illustrates an Augmented Reality (AR) device worn by a surgeon to transmit data to a surgical hub, according to one aspect of the present disclosure.

Fig. 10 illustrates a system for augmenting surgical instrument information using an augmented reality display according to one aspect of the present disclosure.

Fig. 11 illustrates a timeline of a situational awareness surgical procedure in accordance with an aspect of the present disclosure.

Fig. 12A illustrates a surgical display obtained from a surgical imaging device during a laparoscopic sleeve gastrectomy procedure in accordance with an aspect of the present disclosure.

Fig. 12B illustrates an augmented reality image including a surgical display and a virtual secondary view superimposed on the surgical display, according to one aspect of the present disclosure.

Fig. 13A illustrates a surgical display obtained from a surgical imaging device during a laparoscopic sleeve gastrectomy procedure with a portion of the fundus of a patient removed, according to one aspect of the present disclosure.

Fig. 13B illustrates an augmented reality image including a surgical display and a predicted or recommended placement of a stapler, according to one aspect of the disclosure.

FIG. 14 illustrates a flow chart depicting a method by which an interactive surgical system may receive surgical procedure information and suggest the following procedure steps, in accordance with an aspect of the present disclosure.

Fig. 15A illustrates a surgical display obtained from a surgical imaging device during a laparoscopic procedure in accordance with an aspect of the present disclosure.

Fig. 15B illustrates an augmented reality image including a surgical display and an augmented reality virtual object presenting an outline of a ureter beneath tissue to be resected, according to one aspect of the present disclosure.

FIG. 16 illustrates various aspects associated with a surgical procedure that can be tracked by an interactive surgical system and analyzed to formulate an optimization strategy in accordance with an aspect of the present disclosure.

Fig. 17 illustrates aspects of an operating room that can be modeled for tracking purposes in accordance with an aspect of the present disclosure.

Fig. 18 illustrates an example of virtual objection warning of surgical member fatigue in accordance with an aspect of the present disclosure.

Fig. 19A, 19B, and 19C illustrate some exemplary depictions of an optimized operating room according to one aspect of the present disclosure.

Corresponding reference characters indicate corresponding parts throughout the several views. The exemplifications set out herein illustrate various disclosed embodiments, in one form, and such exemplifications are not to be construed as limiting the scope of the invention in any manner.

Detailed Description

The applicant of the present application owns the following U.S. patent applications filed concurrently herewith, the disclosure of each of these patent applications being incorporated herein by reference in its entirety:

U.S. patent application entitled "METHOD FOR INTRAOPERATIVE DISPLAY FOR SURGICAL SYSTEMS", attorney docket number END9352USNP 1/210120-1M;

U.S. patent application Ser. No. "UTILIZATION OF SURGICAL DATA VALUES AND SITUATIONAL AWARENESS TO CONTROL THE OVERLAY IN SURGICAL FIELDVIEW", attorney docket number END9352USNP 2/210120-2;

U.S. patent application entitled "SELECTIVE AND ADJUSTABLE MIXED REALITY OVERLAY IN SURGICAL FIELDVIEW", attorney docket number END9352USNP 3/210120-3;

U.S. patent application entitled "RISK BASED PRIORITIZATION OF DISPLAY ASPECTS IN SURGICAL FIELDVIEW", attorney docket number END9352USNP 4/210120-4;

U.S. patent application entitled "SYSTEMS AND METHODS FOR CONTROLLING SURGICAL DATA OVERLAY", attorney docket number END9352USNP 5/210120-5;

U.S. patent application entitled "SYSTEMS AND METHODS FOR CHANGING DISPLAY OVERLAY OF SURGICAL FIELDVIEW BASED ON TRIGGERING EVENTS", attorney docket number END9352USNP 6/210120-6;

U.S. patent application entitled "INDICATION OF THE COUPLE PAIR OF REMOTE CONTROLS WITH REMOTE DEVICES FUNCTIONS", attorney docket number END9352USNP 8/210120-8;

U.S. patent application entitled "COOPERATIVE OVERLAYS OF INTERACTING INSTRUMENTS WHICH RESULT IN BOTH OVERLAYS BEING EFFECTED", attorney docket number END9352USNP 9/210120-9;

U.S. patent application entitled "ANTICIPATION OF INTERACTIVE UTILIZATION OF COMMON DATA OVERLAYS BY DIFFERENT USERS", attorney docket number END9352USNP 10/210120-10;

U.S. patent application Ser. No. "MIXING DIRECTLY VISUALIZED WITH RENDERED ELEMENTS TO DISPLAY BLENDED ELEMENTS AND ACTIONS HAPPENING ON-SCREEN AND OFF-SCREEN", attorney docket number END9352USNP 11/210120-11;

Named "SYSTEM AND METHOD FOR TRACKING A PORTION OF THE USER AS A PROXY FOR NON

MONITORED INSTRUMENT ", attorney docket number END9352 USNP/210120-12;

U.S. patent application Ser. No. "UTILIZING CONTEXTUAL PARAMETERS OF ONE OR MORE SURGICAL DEVICES TO PREDICT A FREQUENCY INTERVAL FOR DISPLAYING SURGICAL INFORMATION", attorney docket number END9352USNP 13/210120-13;

U.S. patent application entitled "COOPERATION AMONG MULTIPLE DISPLAY SYSTEMS TO provider A HEALTHCARE USER CUSTOMIZED INFORMATION", attorney docket No. END9352 USNP/210120-14;

U.S. patent application entitled "INTRAOPERATIVE DISPLAY FOR SURGICAL SYSTEMS", attorney docket number END9352USNP 15/210120-15;

U.S. patent application entitled "ADAPTATION AND ADJUSTABILITY OR OVERLAID INSTRUMENT INFORMATION FOR SURGICAL SYSTEMS", attorney docket number END9352USNP 16/210120-16; and

U.S. patent application Ser. No. "MIXED REALITY FEEDBACK SYSTEMS THAT COOPERATE TO INCREASE EFFICIENT PERCEPTION OF COMPLEX DATA FEEDS", attorney docket No. END9352USNP 17/210120-17.

The applicant of the present application owns the following U.S. patent applications, the disclosure of each of which is incorporated herein by reference in its entirety:

U.S. patent application Ser. No. 16/209,423, entitled "METHOD OF COMPRESSING TISSUE WITHIN A STAPLING DEVICE AND SIMULTANEOUSLY DISPLAYING THE LOCATION OF THE TISSUE WITHIN THE JAWS", now U.S. patent application publication No. US-2019-0200981-A1;

U.S. patent application Ser. No. 16/209,453, entitled "METHOD FOR CONTROLLING SMART ENERGY DEVICES," now U.S. patent application publication No. US-2019-0201046-A1.

During a surgical procedure, a group of medical professionals (surgical team) collectively provide interventional medical care to a patient in a surgical suite or operating room. According to the surgical procedure, each member of the surgical team is assigned one or more specific roles or functions that are performed together in an effort to provide health care to the patient, producing beneficial results. Each member of the surgical team may be assigned to use one or more pieces of medical equipment that function during the procedure. The steps and procedures involved in surgical procedures can be complex and, taken together, take a significant amount of time to perform. In addition, all actions of members of the surgical team must be coordinated to achieve a successful outcome. Coordination of such efforts requires the correct sharing of information throughout the surgical procedure. It will be appreciated that the total amount of information and data is vast, and that they may vary significantly over time. Thus, sharing all potential surgical information (patient vital sign data, anesthesia data, data related to the operation of each intelligent surgical device, etc.) may result in members of the surgical team being inundated with false information. Surgical team members need to know data about their specific roles and functions at any time throughout the surgical procedure without being affected by additional data unrelated to their functions.

Disclosed herein is a method and system for using an augmented reality-based surgical system to distribute data to each member of a surgical team based on the specific roles or functions of each member of the surgical team throughout a surgical procedure.

Before explaining aspects of the surgical device and generator in detail, it should be noted that the exemplary embodiment is not limited in its application or use to the details of construction and arrangement of parts illustrated in the accompanying drawings and description. The illustrative examples may be implemented alone or in combination with other aspects, variations and modifications and may be practiced or carried out in various ways. Furthermore, unless otherwise indicated, the terms and expressions employed herein have been chosen for the purpose of describing the illustrative examples for the convenience of the reader and are not for the purpose of limitation. Moreover, it is to be understood that the expression of one or more of the aspects, and/or examples described below may be combined with any one or more of the expression of other aspects, and/or examples described below.

Various aspects relate to a screen display for a surgical system for various energy and surgical stapler-based medical devices. Energy-based medical devices include, but are not limited to, radio Frequency (RF) based monopolar and bipolar electrosurgical instruments, ultrasonic surgical instruments, combined RF electrosurgical and ultrasonic instruments, combined RF electrosurgical and mechanical staplers, and the like. The surgical stapler device includes a combination surgical stapler having an electrosurgical device and/or an ultrasonic device. Aspects of the ultrasonic surgical device may be configured to transect and/or coagulate tissue, for example, during a surgical procedure. Aspects of the electrosurgical device may be configured for transecting, coagulating, sealing, welding, and/or desiccating tissue, for example, during a surgical procedure. Aspects of the surgical stapler device can be configured to transect and staple tissue during a surgical procedure, and in some aspects, the surgical stapler device can be configured to deliver RF energy to tissue during a surgical procedure. The electrosurgical device is configured to deliver therapeutic and/or non-therapeutic RF energy to tissue. The elements of the surgical stapler device, electrosurgical device, and ultrasonic device may be used in combination in a single surgical instrument.

In various aspects, the present disclosure provides an on-screen display of real-time information to an OR team during a surgical procedure. In accordance with various aspects of the present disclosure, a number of new and unique screen displays are provided to display various visual information feedback to an OR team on a screen. In accordance with the present disclosure, visual information may include one or more various visual media, whether audible or silent. Generally, visual information includes still photography, movie photography, video or audio recordings, graphic arts, visual aids, models, displays, visual presentation services, and supporting processes. Visual information may be conveyed on any number of display options, such as, for example, a main OR screen, the energy OR surgical stapler device itself, a tablet computer, augmented reality glasses, and the like.

In various aspects, the present disclosure provides a large number of possible lists of options to communicate visual information to an OR team in real-time without providing excessive visual information to the OR team. For example, in various aspects, the present disclosure provides a screen display of visual information to enable a surgeon OR other member of an OR team to selectively activate the screen display, such as an icon surrounding a screen option, to manage a large amount of visual information. The active display may be determined using one or a combination of factors, which may include an energy-based (e.g., electrosurgical, ultrasound) or mechanical-based (e.g., stapler) surgical device in use, estimating the risk associated with a given display, the degree of experience of the surgeon, the surgeon's choice, and so forth. In other aspects, the visual information may include a large amount of data superimposed or overlaid into the surgical field to manage the visual information. In various aspects described below, overlapping images that require video analysis and tracking are included in order to properly overlay data. In contrast to static icons, visual information data transmitted in this manner may provide additional useful visual information to the OR team in a more concise and easily understood manner.

In various aspects, the present disclosure provides techniques for selectively activating a screen display, such as an icon surrounding the screen, to manage visual information during a surgical procedure. In other aspects, the present disclosure provides techniques for determining an active display using one or a combination of various factors. In various aspects, techniques according to the present disclosure may include selecting an energy-based OR mechanical-based surgical device for use as an active display, estimating risk associated with a given display, utilizing the experience level of a surgeon OR OR team making the selection, and so forth.

In other aspects, techniques according to the present disclosure may include overlaying or overlaying a large amount of data onto a surgical field of view to manage visual information. The various display arrangements described in this disclosure relate to superimposing various visual representations of surgical data on a live stream of a surgical field of view. As used herein, the term overlay includes semi-transparent overlays, partial overlays, and/or moving overlays. The graphic overlay may be in the form of a transparent graphic, a translucent graphic, or an opaque graphic, or a combination of transparent, translucent, and opaque elements or effects. Further, the superimposed layers may be positioned on or at least partially on or near objects in the surgical field such as, for example, end effectors and/or critical surgical structures. Some display arrangements may include changes in one or more display elements of the superimposed layers, including changes in color, size, shape, display time, display location, display frequency, highlighting, or combinations thereof, based on changes in display priority values. A graphical overlay is rendered on top of the active display monitor to quickly and efficiently communicate important information to the OR team.

In other aspects, techniques according to the present disclosure may include overlapping images that require analysis of video and tracking in order to properly overlay visual information data. In other aspects, techniques according to the present disclosure may include transmitting rich visual information instead of simple static icons, thereby providing additional visual information to the OR team in a more concise and easily understood manner. In other aspects, the visual overlay may be used in combination with an audible and/or somatosensory overlay (such as thermal, chemical and mechanical devices, and combinations thereof).

The following description relates generally to devices, systems, and methods that provide an Augmented Reality (AR) interactive experience during a surgical procedure. In this context, the surgical field and the images of surgical instruments and other objects present in the surgical field are enhanced by superimposing computer-generated visual, auditory, tactile, somatosensory, olfactory, or other sensory information on the surgical field, the real world images of instruments and other objects present in the surgical field. The image may be streamed in real-time, or may be a still image. Augmented reality is a technology for rendering and displaying virtual or "augmented" virtual objects, data, or visual effects superimposed on a real environment. The real environment may include a surgical field of view. A virtual object superimposed on a real environment may be represented as anchored or in a set position relative to one or more aspects of the real environment. In a non-limiting example, if a real world object leaves the real environment field of view, then a virtual object anchored to the real world object will also leave the augmented reality field of view.

The various display arrangements described in this disclosure relate to superimposing various visual representations of surgical data on a live stream of a surgical field of view. As used herein, the term overlay includes semi-transparent overlays, partial overlays, and/or moving overlays. Further, the superimposed layers may be positioned on or at least partially on or near objects in the surgical field such as, for example, end effectors and/or critical surgical structures. Some display arrangements may include changes in one or more display elements of the superimposed layers, including changes in color, size, shape, display time, display location, display frequency, highlighting, or combinations thereof, based on changes in display priority values.

As described herein, AR is an enhanced version of the real physical world, achieved through the use of digital visual elements, sounds, or other sensory stimuli delivered via technology. Virtual Reality (VR) is a computer-generated environment with scenes and objects that appear to be real, so that users feel themselves immersed in their surroundings. The environment is perceived by a device called a virtual reality headset or helmet. Both Mixed Reality (MR) and AR are considered immersive techniques, but they are not identical. MR is an extension of mixed reality, allowing real and virtual elements to interact in an environment. While AR often adds digital elements to a real-time view through the use of cameras, MR experiences combine elements of both AR and VR, in which real world and digital objects interact.

In an AR environment, one or more computer-generated virtual objects may be displayed with one or more real (i.e., so-called "real world") elements. For example, real-time images or videos of the surrounding environment may be displayed on a computer screen display along with one or more overlaid virtual objects. Such virtual objects may provide supplemental information about the environment or generally enhance the user's perception and participation in the environment. Instead, real-time images or videos of the surrounding environment may additionally or alternatively enhance user engagement with virtual objects shown on the display.

Apparatus, systems, and methods in the context of the present disclosure enhance images received from one or more imaging devices during a surgical procedure. The imaging device may include various endoscopes, AR devices, and/or cameras used during non-invasive and minimally invasive surgical procedures to provide images during open surgical procedures. The image may be streamed in real-time, or may be a still image. The devices, systems, and methods enhance images of a real-world surgical environment by overlaying virtual objects or representations of data and/or real objects on the real-world surgical environment, thereby providing an augmented reality interactive experience. The augmented reality experience may be viewed on a display and/or AR device that allows a user to view the overlaid virtual object on the real world surgical environment. The display may be located in the operating room or remote from the operating room. AR devices are worn on the head of a surgeon or other operating room personnel and typically include two stereoscopic display lenses or screens, one for each eye of the user. Natural light can pass through two transparent or translucent display lenses so that aspects of the real environment are visible, while also projecting light so that the virtual object is visible to the user of the AR device.

Two or more displays and AR devices may be used in a coordinated manner, such as a first display or AR device controlling one or more additional displays or AR devices in a defined character control system. For example, when the display or AR device is activated, the user may select a role (e.g., surgeon, surgical assistant, nurse, etc. during a surgical procedure) and the display or AR device may display information related to the role. For example, the surgical assistant may have a virtual representation of the displayed instrument that the surgeon needs when performing the next step of the surgical procedure. The surgeon's attention to the current step may see different display information than the surgical assistant.

While there are many known screen displays and warnings, the present disclosure provides many new and unique augmented reality interactive experiences during a surgical procedure. Such augmented reality interactive experiences include visual, auditory, tactile, somatosensory, olfactory, or other sensory feedback information to a surgical team inside or outside the operating room. Virtual feedback information superimposed on the real world surgical procedure environment may be provided to an Operating Room (OR) team, including personnel internal to the OR, including, but not limited to, for example, a knife surgeon, a surgeon assistant, a swabbing personnel, an anesthesiologist, and a round nurse, among others. The virtual feedback information may be transmitted over any number of display options, such as a master OR screen display, AR device, energy OR surgical stapler instrument, tablet computer, augmented reality glasses, device, and the like.

Fig. 1 shows a computer-implemented interactive surgical system 1 comprising one or more surgical systems 2 and a cloud-based system 4. The cloud-based system 4 may include a remote server 13 coupled to the storage 5. Each surgical system 2 includes at least one surgical hub 6 in communication with the cloud 4. For example, the surgical system 2 may include a visualization system 8, a robotic system 10, and a hand-held intelligent surgical instrument 12, each configured to communicate with each other and/or with the hub 6. In some aspects, the surgical system 2 may include M hubs 6, N visualization systems 8, O robotic systems 10, and P smart hand-held surgical instruments 12, where M, N, O and P are integers greater than or equal to 1. The computer-implemented interactive surgical system 1 may be configured to provide an augmented reality interactive experience during a surgical procedure as described herein.

Fig. 2 shows an example of a surgical system 2 for performing a surgical procedure on a patient lying on an operating table 14 in a surgical operating room 16. The robotic system 10 is used as part of the surgical system 2 in a surgical procedure. The robotic system 10 includes a surgeon's console 18, a patient side cart 20 (surgical robot), and a surgical robotic hub 22. When the surgeon views the surgical site through the surgeon's console 18 or an Augmented Reality (AR) device 66 worn by the surgeon, the patient-side cart 20 may manipulate at least one removably coupled surgical tool 17 through a minimally invasive incision in the patient. An image of the surgical site of the minimally invasive surgical procedure (e.g., a still image or a live image streamed in real time) may be obtained by the medical imaging device 24. The patient side cart 20 may maneuver the imaging device 24 to orient the imaging device 24. An image of the open surgical procedure may be obtained by the medical imaging device 96. The robotic hub 22 processes the image of the surgical site for subsequent display on the surgeon's console 18 or on an AR device 66 worn by the surgeon or to other personnel in the surgical room 16.

The optical components of imaging device 24, 96 or AR device 66 may include one or more illumination sources and/or one or more lenses. One or more illumination sources may be directed to illuminate multiple portions of the surgical field. The one or more image sensors may receive light reflected or refracted from tissue and instruments in the surgical field.

In various aspects, the imaging device 24 is configured for use in minimally invasive surgical procedures. Examples of imaging devices suitable for use in the present disclosure include, but are not limited to, arthroscopes, angioscopes, bronchoscopes, choledochoscopes, colonoscopes, cytoscopes, duodenoscopes, enteroscopes, esophageal-duodenal scopes (gastroscopes), endoscopes, laryngoscopes, nasopharyngeal-renal endoscopes, sigmoidoscopes, thoracoscopes, and hysteroscopes. In various aspects, the imaging device 96 is configured for use in an open (invasive) surgical procedure.

In various aspects, the visualization system 8 includes one or more imaging sensors, one or more image processing units, one or more storage arrays, and one or more displays strategically placed relative to the sterile field. In one aspect, the visualization system 8 includes interfaces for HL7, PACS, and EMR. In one aspect, the imaging device 24 may employ multispectral monitoring to distinguish between topography and underlying structures. Multispectral images capture image data over a specific range of wavelengths across the electromagnetic spectrum. Wavelengths, including light from frequencies outside the visible range, such as IR and ultraviolet, are separated by filters or devices sensitive to the particular wavelengths. Spectral imaging can extract information that is not visible to the human eye. Multispectral monitoring can reposition the surgical field after completion of the surgical task to perform a test on the treated tissue.

Fig. 2 shows the main display 19 positioned in the sterile field to be visible to an operator at the operating table 14. The visualization tower 11 is positioned outside the sterile zone and comprises a first non-sterile display 7 and a second non-sterile display 9 facing away from each other. The visualization system 8 guided by the hub 6 is configured to be able to coordinate the information flow to operators inside and outside the sterile field using the displays 7, 9, 19. For example, hub 6 may cause visualization system 8 to display AR images of the surgical site recorded by imaging devices 24, 96 on non-sterile displays 7, 9 or by AR device 66, while maintaining a real-time feed of the surgical site on primary display 19 or AR device 66. For example, the non-sterile displays 7, 9 may allow a non-sterile operator to perform diagnostic steps related to a surgical procedure.

Fig. 3 shows the surgical hub 6 in communication with the visualization system 8, the robotic system 10, and the hand-held intelligent surgical instrument 12. Hub 6 includes a hub display 35, an imaging module 38, a generator module 40, a communication module 30, a processor module 32, a memory array 34, and an operating room mapping module 33. The hub 6 further comprises a smoke evacuation module 26 and/or a suction/flushing module 28. In various aspects, the imaging module 38 includes an AR device 66 and the processor module 32 includes an integrated video processor and augmented reality modeler (e.g., as shown in fig. 10). The modular light source may be adapted for use with a variety of imaging devices. In various examples, multiple imaging devices may be placed at different locations in the surgical field to provide multiple views (e.g., non-invasive, minimally invasive, or open surgical procedures). The imaging module 38 may be configured to be switchable between imaging devices to provide an optimal view. In various aspects, the imaging module 38 may be configured to integrate images from different imaging devices and provide an augmented reality interactive experience during a surgical procedure as described herein.

Fig. 4 shows a surgical data network 51 including a modular communication hub 53 configured to enable connection of modular devices located in one or more operating rooms/rooms of a medical facility to a cloud-based system. Cloud 54 may include a remote server 63 (fig. 5) coupled to storage 55. Modular communication hub 53 includes a network hub 57 and/or a network switch 59 in communication with a network router 61. Modular communication hub 53 is coupled to local computer system 60 to process data. The modular devices 1a-1n located in the operating room may be coupled to a modular communication hub 53. The network hub 57 and/or the network switch 59 may be coupled to a network router 61 to connect the devices 1a-1n to the cloud 54 or the local computer system 60. The data associated with the devices 1a-1n may be transmitted via routers to cloud-based computers for remote data processing and manipulation. The operating room devices 1a-1n may be connected to the modular communication hub 53 by a wired channel or a wireless channel. The surgical data network 51 environment can be used to provide an augmented reality interactive experience during a surgical procedure as described herein, and in particular to provide an augmented image in a surgical field of view to one or more remote displays 58.

Fig. 5 illustrates a computer-implemented interactive surgical system 50. The computer-implemented interactive surgical system 50 is similar in many respects to the computer-implemented interactive surgical system 1. The computer-implemented interactive surgical system 50 includes one or more surgical systems 52 that are similar in many respects to the surgical system 2. Each surgical system 52 includes at least one surgical hub 56 in communication with a cloud 54, which may include a remote server 63. In one aspect, the computer-implemented interactive surgical system 50 includes a modular control tower 23 that is connected to a plurality of operating room devices, such as, for example, intelligent surgical instruments, robots, and other computerized devices located in an operating room. As shown in fig. 6, modular control tower 23 includes a modular communication hub 53 coupled to a computer system 60.

Returning to fig. 5, modular control tower 23 is coupled to imaging module 38 (which is coupled to endoscope 98), generator module 27 (which is coupled to energy device 99), smoke extractor module 76, suction/irrigation module 78, communication module 13, processor module 15, storage array 16, smart device/appliance 21 (which is optionally coupled to display 39), and sensor module 29. The operating room devices are coupled to cloud computing resources, such as servers 63, data storage 55, and display 58, via modular control tower 23. The robotic hub 72 may also be connected to the modular control tower 23 and to the server 63, the data storage 55, and the display 58. The device/instrument 21, visualization system 58, etc. may be coupled to the modular control tower 23 via a wired or wireless communication standard or protocol, as described herein. The modular control tower 23 may be coupled to a hub display 65 (e.g., monitor, screen) to display the received enhanced images, including overlaid virtual objects in the real surgical field received from the imaging module 38, the device/instrument display 39, and/or other visualization system 58. Hub display 65 may also display data received from devices connected to modular control tower 23 in combination with the image and the overlay image.

Fig. 6 shows a surgical hub 56 that includes a plurality of modules coupled to the modular control tower 23. The modular control tower 23 includes a modular communication hub 53 (e.g., a network connectivity device) and a computer system 60 to provide, for example, enhanced local processing, visualization, and imaging of surgical information. Modular communication hub 53 may be hierarchically configured to connect to extend the number of modules (e.g., devices) that may be connected to modular communication hub 53 and to transmit data associated with the modules to computer system 60, cloud computing resources, or both. Each of the hubs 57/switches 59 in the modular communications hub 53 may include three downstream ports and one upstream port. The upstream hub 57/switch 59 is connected to the processor 31 to provide a communication connection with cloud computing resources and a local display 67. Communication with cloud 54 may be through a wired or wireless communication channel.

The computer system 60 includes a processor 31 and a network interface 37. The processor 31 is coupled to a communication module 41, a storage device 45, a memory 46, a non-volatile memory 47 and an input/output interface 48 via a system bus. The system bus may be any of several types of bus structures including a memory bus or memory controller, a peripheral bus or external bus, and/or a local bus using any of a variety of available bus architectures.

The processor 31 includes an augmented reality modeler (e.g., as shown in fig. 10) and may be implemented as a single or multi-core processor, such as those provided by texas instruments (Texas Instruments) under the trade name ARM Cortex. In one aspect, the processor may be an on-chip memory available from, for example, texas instruments (Texas Instruments) LM4F230H5QR ARM Cortex-M4F processor core including 256KB of single-cycle flash memory or other non-volatile memory (up to 40 MHz), a prefetch buffer for improving execution above 40MHz, 32KB single-cycle Sequential Random Access Memory (SRAM), loaded withInternal read-only memory (ROM) of software, 2KB electrically erasable programmable read-only memory (EEPROM), and/or one or more Pulse Width Modulation (PWM) modules, one or more Quadrature Encoder Inputs (QEI) analog, one or more 12-bit analog-to-digital converters (ADC) with 12 analog input channels, the details of which can be seen in the product data sheet.

The system memory includes volatile memory and nonvolatile memory. A basic input/output system (BIOS), containing the basic routines to transfer information between elements within the computer system, such as during start-up, is stored in nonvolatile memory. For example, the non-volatile memory may include ROM, programmable ROM (PROM), electrically Programmable ROM (EPROM), EEPROM, or flash memory. Volatile memory includes Random Access Memory (RAM), which acts as external cache memory. In addition, RAM is available in a variety of forms, such as SRAM, dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDR SDRAM) Enhanced SDRAM (ESDRAM), synchronous Link DRAM (SLDRAM), and Direct Rambus RAM (DRRAM).

Computer system 60 also includes removable/non-removable, volatile/nonvolatile computer storage media such as, for example, magnetic disk storage. Disk storage includes, but is not limited to, devices such as magnetic disk drives, floppy disk drives, tape drives, jaz drives, zip drives, LS-60 drives, flash memory cards, or memory sticks. In addition, the disk storage can include storage media separately or in combination with other storage media including, but not limited to, an optical disk drive such as a compact disk ROM device (CD-ROM), compact disk recordable drive (CD-R drive), compact disk rewritable drive (CD-RW drive) or a digital versatile disk ROM drive (DVD-ROM). To facilitate connection of the disk storage devices to the system bus, a removable or non-removable interface may be used.

In various aspects, the computer system 60 of fig. 6, the imaging module 38 and/or the visualization system 58 and/or the processor module 15 of fig. 4-6 may include an image processor, an image processing engine, a Graphics Processing Unit (GPU), a media processor, or any special-purpose Digital Signal Processor (DSP) for processing digital images. The image processor may employ parallel computation with single instruction, multiple data (SIMD) or multiple instruction, multiple data (MIMD) techniques to increase speed and efficiency. The digital image processing engine may perform a series of tasks. The image processor may be a system on a chip having a multi-core processor architecture.

Fig. 7 shows an augmented reality system 263 that includes an intermediate signal combiner 64 positioned in the communication path between the imaging module 38 and the surgical hub display 67. The signal combiner 64 combines audio and/or image data received from the imaging module 38 and/or the AR device 66. The surgical hub 56 receives the combined data from the combiner 64 and superimposes the data provided to the display 67 on which the superimposed data is displayed. Imaging device 68 may be a digital video camera and audio device 69 may be a microphone. The signal combiner 64 may include a wireless heads-up display adapter to couple to an AR device 66 placed in the communication path of the display 67 to the console, allowing the surgical hub 56 to superimpose data on the display 67.

Fig. 8 illustrates an Augmented Reality (AR) system including an intermediate signal combiner positioned in a communication path between an imaging module and a surgical hub display. Fig. 8 shows AR device 66 worn by surgeon 73 to transmit data to surgical hub 56. Peripheral information of AR device 66 does not include active video. Instead, the peripheral information includes only the device settings or signals that do not have the same refresh rate requirements. The interaction may augment the surgeon's 73 information based on linking with preoperative Computed Tomography (CT) or other data linked in the surgical hub 56. The AR device 66 may identify structures, such as querying whether the instrument is contacting a nerve, vessel, or adhesion. The AR device 66 may include preoperative scan data, optical views, tissue interrogation features obtained throughout the procedure, and/or processing in the surgical hub 56 for providing answers. The surgeon 73 may dictate notes to the AR device 66 for storage in the hub storage device 45 with the patient data for later reporting or follow-up.

The AR device 66 worn by the surgeon 73 is linked to the surgical hub 56 with audio and visual information to avoid the need for superposition and to allow the display information to be customized around the perimeter of the field of view. The AR device 66 provides a signal from the device (e.g., instrument), answers queries about device settings or location information linked to the video to identify quadrants or locations. The AR device 66 has audio control and audio feedback from the AR device 66. The AR device 66 is able to interact with all other systems in the operating room and has feedback and interaction available wherever the surgeon 73 looks. For example, the AR device 66 may receive voice or gesture initiated commands and inquiries from the surgeon, and the AR device 66 may provide feedback in the form of one or more modalities including audio, visual, or tactile touches.

Fig. 9 shows a surgeon 73 wearing AR device 66, a patient 74, and a camera 96 may be included in operating room 75. The AR device 66 worn by the surgeon 73 may be used to present virtual objects superimposed on a real-time image of the surgical field to the surgeon 73 via an augmented reality display 89 or via a hub-connected display 67. The real-time image may include a portion of the surgical instrument 77. The virtual object may not be visible to other people (e.g., surgical assistants or nurses) within the operating room 75, although they may also be wearing the AR device 66. Even if another person is viewing operating room 75 using AR device 66, that person may not see the virtual object or be able to see the virtual object in the augmented reality shared with surgeon 73, or be able to see a modified version of the virtual object (e.g., according to a unique customization to surgeon 73) or be able to see a different virtual object.

The virtual objects and/or data may be configured to appear on a portion of the surgical instrument 77 or in a surgical field captured by the imaging module 38, by the imaging device 68 during a minimally invasive surgical procedure, and/or by the camera 96 during an open surgical procedure. In the illustrated example, the imaging module 38 is a laparoscopic camera that provides real-time feeding of the surgical field during a minimally invasive surgical procedure. The AR system may present a virtual object that is fixed to a real object regardless of the perspective of one or more observers (e.g., surgeon 73) of the AR system. For example, the virtual object may be visible to an observer of the AR system inside the operating room 75, and not visible to an observer of the AR system outside the operating room 75. When an observer enters operating room 75, the virtual object may be displayed to the observer outside operating room 75. The augmented image may be displayed on the surgical hub display 67 or the augmented reality display 89.

The AR device 66 may include one or more screens or lenses, such as a single screen or two screens (e.g., one screen for each eye of the user). The screen may allow light to pass through the screen such that aspects of the real environment are visible when the virtual object is displayed. The virtual object may be made visible to the surgeon 73 by projected light. The virtual object may appear to have some degree of transparency or may be opaque (i.e., occlude aspects of the real environment).

The AR system may be viewable to one or more observers and may include differences between views available to one or more observers while maintaining some aspects common between views. For example, the heads-up display may change between two views, while virtual objects and/or data may be fixed to real objects or regions in the two views. Aspects, such as color, illumination, or other changes of the object, may be changed between views without changing the fixed location of the at least one virtual object.

The user may consider virtual objects and/or data presented in the AR system as opaque or as including a degree of transparency. In one example, a user may interact with a virtual object, such as by moving the virtual object from a first location to a second location. For example, the user may move the object with his own hand. This may be accomplished virtually in an AR system by determining that the hand has moved to a position coincident with or adjacent to the object (e.g., using one or more cameras that may be mounted on the AR device 66, such as AR device camera 79 or a separate camera 96, and which may be static or controllable to move) and causing the object to move in response. The virtual aspect may comprise a virtual representation of the real world object or may comprise a visual effect, such as a lighting effect or the like. The AR system may include rules that govern the behavior of the virtual object, such as subjecting the virtual object to gravity or friction, or may include other predefined rules that exclude real world physical constraints (e.g., floating objects, perpetuation, etc.). The AR device 66 may include a camera 79 (not confused with camera 96, separate from the AR device 66) located on the AR device 66. The AR device camera 79 or camera 96 may include an infrared camera, an infrared filter, a visible light filter, multiple cameras, a depth camera, and the like. The AR device 66 may project the virtual item on a representation of the real environment so as to be viewable by the user.

The AR device 66 may be used in an operating room 75 during a surgical procedure, for example, performed on a patient 74 by a surgeon 73. The AR device 66 may project or display virtual objects such as virtual objects during a surgical procedure to enhance the vision of the surgeon. The surgeon 73 may view the virtual object using the AR device 66, a remote control for the AR device 66, or may interact with the virtual object, such as using a hand to "interact" with the virtual object or a gesture recognized by a camera 79 of the AR device 66. The virtual object may augment a surgical tool, such as surgical instrument 77. For example, the virtual object may appear (to the surgeon 73 viewing the virtual object through the AR device 66) to be coupled to or maintained a fixed distance from the surgical instrument 77. In another example, the virtual object may be used to guide a surgical instrument 77 and may appear to be fixed to the patient 74. In some examples, the virtual object may react to movement of other virtual objects or real world objects in the surgical field of view. For example, the virtual object may be changed while the surgeon is manipulating a surgical instrument that is proximate to the virtual object.

The augmented reality display system imaging device 38 captures a real image of the surgical field during the surgical procedure. The augmented reality displays 89, 67 present a superposition of the operational aspects of the surgical instrument 77 over the real image of the surgical field. Surgical instrument 77 includes communication circuitry 231 to communicate operational aspects and functional data from surgical instrument 77 to AR device 66 via communication circuitry 233 on AR device 66. Although surgical instrument 77 and AR device 66 are shown as RF wireless communication between circuits 231, 233 as indicated by arrow B, C, other communication techniques (e.g., wired, ultrasonic, infrared, etc.) may be employed. The overlay is related to the operational aspects of the active visualization of the surgical instrument 77. The overlay combines aspects of tissue interaction in the surgical field with functional data from the surgical instrument 77. The processor portion of AR device 66 is configured to receive operational aspects and functional data from surgical instrument 77, determine overlays relating to the operation of surgical instrument 77, and combine tissue aspects in the surgical field with functional data from surgical instrument 77. The enhanced image indicates warnings regarding device performance considerations, warnings of incompatible usage, warnings regarding incomplete capture. Incompatible uses include tissue out of range conditions and tissue not properly balanced within the jaws of the end effector. The additional enhanced image provides an indication of an incident, including an indication of tissue tension and an indication of foreign object detection. Other enhanced image indicates device status overlay and instrument indication.

Fig. 10 illustrates a system 83 for enhancing a surgical field image with information using an AR display 89 in accordance with at least one aspect of the present disclosure. The system 83 may be used to perform the techniques described below, for example, by using the processor 85. The system 83 includes one aspect of the AR device 66 that may communicate with a database 93. The AR device 66 includes a processor 85, a memory 87, an AR display 89, and a camera 79. The AR device 66 may include a sensor 90, a speaker 91, and/or a haptic controller 92. Database 93 may include an image store 94 or a pre-operative plan store 95.

The processor 85 of the AR device 66 includes an augmented reality modeler 86. The augmented reality modeler 86 may be used by the processor 85 to create an augmented reality environment. For example, the augmented reality modeler 86 may receive an image of an instrument in a surgical field of view, such as from the camera 79 or the sensor 90, and create an augmented reality environment to fit within a displayed image of the surgical field of view. In another example, physical objects and/or data may be superimposed on the surgical field of view and/or the surgical instrument image, and the augmented reality modeler 86 may use the physical objects and data to present an augmented reality display of the virtual objects and/or data in the augmented reality environment. For example, the augmented reality modeler 86 may use or detect an instrument at a surgical site of a patient and present virtual objects and/or data on the surgical instrument and/or images of the surgical site in a surgical field captured by the camera 79. The AR display 89 may display an AR environment superimposed on a real environment. Display 89 may display virtual objects and/or data using AR device 66, such as a fixed location in an AR environment.

The AR device 66 may include a sensor 90, such as an infrared sensor. The camera 79 or sensor 90 may be used to detect movements, such as gestures by a surgeon or other user, which the processor 85 may interpret as user attempts or intended interactions with the virtual target. The processor 85 may identify objects in the real environment, such as by using the processing information received by the camera 79. In other aspects, the sensor 90 may be a tactile, auditory, chemical, or thermal sensor to generate corresponding signals that may be combined with various data feeds to create an enhanced environment. The sensors 90 may include binaural audio sensors (spatial sound), inertial measurement sensors (accelerometers, gyroscopes, magnetometers), environmental sensors, depth camera sensors, hand-eye tracking sensors, and voice command recognition functions.

For example, during a surgical procedure, the AR display 89 may present virtual features corresponding to physical features hidden by anatomical aspects of the patient, such as within the surgical field, while allowing the surgical field to be viewed through the AR display 89. The virtual feature may have a virtual position or orientation corresponding to a first physical position or orientation of the physical feature. In one example, the virtual position or orientation of the virtual feature may include an offset from a first physical position or orientation of the physical feature. The offset may include a predetermined distance from the augmented reality display, a relative distance from the augmented reality display to an anatomical aspect, and the like.

In one example, the AR device 66 may be a single AR device. In one aspect, AR device 66 may be a Hololens 2AR device manufactured by Microsoft corporation of Redmond, wash. The AR device 66 includes goggles with lenses and binaural audio features (spatial sound), inertial measurement devices (accelerometers, gyroscopes, magnetometers), environmental sensors, depth and video cameras, hand-eye tracking, and voice command recognition functions. It provides an improved field of view with high resolution by using a mirror to guide the waveguide in front of the wearer's eye. The image can be magnified by changing the angle of the mirror. It also provides eye tracking to identify the user and adjust the lens width for the particular user.

In another example, the AR device 66 may be a SNAPCHAT SPECTACLES AR device. The AR device is able to capture paired images and recreate a 3D depth map, add virtual effects, and replay 3D video. The AR device includes two HD cameras to capture 3D photos and videos at 60fps while four built-in microphones record immersive high-fidelity audio. The images from the two cameras combine to build a geometric map around the user's real world, providing a new perception of depth. The photos and videos may be synchronized to the external display device wirelessly.

In yet another example, AR device 66 may be a Google Glass 2AR device. The AR device provides inertial measurement (accelerometer, gyroscope, magnetometer) information superimposed on the lens (outside the field of view) to supplement the information.

In another example, AR device 66 may be Amazon's Echo Frames AR device. The AR device has no camera/display. The microphone and speaker are connected to Alexa. The AR device provides less functionality than a heads-up display.

In yet another example, AR device 66 may be a North (Google) Focals AR device. The AR device provides notification push/smart watch simulation; inertial measurement, screen overlay of information (weather, calendar, messages), voice control (Alexa) integration. The AR device provides a basic heads-up display function.

In another example, the AR device 66 may be a Nreal AR device. The AR device includes spatial sound, two ambient cameras, photo cameras, IMU (accelerometer, gyroscope), ambient light sensor, proximity sensor functions. Nebula projects application information onto the lens.

In various other examples, AR device 66 may be any of the following commercially available AR devices: MAGIC LEAP 1, epson Moverio, vuzix Blade AR, zenFone AR, microsoft AR eyeglass prototype, eyeTap to generate light collinear with ambient light directly into the retina. For example, the beam splitter makes the same light seen by the eye available for computer processing and superimposing information. AR visualization systems include HUDs, contact lenses, glasses, virtual Reality (VR) headphones, virtual retinal displays, operating room displays, and/or smart contact lenses (biomimetic lenses).

The multi-user interface for AR device 66 includes a virtual retinal display (such as a raster display drawn directly on the retina rather than on a screen in front of the eye), a smart television, a smart phone, and/or a spatial display (such as a Sony spatial display system).

Other AR technologies may include, for example, AR capture devices and software applications, AR creation devices and software applications, and AR cloud devices and software applications. The AR capture device and software applications include, for example, apple Polycam application Ubiquity 6 (Mirrorworld using display. Land app), which a user can scan and obtain 3D images of the real world (to create a 3D model). AR creation means and software applications include, for example Adobe Aero, vuforia, ARToolKit, google ARCore, APPLE ARKIT, MAXST, aurasma, zappar, blippar. AR cloud devices and software applications include Facebook, google (world geometry, object recognition, predictive data), amazon AR cloud (business), microsoft Azure, samsung Project Whare, niantic, MAGIC LEAP, for example.

Situational awareness refers to the ability of some aspects of a surgical system to determine or infer information related to a surgical procedure from data received from databases and/or instruments. The information may include the type of surgery being performed, the type of tissue being operated on, or the body cavity being the subject of the surgery. With context information associated with the surgical procedure, the surgical system may, for example, improve the manner in which the surgical system controls the modular devices (e.g., robotic arms and/or robotic surgical tools) connected thereto, and provide context information or advice to the surgeon during the course of the surgical procedure.

Fig. 11 shows a time axis of a situational awareness surgical procedure. Fig. 11 shows a timeline 5200 of an exemplary surgical procedure and context information that the surgical hub 5104 can derive from data received from the data source 5126 at each step of the surgical procedure. The time axis 5200 depicts typical steps that nurses, surgeons and other medical personnel will take during a segmental lung removal procedure, starting from the establishment of an operating room and until the patient is transferred to a post-operative recovery room. The situation aware surgical hub 5104 receives data from the data source 5126 throughout the surgical procedure, including data generated each time a medical professional utilizes the modular device 5102 paired with the surgical hub 5104. The surgical hub 5104 can receive this data from the paired modular device 5102 and other data sources 5126 and continually derive inferences about ongoing surgery (i.e., background information), such as which step of the surgery to perform at any given time, as new data is received. The situational awareness system of the surgical hub 5104 can, for example, record data related to the procedure used to generate the report, verify steps that medical personnel are taking, provide data or cues (e.g., via a display screen) that may be related to a particular procedure, adjust the modular device 5102 based on context (e.g., activate a monitor, adjust the FOV of a medical imaging device, or change the energy level of an ultrasonic surgical instrument or RF electrosurgical instrument), and take any other such action described herein.

First 5202, a hospital staff member retrieves the patient's EMR from the hospital's EMR database. Based on patient data selected in the EMR, the surgical hub 5104 determines that the procedure to be performed is a thoracic procedure.

Second 5204, the staff member scans the incoming medical supplies for the procedure. The surgical hub 5104 cross-compares the scanned supplies with the list of supplies utilized in the various types of protocols and confirms that the combination of supplies corresponds to the chest protocol. In addition, the surgical hub 5104 can also determine that the procedure is not a wedge procedure (because the incoming supplies lack certain supplies required for, or otherwise do not correspond to, a chest wedge procedure).

Third 5206, the medical personnel scans the patient belt via a scanner 5128 communicatively connected to the surgical hub 5104. The surgical hub 5104 may then confirm the identity of the patient based on the scanned data.

Fourth 5208, the medical staff opens the auxiliary equipment. The auxiliary devices utilized may vary depending on the type of surgery and the technique to be used by the surgeon, but in this exemplary case they include smoke evacuators, insufflators and medical imaging devices. When activated, the ancillary equipment as the modular device 5102 may automatically pair with the surgical hub 5104 located in a specific vicinity of the modular device 5102 as part of its initialization process. The surgical hub 5104 may then derive background information about the surgical procedure by detecting the type of modular device 5102 paired therewith during this pre-operative or initialization phase. In this particular example, the surgical hub 5104 determines that the surgical procedure is a VATS procedure based on this particular combination of paired modular devices 5102. Based on a combination of data from the patient's EMR, a list of medical supplies to be used in the procedure, and the type of modular device 5102 connected to the hub, the surgical hub 5104 can generally infer the particular procedure that the surgical team will perform. Once the surgical hub 5104 knows the particular procedure being performed, the surgical hub 5104 can retrieve the procedure from memory or the cloud and then cross-reference the data it subsequently receives from the connected data sources 5126 (e.g., the modular device 5102 and the patient monitoring device 5124) to infer the procedure being performed by the surgical team.

Fifth 5210, the staff attaches EKG electrodes and other patient monitoring devices 5124 to the patient. The EKG electrode and other patient monitoring device 5124 can be paired with the surgical hub 5104. As the surgical hub 5104 begins to receive data from the patient monitoring device 5124, the surgical hub 5104 thus confirms that the patient is in the operating room.

Sixth 5212, medical personnel induce anesthesia in patients. The surgical hub 5104 may infer that the patient is under anesthesia based on data (including EKG data, blood pressure data, ventilator data, or a combination thereof) from the modular device 5102 and/or the patient monitoring device 5124. At the completion of the sixth step 5212, the preoperative portion of the lung segmental resection procedure is completed and the operative portion begins.

Seventh 5214, collapse of the lungs of the patient being operated on (while ventilation is switched to the contralateral lung). The surgical hub 5104 may infer from the ventilator data that the patient's lungs have collapsed. The surgical hub 5104 can infer that the surgical portion of the procedure has begun, as it can compare the detection of the patient's lung collapse to the expected steps of the procedure (which can be previously accessed or retrieved), thereby determining that collapsing the lung is a surgical step in that particular procedure.

Eighth 5216, a medical imaging device 5108 (e.g., an endoscope) is inserted and video from the medical imaging device is activated. The surgical hub 5104 receives medical imaging device data (i.e., still image data or live streaming video in real time) through its connection with the medical imaging device. After receiving the medical imaging device data, the surgical hub 5104 may determine that the laparoscopic portion of the surgical procedure has begun. In addition, the surgical hub 5104 may determine that the particular procedure being performed is a segmental resection, rather than a pneumonectomy (note that the surgical hub 5104 has excluded a wedge procedure based on data received at the second step 5204 of the procedure). The data from the medical imaging device 124 (fig. 2) may be used to determine background information related to the type of procedure being performed in a number of different ways, including by determining the angle of the visual orientation of the medical imaging device relative to the patient's anatomy, monitoring the number of medical imaging devices utilized (i.e., activated and paired with the surgical hub 5104), and monitoring the type of visualization device utilized.

For example, one technique for performing a vat lobectomy places the camera in the lower anterior corner of the patient's chest over the septum, while one for performing a vat segmented resection places the camera in an anterior intercostal position relative to the segmented slit. For example, using pattern recognition or machine learning techniques, the situational awareness system may be trained to recognize the positioning of the medical imaging device from the visualization of the patient anatomy. As another example, one technique for performing a vat lobectomy utilizes a single medical imaging apparatus, while another technique for performing a vat segmented excision utilizes multiple cameras. As another example, a technique for performing vat segmental resections utilizes an infrared light source (which may be communicatively coupled to a surgical hub as part of a visualization system) to visualize segmental slots that are not used in vat pulmonary resections. By tracking any or all of this data from the medical imaging device 5108, the surgical hub 5104 can thus determine the particular type of surgical procedure being performed and/or the technique for the particular type of surgical procedure.

Ninth 5218, the surgical team begins the anatomic steps of the procedure. The surgical hub 5104 can infer that the surgeon is in the process of dissecting to mobilize the patient's lungs because it receives data from the RF generator or ultrasound generator indicating that the energy instrument is being fired. The surgical hub 5104 can cross-reference the received data with a retrieval step of the surgical procedure to determine that the energy instrument fired at that point in the method (i.e., after completion of the previously discussed surgical step) corresponds to an anatomical step.

Tenth 5220, the surgical team proceeds to the ligation step of the procedure. The surgical hub 5104 can infer that the surgeon is ligating arteries and veins because it receives data from the surgical stapling and severing instrument indicating that the instrument is being fired. Similar to the previous steps, the surgical hub 5104 can derive the inference by cross-referencing the receipt of data from the surgical stapling and severing instrument with the retrieval steps in the method.

Eleventh 5222, a segmental resection portion of the procedure is performed. The surgical hub 5104 infers that the surgeon is transecting soft tissue based on data from the surgical instrument, including data from the staple cartridge. The cartridge data may correspond to the size or type of staples fired by the instrument. The cartridge data may indicate the type of tissue being stapled and/or transected for different types of staples employed in different types of tissue. The type of staples being fired is used for soft tissue or other tissue types to enable the surgical hub 5104 to infer that a segmental resection procedure is being performed.

Twelfth 5224, the node dissection step is performed. The surgical hub 5104 may infer that the surgical team is dissecting a node and performing a leak test based on data received from the generator indicating that an RF or ultrasonic instrument is being fired. For this particular procedure, the use of an RF or ultrasonic instrument after transecting the soft tissue corresponds to a node dissection step, which allows the surgical hub 5104 to make this inference. It should be noted that the surgeon switches back and forth between surgical stapling/cutting instruments and surgical energy (i.e., RF or ultrasonic) instruments periodically, depending on the particular step in the procedure, as the different instruments are better suited for the particular task. Thus, the particular sequence in which the stapling/severing instrument and the surgical energy instrument are used may dictate the steps of the procedure that the surgeon is performing. At the completion of the twelfth step 5224, the incision is closed and the post-operative portion of the procedure begins.

Thirteenth 5226, the patient is reversed from anesthesia. For example, the surgical hub 5104 may infer that the patient is waking from anesthesia based on ventilator data (i.e., the patient's respiration rate begins to increase).

Finally, fourteenth 5228, the medical personnel remove various patient monitoring devices 5124 from the patient. Thus, when the surgical hub 5104 loses EKG, BP and other data from the patient monitoring device 5124, the hub can infer that the patient is being transferred to the recovery room. The surgical hub 5104 can determine or infer when each step of a given surgical procedure occurs from data received from various data sources 5126 communicatively coupled to the surgical hub 5104.

In addition to using patient data from the EMR database to infer the type of surgical procedure to be performed, the situational awareness surgical hub 5104 may also use patient data to generate control adjustments for the paired modular device 5102, as shown in a first step 5202 of the timeline 5200 shown in fig. 11.

Augmented reality display of invisible procedure steps

It should be appreciated that a computer-implemented interactive surgical system may include one or more surgical systems and cloud-based systems. The cloud-based system may include a remote server coupled to a storage device. Each surgical system includes at least one surgical hub in communication with the cloud. For example, the surgical system may include a visualization system, a robotic system, and one or more hand-held intelligent surgical instruments, each configured to communicate with each other and/or with a hub. The surgical hub can dynamically determine which devices are in use and the locations of those devices relative to each other and to critical and anatomical structures as identified by the system. Based on the location of these devices, the patient anatomy, and procedure steps in the operating room, the augmented reality display may be updated to depict one or more auxiliary augmented reality views. Such auxiliary augmented reality views may consist of views within the surgical field that are not visible to the surgeon in the primary field of view. In one aspect, such auxiliary augmented reality views may depict anatomical structures hidden in the primary field of view by other tissues. In another aspect, such an auxiliary augmented reality view may depict a view of the handheld intelligent surgical instrument from a second viewpoint (e.g., an underside view of the handheld intelligent surgical instrument).

It will be appreciated that computer-implemented interactive surgical systems constantly acquire device position and usage data during a procedure. The interactive surgical system may also continuously receive visual tracking or imaging information of various intelligent surgical instruments or other devices with respect to the patient anatomy. The surgical system may also retain imaging information throughout the surgical procedure. Through connection with the cloud-based system, the surgical system may also retain imaging information of the patient anatomy from a previous surgical procedure.

The computer-implemented interactive surgical system may determine a state of the smart surgical instrument based on device movements of the smart surgical instrument while in use. Such movement data may be obtained from the smart device itself, for example, based on a tri-axial accelerometer disposed within the device. Alternatively, the movement data may be obtained from a visualization device that may optically track the movement of the surgical instrument. The interactive surgical system may further include an anatomical image recognition algorithm configured to receive imaging data of the anatomical structure and determine a property and location of the anatomical structure. The current step of the surgical procedure may be identified by the interactive surgical system using a combination of the motion of the surgical device and the anatomical determination surrounding the surgical device.

In some aspects, the interactive surgical system may use imaging data about the surgical device obtained from the visualization device to determine whether the surgical device is a suitable device for the current step in the surgical procedure. The augmented reality device may provide a virtual object (such as an icon or text box) as a warning that superimposes an image of the surgical device in the augmented reality display to provide a warning to the surgical device user that the device is incorrect for the procedure.

In the step where the surgeon cannot see certain portions of the end effector of the intelligent surgical device, the augmented reality display may include a generated secondary view to show the surgeon the location of the device that cannot be seen in the current field of view. By way of example, fig. 12A depicts a surgical display 11000 obtained from a surgical imaging device during a laparoscopic sleeve gastrectomy procedure, with a portion of the fundus of the patient removed. Fig. 12A shows the stomach 11002 when the surgeon uses the stapler 11004 to staple the inner edges of the stomach 11002 together and cut away the remaining stomach portion 11003. As can be observed, the surgical display 11000 depicts a top surface 11006 of an end effector of the stapler 11004. The surgeon may wish to see the underside of the stapler before firing the stapler 11004 in order to ensure that the two edges of the resected stomach are sealed together. The surgeon may not want to rotate the stapler 11004 while it is clamped to the stomach 11002 to see the backside of the device prior to firing. Such rotation of the stapler 11004 may result in pulling the tissue in a manner that may impair the stapling function. Fig. 12B depicts an augmented reality image 11012 that includes a surgical display 11000 and a virtual secondary view 11010 superimposed over the surgical display. The virtual secondary view 11010 may include an augmented reality depiction of a side view 11014 of the stapler. The augmented reality depiction of the side view 11014 of the stapler can depict a side view 11016 of a portion of the stomach of a patient grasped by the lower jaw 11018 of the stapler. In this way, the virtual secondary view 11010 may allow a surgeon to visualize the underside of the stapler and see the underside without having to over manipulate tissue.

In some aspects, the augmented reality image 11012 may display information other than an imaging view that is not available to the surgeon (such as a side view 11014 of the stapler and a side view 11016 of a portion of the patient's stomach). For example, the virtual secondary view 11010 may also include visual indicators 11020 regarding the status of a procedure or tissue. For example, an alert 11021 indicating a tissue state may be depicted. In another example, the device status indicator 11022 can indicate a status of a current operation of the stapler.

In one aspect, the virtual secondary view 11010 may be created using predictive modeling of the patient based on previous anatomical images of the patient or images of similar anatomical portions of other patients undergoing the same surgical procedure. In another aspect, the virtual secondary view 11010 may be created from real-time images obtained from secondary cameras used during the procedure. In one example, the surgeon may request the virtual secondary view 11010 from the interactive surgical system using a gesture or verbal command. The interactive surgical system may alert the surgeon to adjust the position of the secondary camera position so that an auxiliary view may be created.

In another aspect, the virtual secondary view 11010 can be used to identify and display critical anatomy throughout a surgical procedure. As an example, such a persistent secondary view may be used to maintain an image of a tumor disposed on an organ of interest on a display device throughout a surgical procedure. Such a persistent display may allow the surgeon to switch between superimposing the view on the image of the current procedure and having it as a secondary view to the display side.

Predictive analysis of procedures and AI learning-displaying power of active devices

It should be appreciated that a computer-implemented interactive surgical system may include one or more surgical systems and cloud-based systems. The cloud-based system may include a remote server coupled to a storage device. Each surgical system includes at least one surgical hub in communication with the cloud. For example, the surgical system may include a visualization system, a robotic system, and one or more hand-held intelligent surgical instruments, each configured to communicate with each other and/or with a hub. The surgical hub can dynamically determine which devices are in use and the locations of those devices relative to each other and to critical and anatomical structures as identified by the system. Additionally, the computer-implemented interactive surgical system and/or cloud-based system may include an artificial intelligence ("AI") system configured to monitor data extracted from previous cases of the same protocol type and import data from current cases. Cloud-based data specific to the surgeon operating in a particular surgical case (and/or all such surgical cases completed), along with device location data, may allow the AI system to identify the current procedure step and use this information to predict the next step of the procedure. Using the prediction, the augmented reality display may be updated to present predicted next actions to take and/or predicted outcomes based on previous cases.

The computer-implemented interactive surgical system may determine a state of the smart surgical instrument based on device movements of the smart surgical instrument while in use. Such movement data may be obtained from the smart device itself, for example, based on a tri-axial accelerometer disposed within the device. Alternatively, the movement data may be obtained from a visualization device that may optically track the movement of the surgical instrument. The interactive surgical system or AI system may further include an anatomical image recognition algorithm configured to receive imaging data of the anatomical structure and determine a property and location of the anatomical structure. The current step of the surgical procedure may be identified by the interactive surgical system using a combination of the motion of the surgical device and the anatomical determination surrounding the surgical device.

It will be appreciated that computer-implemented interactive surgical systems constantly acquire device position and usage data during a procedure. The interactive surgical system may also continuously receive visual tracking or imaging information of various intelligent surgical instruments or other devices with respect to the patient anatomy. The surgical system may also retain imaging information throughout the surgical procedure. Through connection with the cloud-based system, the surgical system may also retain imaging information of the patient anatomy from a previous surgical procedure, or imaging information from a related surgical procedure from a different patient anatomy.

In one aspect, the data may be sent to a cloud data source configured to be able to store in memory all previous protocol data from additional hub connection cases. The data may be mined and analyzed to predict the next step most likely to be taken by the surgeon. Non-limiting examples of predictive modeling may use one or more of classification models, regression models, and Markov chain models. The prior protocol data may include imaging data and data obtained from a particular device when the particular device is used in a protocol. The device-related data may include, for example, power level, timing parameters, staple type, device position and orientation data, and other operating parameters. The surgical case analyzed may encompass any number of related or identical procedures. In some instances, only relevant cases completed by a particular surgeon performing the procedure may be analyzed. In one aspect, a surgeon performing the current procedure may select which previous case(s) should be analyzed as being related to the case at hand in order to make predictive recommendations.

Using the surgeon-specified predictions, the tracked position and orientation of the surgical device in use, and the patient anatomy, the augmented reality display can be updated to show the predictions of the next surgical action (e.g., predicted position of the stapler for its next stapling operation). It will be appreciated that the augmented reality display may be shown on any display device within or outside the operating room. In some aspects, the augmented reality display may be displayed on a primary display or a primary display in the operating room. Alternatively, the augmented reality display may be displayed on one or more alternative displays, such as a tablet device, secondary monitor, or even a display device associated with a particular smart surgical device (such as a device generator display). This prediction may be accompanied by additional device-specific recommendations, such as staple reload sizes predicted based on the patient's anatomy, or recommended use of buttress materials to reduce leakage based on observations from previous stapler operations, patient disease states, etc.

Other augmented reality displays may include recommendations related to the use of additional surgical devices available to complete a procedure with improved results. The intelligent surgical system may also adjust the communication priority between the intelligent surgical devices within the hub network by tracking procedure steps during an ongoing surgical procedure and comparing those procedure steps with previously obtained data stored in the cloud system. Thus, based on the surgical history, a second smart surgical device that would be needed after use of the first surgical device may prioritize its communication flow according to its intended use. For example, after all stapler fires are completed in the gastric sleeve procedure, communication flow from the needle driver may take precedence over other devices.

By way of example, fig. 13A depicts a surgical display 11000 obtained from a surgical imaging device during a laparoscopic sleeve gastrectomy procedure, with a portion of the fundus of the patient removed, similar to the depiction of fig. 12A. Fig. 13A shows the stomach 11002 when the surgeon uses the stapler 11004 to staple the inner edges of the stomach 11002 together and cut away the remaining stomach portion 11003. As can be observed, the surgical display 11000 depicts the top surface 11006 of the end effector of the stapler 11004 at a particular location on the stomach 11002. The surgeon may be uncertain as to how or where to position the stapler 11004 for the next stapling and cutting operation.

Fig. 13B depicts an augmented reality image 11030 including a surgical display 11000 and a predicted or recommended placement of a stapler 11032. In some aspects, the augmented reality image 11030 may display auxiliary information 11034 in addition to the predicted or recommended placement of the stapler 11032. For example, the auxiliary information 11034 may include recommendations 11036 for changing device operating parameters, such as the type of stapler to be used with the stapler. The recommendation may include statistics regarding success rates of new operating parameters in similar surgical procedures. In another example, the auxiliary information 11034 may also include one or more warnings 11038 regarding the status of tissue being manipulated by the surgical device. Warning 11038 may include recommended remedial steps that may be used to solve the problem. As an example, warning 11038 may indicate that the staple line is applying pressure to the tissue, which may result in tearing or incomplete healing at the staple line. It may be recommended to use a buttress material to help seal the tissue.

FIG. 14 is a flow chart 11500 depicting a method by which the interactive surgical system may receive surgical procedure information and suggest the next procedure steps. In the first step of process 11510, the surgeon has completed surgical step a. Data related to surgical step a may be sent 11512 from the communication hub to a cloud-based data source. An artificial intelligence engine in the cloud-based data source can predict 11514 the next step in the surgical procedure and send the data to the communication hub. The communication hub may then predict 11516 the location in the surgical field and the surgical device that may be used by displaying one or more virtual objects on the augmented reality display. The communication hub may then communicate 11518 with a cloud data source to determine the best surgical outcome of the predicted procedure, location, and device based on parameters associated with the patient. These parameters may be compared to similar patient statistics, surgical information, and surgical device characteristics in a cloud-based database. The hub may communicate 11520 to the surgeon the parameters related to use, orientation, location, and device parameters as virtual objects displayed on her or his associated augmented reality display device. The surgeon may then complete 11522 the next surgical step recommended. The method may continue in this manner for each subsequent set of surgical procedures.

Device and interaction association appliance and user hub/network sensing

It will be appreciated that computer-implemented interactive surgical systems constantly acquire device position and usage data during a procedure. The interactive surgical system may also continuously receive visual tracking or imaging information of various intelligent surgical instruments or other devices with respect to the patient anatomy. The surgical system may also retain imaging information throughout the surgical procedure. Through connection with the cloud-based system, the surgical system may also retain imaging information of the patient anatomy from a previous surgical procedure, or imaging information from a related surgical procedure from a different patient anatomy.

The augmented reality interactive surgical system includes a plurality of data-connected intelligent surgical devices. The surgical system can dynamically determine which devices are in use and where those devices are in space relative to each other and relative to critical structures and anatomical structures identified by the system. Based on the location of these devices and the user's preferences/locations, the system may prioritize data communications. Data communication prioritization may be enabled for devices that are near critical structures, and may include increased alarm sensitivity at critical procedure steps and/or around specific anatomical structures. In response to data communication prioritization, the augmented reality display may be quickly updated to notify a surgeon or other member of the operating room staff of the device location, high risk areas, and other sensitive locations.

Via spatial tracking, the interactive surgical system may adjust the augmented reality display to accommodate procedure steps, such as highlighting key structures when the instruments are nearby, or setting an alarm if the instruments are too close to each other. Thus, the system keeps track of the augmented reality information, but only displays the information when it is important to the surgeon. For example, augmented reality visualization of hidden structures may not always be enabled, but may be triggered by the location of the smart medical device. In this way, the surgeon may continue surgery as usual until a high risk area is identified or there is a risk of unintended damage. In such cases, augmented reality visualization may be enabled and the surgeon notified of any impending problems.

In one example, the interactive surgical system may identify the location of critical anatomical structures (such as ureters) that may not otherwise be visible to the surgeon. For example, an artificial intelligence module in a cloud system can include an anatomical model that can take images of a surgical field and predict or estimate the location of nearby or underlying anatomical structures. Although the ureter may not be readily visible, images of the ureter may appear as augmented reality virtual objects on one or more augmented reality displays. In one option, such augmented reality virtual objects may be displayed when the system detects/predicts that the end effector of the connected device has reached within a specified distance of the critical structure.

In another example, if the temperature reaches a level and the ultrasonic instrument is near the intestinal wall, the augmented reality virtual object may include a highlighting overlaid on a display of an end effector of the ultrasonic instrument.

The alternative display options may be based on the surgeon's preference. Some options may include a persistent display of augmented reality visualizations of key structures. Alternatively, the display of the augmented reality visualization of the critical structures may be enabled only during a particular portion of the procedure or only when the energy device is in use. These visualization options may depend on a combination of the position of the monitoring device relative to the patient anatomy (via, for example, a scope, scan), processing this information in an interactive surgical system, and achieving a desired augmented reality display based on the surgical context.

By way of example, fig. 15A depicts a surgical display 11040 obtained from a surgical imaging device during a laparoscopic procedure. The procedure may include the use of an ultrasonic cutter 11042 and an auxiliary tissue clamp 11044. The surgeon may wish to grasp the tissue mass 11046 with the tissue clamp 11044 and resect a portion thereof using the ultrasonic cutter 11042. The surgeon may not be aware that a portion of the patient's ureter is immediately below tissue 11046.

Fig. 15B depicts an augmented reality image 11050 including a surgical display 11040 and an augmented reality virtual object 11056 that presents the outline of the ureter beneath the tissue to be resected. In some aspects, the augmented reality image 11050 may display auxiliary information 11057 in addition to the augmented reality virtual object 11056. For example, the auxiliary information 11057 may include a tissue-related warning 11058 of the position of the ultrasonic cutter 11052 or auxiliary tissue clamp 11054 too close to the ureter. The auxiliary information 11057 may include device-related warnings 11059 that the ultrasonic cutter 11052 may be too close to the location of the auxiliary tissue clamp 11054.

The augmented reality display may be used to provide any relevant guidance to the surgeon regarding the procedure or the device used in the procedure. For example, if the interactive surgical system determines that the surgeon, i.e., the trained surgeon, is inexperienced based on the skill history of the surgeon, the trained surgeon (such as the surgeon resident) may receive a greater amount of feedback in the user training mode. The amount of feedback presented may be ranked based on a "training curve" related to the skill level of the surgeon, and ranking may be expedited if the surgeon shows improvement in learned skills. Feedback can be tailored to provide guidance in areas where skill improvement is desired. The personalized feedback may be based on data and images stored in the cloud system of the individual surgeon based on past performance and surgical experience and device usage recorded by the surgeon during past surgical procedures. Examples of surgical results that may indicate a need for technical improvement may include surgical site bleeding, double burns of cauterized tissue, or tissue marking.

The augmented reality display may also be used to recommend a particular device to a surgeon during a procedure. Improved or updated devices may be recommended to replace the surgical device being used during the procedure. Information indicating how to use such improved devices in current surgery may be provided in an augmented reality display. The augmented reality display may also provide statistics regarding the results of similar surgical procedures performed using the recommended device as compared to the results of surgical procedures using the present device.

Linking users to form an AR ecosystem

It will be appreciated that computer-implemented interactive surgical systems constantly acquire device position and usage data during a procedure. The interactive surgical system may also continuously receive visual tracking or imaging information of various intelligent surgical instruments or other devices with respect to the patient anatomy. The augmented reality interactive surgical system includes a plurality of data-connected intelligent surgical devices and one or more display devices configured to provide information to members of a surgical team regarding surgical operation, patient status, and operation of the intelligent surgical devices used throughout. The surgical system can dynamically determine which devices are in use and where those devices are in space relative to each other and relative to critical structures and anatomical structures identified by the system.

In some aspects, each member of the surgical team may be associated with one or more display devices to provide information related to the surgical procedure. Each display device may display images obtained from one or more imaging devices and an augmented reality virtual object superimposed on the imaging data. The display associated with any member of the surgical team may be customized according to the functional roles of the surgical team members. For example, the display of the members of the surgical team may be customized to include virtual objects associated with information or instrumentation controlled by the members of the surgical team. The interactive surgical system may monitor instruments and devices under the control of each surgical team member in the operating room. The data displayed on each display device may depend on which user controls the surgical device and the surgical role of that user. The display information for the user may change as instruments or devices enter or leave their control. For example, the surgical system may track instrument exchanges between surgeons or between surgeons and nurses. The augmented reality display may adjust the nature, type, and/or location of virtual objects in the augmented reality display associated with the affected surgical team member. For example, when a surgical team member relinquishes control of a surgical device, a virtual object associated with control of the surgical device may disappear from a display associated with she or he. Similarly, when a second surgical team member accepts control of the surgical device, a virtual object associated with control of the surgical device may appear on a display associated with her or him.

In some aspects, the intelligent surgical system may be capable of determining in-situ and in-vitro aspects of the instruments in use and associated users controlling those instruments. The surgical system may also be capable of correlating actions occurring outside the body with movements occurring inside the body to verify the correct correlation of the two aspects.

As disclosed above, each member of the surgical team may be associated with her/his own display device configured to be able to display an augmented reality display tailored to the activities and roles of the surgical team members. Such displays may include any suitable type of display, including a primary or primary operating room display, one or more secondary operating room displays, a display associated with one or more tablet devices, a laptop display, a smart phone display, or a display associated with a separate surgical device (such as a patient monitoring device or anesthesia delivery/monitoring device). The purpose of each of these devices is to provide information tailored to the individual's functional role.

Not only is the display customized for a particular surgical team member, but the team member can also modify her or his display to show the display of another member in the team, such as by sliding a touch activation screen, using a gesture, or by verbal command. In this way, a team member may be able to "pull" the display from the display device of another team member. Alternatively, some team members may have the right to "push" their own display onto the display device of other members of the surgical team.

In some aspects, each member may not have a physical display device. In contrast, multiple members of a surgical team may rely on shared or common display devices. In this case, the customized experience may be derived from a wearable image filtering device such as a glasses, heads-up display, or contact lens. The common display device may display an image including virtual objects associated with all members of the surgical team. The virtual objects may be color coded or otherwise visually coded such that each member of the surgical team may be able to view only those virtual objects associated with she or he by using the wearable image filtering device. The wearable image filtering device may filter images displayed on the public display device based on color filtering, polarization filtering, or temporal filtering of the rapid change imaging. Color and polarization filtering may allow a user to see only light emitted at a preselected wavelength or polarization state. The temporal filtering may coordinate the timing of the blanking condition of the wearable filter with the time that the particular image is displayed on the common display device. Color filter contact lenses or goggles may be used for rapid prototyping and information-level information collection. Alternatively, a zoom feature or UV light activation option may be incorporated into the wearable filter.

The interactive surgical system may also manage the priority of communications between members of the surgical team and/or with the hub or cloud system. Thus, communications caused by functions or activities deemed critical to the procedure may take precedence over communications such as may be associated with conventional patient monitoring at any given time. These communications may be prioritized among members of the surgical team. Thus, the relevant data generated by the critical devices used during a particular portion of the procedure can be communicated directly to all relevant members of the surgical team to communicate device useful information. In one non-limiting example, the surgeon may always need to know if the ultrasonic blade is hot, but the anesthesiologist may only need to know when the hot blade is in contact with the critical anatomy. In this case, the anesthesiologist may be notified by vibration feedback associated with the temperature of the ultrasonic blade only when the hot blade contacts the critical anatomy. The surgeon may receive tactile feedback (e.g., an optical vibration response) from the hot blade and receive notification/optical vibration as the blade as a whole approaches critical structures. As another example, devices and/or procedures that cause hemostasis difficulties may be prioritized. Data specific portions of the procedure (e.g., the type and source of blood flow) that may be needed to monitor hemostasis may be shared between surgical teams.

While each member of the surgical team may have her or his own customized augmented reality display, in some cases, separate display options may be overridden. As one example, during an emergency situation, everyone may see the same display or obtain the same priority alert based on the detected situation. Each member of the surgical team may obtain a standard, non-negotiable master setting or view. However, each individual may add additional settings and preferences to the standard image.

In some aspects, communications between particular members of the surgical team may be prioritized to result in pairing information with particular members for direct communication. In one example, the attending surgeon may share what she or he is seeing, and her or his display/wearable preferences may be extended to members of the surgical team that are specifically selected. For example, the attending surgeon may "push" her or his display to the surgeon resident so that the surgeon resident can see or feel the same thing as the attending surgeon. A ping system with a wearable device may inform other surgical team members to switch their corresponding augmented reality displays. As one example, a surgeon or physician assistant may require an anesthesiologist to switch their view/preference to the surgeon's view/preference. It will be appreciated that surgical team members contacted in this manner may decline to invite if there is a higher priority task at hand.

The surgical system may initiate a communication pairing with the intelligent surgical device and then associate a user of the device with the device while the device is active. The above discloses that the intelligent surgical device may be controlled by a member of a surgical team. After the interactive surgical system has established a communication pairing with the device itself, the interactive surgical system may identify the association of the person with the device. The system may first determine that the device is located within a surgical suite. The system may then identify when the device is removed from its sterile packaging, thereby being passively available for team member selection. Once the surgical team member begins operating the device, the interactive surgical system may then identify the status of the intelligent device as being actively selected, and the system may then identify the control or association between the device and the surgical team member.

Monitoring device interactions with a user and changes in display requirements

The interactive surgical system may determine the task of the members of the surgical team based on one or more of situational awareness of the state of the intelligent surgical instrument, the functional roles of the team members, and steps in the procedure. This situational awareness can be used to adapt augmented reality virtual object display information based on the task at hand.

In one non-limiting example, a surgeon may deploy a loaded stapler positioned on an organ to perform a transection. The augmented reality display may include a virtual object that displays information related to one or more of detected firing force (FTF), latency, and tissue tension, as well as bin color, status, and travel position of the smart medical device. Once firing is complete, the surgeon may release tissue, close, and remove the device. Since the staple cartridge of the device is now depleted, the augmented reality display may indicate that the stapler cannot be reused prior to reloading. The augmented reality display may use the colored virtual object as an overlay over the image of the stapler to indicate that it is not available for additional use before reloading the staples. When the stapler is handed over to the scrub nurse, the virtual object representing the stapler can be transferred to the nurse's augmented reality display. At the same time, the virtual object representing the stapler may be removed from the surgeon's augmented reality display to indicate that control of the device has been transferred to the nurse. The nurse's augmented reality display may then indicate the firing status of the stapler and indicate the steps required to reload the instrument with a new unfired cartridge. Such virtual objects may include indications of buttons of the stapler to be pressed, their order, and suggestions for the next bin color based on awareness of the procedure plan and the steps the intelligent medical device is currently in between. In some aspects, the nurse's augmented reality display may also be linked to other displays to show the location of the desired bins, even to show the compatibility of the bins being loaded with the devices and protocols.

In some aspects, the interactive surgical system may track aspects of the operating room procedure. In some examples, the interactive surgical system may track the use and control of multiple intelligent surgical devices used during a procedure, the location and activity of surgical team members, and the settings of equipment and patients within the operating room itself. Additional trackable aspects of the interactive surgical system can include surgical access points and registering them with the patient, instrument position, orientation or status, or lost or misplaced device. In some additional aspects, the interactive surgical system can identify surgical procedure steps in the procedure. The surgical system may display virtual objects with background highlighting on one or more augmented reality displays to indicate critical or time sensitive steps or to indicate that a particular step is at a higher risk level than other steps.

In some aspects, the interactive surgical system may track skills or capabilities of members of a surgical team in an operating room. In addition, the interactive surgical system may track the individual locations of members of the surgical team in the operating room and their functions. The location of personnel entering and leaving the operating room can also be tracked. The interactive surgical system may monitor surgical staff movements, interactions, and movements in the operating room to improve the layout. Aspects related to the use of intelligent surgical devices may be tracked, such as hand advantages of team members using such devices, or placing surgical devices on a workstation before or after use to optimize efficiency.

FIG. 16 depicts various aspects associated with a surgical procedure that can be tracked by an interactive surgical system and analyzed to formulate an optimization strategy. Patient-related data may include patient location, pre-operative and post-operative scheduling. The material data may include a list and location of consumables (such as gauze and wet tissues) and their management in a hospital supply chain. The location related data may include not only the operating room but also auxiliary rooms such as storage facilities and work sites. Control matters may include rules, regulations, and laws related to hospital business practices, medical, political, economic, and legal constraints on the practices. Functional data is all data related to the various steps, processes and sub-processes involved in a surgical procedure. The flow data encompasses the actual sequence of procedures and processes, including models. The operational data relates to the application of various surgical equipment, devices and tools. Information items include all data acquired during a procedure, the documentation of the procedure, and the data model. Finally, the organization data may include human resource data from the hospital, including staff identity, surgical role, and organization model of the hospital itself.

Fig. 17 depicts aspects of an operating room 11100 that may be modeled for tracking purposes. The depicted operating room 11100 may generally represent only an operating room, and the layout may be suitable for any real world example of an operating room. The operating room 11100 has a portal 11102 through which patients, surgical team members, and equipment can enter. The operating room may have an outlet 11116 through which used equipment may flow for disposal or recycling. There may be a central surgical room 11118 that includes a sterile field 11108 for patients and surgeons and an anesthetic field 11110 for anesthesiologists and anesthesiologists. The central surgical room 11118 may also include the location of any surgical robots involved in the surgical procedure. Surrounding the central surgical room 11118 may be a circulation zone 11106 in which nurses, technicians, and other personnel may move so as not to interfere with the surgical procedure performed in the surgical room 11118. The cartographic region 11104 where clinical records can be made is accessible through the circulation region 11106. Outside of the operating room 11100 may be a perimeter corridor 11112 that may include a scrub station 11114 for surgical team members to wash their hands and wear their personal protective equipment.

The interactive surgical system may track changes in posture of members of the surgical team, such as rocking back and forth while standing. Such postural activity may be an indication of an increased level of fatigue of members of the surgical team. In response, the interactive surgical system may display a virtual object associated with the tired member of the surgical team on the augmented reality display. The interactive surgical system may alert the support staff member that it may be possible to allow the current staff member or surgeon to rest by displaying the appropriate virtual object on the support staff member's corresponding augmented reality display. In one aspect, the virtual object may be an indicator of the fatigue status of team members. An example of such virtual objection warning of surgical member fatigue 11060 is depicted in fig. 18.

As part of the ability of the interactive surgical system to track aspects of the operating room procedure, the interactive surgical system may include optimization algorithms to improve performance during the surgical procedure. Such optimizations may include optimizations of the correct surgical tool used at the correct time, such as determining that all required surgical tools are available and conveniently placed before the procedure begins. The system may determine an optimized operating room layout of the device, optimized patient placement, or optimized access to the device or operating room door.

Upon completion of the procedure, the interactive surgical system may analyze and suggest changes in the operating room procedure to minimize areas of inefficiency. This analysis may affect the protocol planning of all future comparable protocols. The analysis may be tailored for each unique operation. The size, shape, number of staff members, entrance and exit arrangements, location of supplies in the operating room, and external materials required to access the operating room can all be analyzed by the interactive surgical system based on the movements and actions of the surgical team members during the surgical procedure and the outcome of the surgical procedure. The interactive surgical system may virtually run multiple scenarios to determine the "best" workflow. Future similar protocols may use updated procedures to improve efficiency and reduce fatigue. For example, improved logistical and surgical efficiency may be indicated by enabling virtual objects on the augmented reality display. Such virtual objects may include a graphical overlay over an image of the device layout, device location, and patient layout to visualize the flow and utilization of the product.

Some exemplary illustrations of an optimized operating room are depicted in fig. 19A, 19B, and 19C. Each of the illustrations 11200a, 11200b, 11200c depicts a possible organization of components of an operating room, including one or more support zones 11202a, 11202b, 11202c, transition zones 11204a, 11204b, 11204c (corresponding to the circulation zone 11106 in fig. 17), and one or more supply zones 11206a, 11206b, 11206c. The anesthetic zones are shown at 11208a, 11208b, 11208c, and the surgical tables 11210a, 11210b, 11210c are located in a central surgical room (11118 in fig. 17). In these depictions, surgical tables 11210a, 11210b, 11210c are subdivided into right, left, and foot regions.

After the surgery is completed, the superimposed layers are projected to indicate what treatment method should be used for the various instruments and materials in the operating room. In some cases, the cells may be directed into their own waste stream. The disposable portion of the used medical device or smart medical device may be directed to a medical waste disposal area. Unused material may be directed back into inventory or left in the operating room. The packaging of the medical device or disposable medical supplies may be tracked to ensure proper recovery, reuse, or disposal.

Customization of virtual object data based on device ownership and task requirements

In some aspects, the augmented reality display may be customizable by its associated user to enable display of virtual objects related to the intelligent surgical instrument under the control of the user. The interactive surgical system may monitor the status of the intelligent surgical instrument under the control of each member of the surgical team in the operating room. The interactive surgical system may then control the display of one or more virtual objects on an augmented reality display associated with the surgical team member as a result of both the device under the control of the surgical team member and the task or situation in which the team member is acting.

The interactive surgical system may track which surgical team member is using which augmented reality display device. This may include conventional monitors, but may also include augmented reality glasses, wearable devices, auxiliary displays, displays on the instrument, and stationary device control displays. Then, using its understanding of the instrument being used by the surgical team member, the interactive surgical system may adjust the display of virtual objects having utility to the hand task to the display associated with the surgical team member. In one aspect, such virtual objects may be displayed on an augmented reality display of a surgical team member controlling the intelligent surgical device. Alternatively, such virtual objects may be displayed on an augmented reality display of several or all members of the surgical team. This may occur for all users in the OR and all instruments and displays they each use simultaneously. In some aspects, the virtual object may appear as a color highlighting of the image surrounding the surgical device. Highlighting may appear as a contour or color overlay displayed over the image of the surgical device. Other virtual objects may include a secondary window display that displays a text message or secondary image containing tissue visible in the surgical field. In some other aspects, the secondary window display may include images derived from models calculated by artificial intelligence modules in the cloud system. These models may include images that hide invisible tissue in the surgical field of view or alternate views of the intelligent medical instrument used by the surgeon.

In some aspects, the augmented reality display may be customizable by its associated user to enable display of virtual objects related to the status of the intelligent surgical device under the control of the associated user or another surgical team member. The state may include a device powered on, it is performing one of several types of functions, a power level of the device, a state of an auxiliary component (such as a pin), an error condition, and similar functional states.

In some aspects, the augmented reality display may be customizable by its associated user to enable display of virtual objects related to an event, such as application of another surgical device to a patient. The virtual object may display a counter or timer, such as a linear counter or a cyclic counter. A timer may be used to time events.

In some aspects, the augmented reality display may be customizable by its associated user to enable display of virtual objects related to the state of the paired device. In one example, the virtual object may be a highlighting superimposed on the stapler, the color or intensity of which is related to the energy level of the stapler. In some examples, highlighting may be applied to the augmented reality display only when the stapler is controlled by a surgeon associated with the display device.

In some aspects, the augmented reality display may be customizable by its associated user to enable display of virtual objects related to the state of the active device. The intelligent surgical device located in the operating room may be in active communication (active pairing) with an interactive surgical system, communication hub, cloud system, or other equipment. The surgical device may or may not be actively used by the surgeon, but may simply be placed on the meo stent for use. The device actively used by the surgeon may be the device that the surgeon is currently using or holding. The device used by the surgeon may be the device currently used by the surgeon to perform certain actions in the surgical field.

In some aspects, virtual objects related to the state of the active device may disappear from the display after a fixed non-use time. In some aspects, a virtual object related to the state of an active device may display device-related information only when the device is active or in use.

In some aspects, an augmented reality display may be customized by its associated user based on user input or requests to enable the display. Depending on the technical capabilities of the augmented reality display, input may be received from a keyboard, mouse, verbal commands, gestures, tactile input on a touch-sensitive screen, a stylus, or any other information input device.

It will be appreciated that a plurality of augmented reality display devices may be used within an operating room. One or more primary widescreen displays are available to all members of the surgical team. Alternatively, other display type devices may be associated with each of the members of the surgical team by the interactive surgical system. Such devices may include one or more laptop devices, tablet devices, or wearable devices such as augmented reality headsets. Flat panel display devices may be different from larger display devices within a standard OR. If the interactive surgical system determines or the user indicates that they are using a flat screen, the virtual objects displayed on the augmented reality display device may be adjusted to accommodate smaller screens in terms of display position, aspect ratio, color, or other visual design. The user may determine which virtual objects exist and which virtual objects should be excluded.

Surgical team members may interact with a portion of their associated augmented reality display to determine where to display a particular virtual object. In other aspects, the surgical team member may scan, take a photograph of, or enter a classifier that will specify the display of the overlay, configuration, or location. In other aspects, the surgical team member may interact with a portion of the augmented reality display via a separate device (e.g., a wearable device) in order to identify a predetermined configuration or input to customize the layout of the augmented reality display associated with the user. In other aspects, the user's audio or visual source may be coupled with the instrument they control. In some other aspects, virtual objects showing links or interactive displays of multiple instruments may be displayed on multiple augmented reality displays or on a primary or main operating room display along with a more detailed summary than when each of the separate displays are separately displayed.

While individual members of a surgical team may be able to customize the display of virtual objects on their associated augmented reality displays, the display information for a particular surgical team member may change as intelligent surgical instruments enter or leave their control. In some aspects, the interactive surgical system tracks instrument exchanges and ownership changes between members of the surgical team, and may adjust the augmented reality display data based on the ownership changes. Furthermore, the interactive surgical system is able to determine in-situ and in-vitro aspects of the instruments in use and the associated users controlling those instruments. The interactive surgical system may correlate actions occurring outside the body with movements occurring inside the body to verify the correct correlation of the two aspects.

Various additional aspects of the subject matter described herein are set forth in the following numbered embodiments:

Example 1: a method of distributing information among members of a surgical team, the method comprising: receiving, by a modular control tower, imaging data from a plurality of imaging devices; receiving, by the modular control tower, device-related data from each of a plurality of intelligent surgical instruments; associating, by the modular control tower, a display device with a member of the surgical team; defining, by the modular control tower, a functional role of the member of the surgical team; and displaying, by the modular control tower, an augmented reality display through the display device, wherein the augmented reality display on the display device includes a virtual object based on the imaging data, the device-related data, the functional role of the member of the surgical team, and a surgical activity of the member of the surgical team.

Example 2: the method of embodiment 1, wherein receiving, by the modular control tower, the device-related data comprises controlling, by the modular control tower, one or more intelligent surgical instruments of the plurality of intelligent surgical instruments by receiving, by the modular control tower, data defining a member of the surgical team.

Example 3: the method of embodiment 2, further comprising displaying, by the modular control tower, virtual objects associated with control functions of the one or more of the plurality of intelligent surgical instruments by the member of the surgical team on an augmented reality display associated with the member of the surgical team controlling the one or more of the plurality of intelligent surgical instruments.

Example 4: the method of embodiment 3, further comprising: determining, by the modular control tower, in-situ and in-vitro aspects of one or more of the plurality of intelligent surgical instruments controlled by the member of the surgical team; and displaying, by the modular control tower, virtual objects associated with the in-situ aspect of the one or more of the plurality of intelligent surgical instruments on the display device associated with the member of the surgical team controlling the one or more of the plurality of intelligent surgical instruments.

Example 5: the method of embodiment 4, further comprising correlating, by the modular control tower, the in-situ aspect of the one or more of the plurality of intelligent surgical instruments with the in-vitro aspect of the one or more of the plurality of intelligent surgical instruments.

Example 6: the method of embodiment 3, further comprising altering, by the modular control tower, the virtual object displayed on the augmented reality display associated with the member of the surgical team controlling the one or more of the plurality of intelligent surgical instruments when the member of the surgical team relinquishes control of the one or more of the plurality of intelligent surgical instruments.

Example 7: the method of embodiment 6, further comprising altering, by the modular control tower, an augmented reality display on a display device associated with a member of the surgical team receiving control of the one or more of the plurality of intelligent surgical instruments.

Example 8: the method of embodiment 1, further comprising causing, by a first member of the surgical team, a display device associated with a second member of the surgical team to display an augmented reality display associated with the first member of the surgical team.

Example 9: the method of embodiment 1, further comprising causing, by a first member of the surgical team, a display device associated with the first member of the surgical team to display an augmented reality display associated with a second member of the surgical team.

Example 10: the method of embodiment 1, further comprising adjusting, by the member of the surgical team, the virtual object of the augmented reality display associated with the member of the surgical team.

Example 11: the method of embodiment 1, further comprising adjusting, by the member of the surgical team, an aspect of the virtual object displayed on an augmented reality display associated with the member of the surgical team.

Example 12: the method of embodiment 11, wherein adjusting aspects of the virtual objects displayed on the augmented reality display includes adjusting a position of one or more of the virtual objects on the augmented reality display.

Example 13: an automated surgical system, comprising: a modular control tower; a plurality of imaging devices in data communication with the modular control tower; a plurality of intelligent surgical instruments; and a plurality of display devices in data communication with the modular control tower. Each display device of the plurality of display devices is associated with one or more members of a surgical team through the modular control tower, and each member of the one or more members of the surgical team is defined by a functional role. The modular control tower includes a controller in data communication with one or more memory components configured to store instructions that, when executed by the controller, cause the controller to: receiving imaging data from the plurality of imaging devices; receiving device-related data from each of the plurality of intelligent surgical instruments; and displaying an augmented reality display on each of the plurality of display devices. The augmented reality display on a designated display device may include a virtual object based on the imaging data, the device-related data, the functional role of a designated member of the surgical team associated with the designated display device, and a surgical activity of the designated member of the surgical team.

Example 14: the system of embodiment 13, wherein the augmented reality display on each of the plurality of display devices is the same.

Example 15: the system of embodiment 13, wherein the augmented reality display of the designated display device is customizable by the designated member of the surgical team associated with the designated display device.

Example 16: the system of embodiment 13, wherein the virtual object of the augmented reality display depends on a type of the one or more display devices.

Example 17: the system of embodiment 13, wherein the virtual object of the augmented reality display on the designated display device is dependent on one or more intelligent surgical instruments controlled by the designated member of the surgical team.

Example 18: the system of embodiment 17, wherein the virtual object of the augmented reality display changes when the designated member of the surgical team relinquishes control of the one or more intelligent surgical instruments.

Example 19: the system of embodiment 17, wherein the virtual object of the augmented reality display on the designated display device comprises a prediction of a second surgical activity by the designated member of the surgical team.

Example 20: the system of embodiment 17, wherein the virtual object of the augmented reality display is dependent on a distance of the one or more intelligent surgical instruments from a critical anatomy of a patient.

Example 21: the system of embodiment 17, wherein the virtual object of the augmented reality display indicates that the one or more intelligent surgical instruments controlled by the designated member of the surgical team are improperly used or are not correct for the surgical activity.

Example 22: the system of embodiment 17, wherein the virtual object of the augmented reality display depicts a view of the one or more intelligent surgical instruments that is not visible to the designated member of the surgical team.

While various forms have been illustrated and described, it is not the intention of the applicant to restrict or limit the scope of the appended claims to such detail. Many modifications, variations, changes, substitutions, combinations, and equivalents of these forms may be made by those skilled in the art without departing from the scope of the disclosure. Furthermore, the structure of each element associated with the described form may alternatively be described as a means for providing the function performed by the element. In addition, where materials for certain components are disclosed, other materials may be used. It is, therefore, to be understood that the foregoing detailed description and the appended claims are intended to cover all such modifications, combinations, and variations as fall within the scope of the disclosed forms of the invention. The appended claims are intended to cover all such modifications, variations, changes, substitutions, modifications and equivalents.

The foregoing detailed description has set forth various forms of the apparatus and/or methods via the use of block diagrams, flowcharts, and/or examples. Insofar as such block diagrams, flowcharts, and/or examples contain one or more functions and/or operations, it will be understood by those within the art that each function and/or operation within such block diagrams, flowcharts, or examples can be implemented, individually and/or collectively, by a wide range of hardware, software, firmware, or virtually any combination thereof. Those skilled in the art will recognize that some aspects of the forms disclosed herein may be equivalently implemented in integrated circuits, as one or more computer programs running on one or more computers (e.g., as one or more programs running on one or more computer systems), as one or more programs running on one or more processors (e.g., as one or more programs running on one or more microprocessors), as firmware, or as virtually any combination thereof, and that designing the circuitry and/or writing the code for the software and/or hardware would be well within the skill of one of skill in the art in light of this disclosure. In addition, those skilled in the art will appreciate that the mechanisms of the subject matter described herein are capable of being distributed as a program product or products in a variety of forms, and that an illustrative form of the subject matter described herein applies regardless of the particular type of signal bearing media used to actually carry out the distribution.

Instructions for programming logic to perform the various disclosed aspects can be stored within a memory in a system, such as Dynamic Random Access Memory (DRAM), cache, flash memory, or other memory. Furthermore, the instructions may be distributed via a network or by other computer readable media. Thus, a machine-readable medium may include any mechanism for storing or transmitting information in a form readable by a machine (e.g., a computer), but is not limited to floppy diskettes, optical disks, compact disk read-only memories (CD-ROMs), and magneto-optical disks, read-only memories (ROMs), random Access Memories (RAMs), erasable programmable read-only memories (EPROMs), electrically erasable programmable read-only memories (EEPROMs), magnetic or optical cards, flash memory, or a tangible, machine-readable storage device for use in transmitting information over the internet via electrical, optical, acoustic, or other form of propagated signals (e.g., carrier waves, infrared signals, digital signals, etc.). Thus, a non-transitory computer-readable medium includes any type of tangible machine-readable medium suitable for storing or transmitting electronic instructions or information in a form readable by a machine (e.g., a computer).

As used in any aspect herein, the term "control circuitry" may refer to, for example, hardwired circuitry, programmable circuitry (e.g., a computer processor including one or more individual instruction processing cores, processing units, processors, microcontrollers, microcontroller units, controllers, digital Signal Processors (DSPs), programmable Logic Devices (PLDs), programmable Logic Arrays (PLAs), field Programmable Gate Arrays (FPGAs)), state machine circuitry, firmware storing instructions executed by the programmable circuitry, and any combination thereof. The control circuitry may be implemented collectively or individually as circuitry forming part of a larger system, such as an Integrated Circuit (IC), an Application Specific Integrated Circuit (ASIC), a system-on-a-chip (SoC), a desktop computer, a laptop computer, a tablet computer, a server, a smart phone, or the like. Thus, as used herein, "control circuitry" includes, but is not limited to, electronic circuitry having at least one discrete circuit, electronic circuitry having at least one integrated circuit, electronic circuitry having at least one application specific integrated circuit, electronic circuitry forming a general purpose computing device configured by a computer program (e.g., a general purpose computer configured by a computer program that at least partially implements the methods and/or apparatus described herein, or a microprocessor configured by a computer program that at least partially implements the methods and/or apparatus described herein), electronic circuitry forming a memory device (e.g., forming a random access memory), and/or electronic circuitry forming a communication device (e.g., a modem, communication switch, or optoelectronic device). Those skilled in the art will recognize that the subject matter described herein may be implemented in analog or digital fashion, or some combination thereof.

As used in any aspect herein, the term "logic" may refer to an application, software, firmware, and/or circuitry configured to be capable of performing any of the foregoing operations. The software may be embodied as software packages, code, instructions, instruction sets, and/or data recorded on a non-transitory computer readable storage medium. The firmware may be embodied as code, instructions or a set of instructions and/or data that are hard-coded (e.g., non-volatile) in a memory device.

As used in any aspect herein, the terms "component," "system," "module," and the like can refer to a control circuit, a computer-related entity, hardware, a combination of hardware and software, or software in execution.

As used in any aspect herein, an "algorithm" refers to an organized sequence of steps leading to a desired result, wherein "step" refers to the manipulation of physical quantities and/or logical states, which may, but need not, take the form of electrical or magnetic signals capable of being stored, transferred, combined, compared, and otherwise manipulated. Are often used to refer to signals such as bits, values, elements, symbols, characters, terms, numbers, or the like. These and similar terms may be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities and/or conditions.

The network may comprise a packet switched network. The communication devices may be capable of communicating with each other using the selected packet switched network communication protocol. One exemplary communication protocol may include an ethernet communication protocol that may be capable of allowing communication using transmission control protocol/internet protocol (TCP/IP). The ethernet protocol may conform to or be compatible with the ethernet Standard and/or a higher version of the Standard, titled "IEEE 802.3Standard" published by the Institute of Electrical and Electronics Engineers (IEEE) at month 12 of 2008. Alternatively or additionally, the communication devices may be capable of communicating with each other using an x.25 communication protocol. The x.25 communication protocol may conform to or be compatible with standards promulgated by the international telecommunications union telecommunication standardization sector (ITU-T). Alternatively or additionally, the communication devices may be capable of communicating with each other using a frame relay communication protocol. The frame relay communication protocol may conform to or be compatible with standards promulgated by the international telegraph and telephone Consultation Committee (CCITT) and/or the American National Standards Institute (ANSI). Alternatively or additionally, the transceivers may be capable of communicating with each other using an Asynchronous Transfer Mode (ATM) communication protocol. The ATM communication protocol may conform to or be compatible with the ATM standard promulgated by the ATM forum at month 8 of 2001 under the name "ATM-MPLS Network Interworking 2.0.0" and/or a higher version of that standard. Of course, different and/or later developed connection oriented network communication protocols are likewise contemplated herein.

Unless specifically stated otherwise as apparent from the above disclosure, it is appreciated that throughout the above disclosure, discussions utilizing terms such as "processing," "computing," "calculating," "determining," "displaying" or the like, refer to the action and processes of a computer system, or similar electronic computing device, that manipulates and transforms data represented as physical (electronic) quantities within the computer system's registers and memories into other data similarly represented as physical quantities within the computer system memories or registers or other such information storage, transmission or display devices.

One or more components may be referred to herein as "configured to be capable of", "operable/operative", "adapted/adaptable", "capable of", "conformable/conforming" and the like. Those skilled in the art will recognize that "configured to be capable of" may generally encompass active and/or inactive and/or standby components unless the context indicates otherwise.

The terms "proximal" and "distal" are used herein with respect to a clinician manipulating a handle portion of a surgical instrument. The term "proximal" refers to the portion closest to the clinician, and the term "distal" refers to the portion located away from the clinician. It will also be appreciated that for simplicity and clarity, spatial terms such as "vertical," "horizontal," "upper," and "lower" may be used herein with respect to the drawings. However, surgical instruments are used in many orientations and positions, and these terms are not intended to be limiting and/or absolute.

Those skilled in the art will recognize that, in general, terms used herein, and particularly in the appended claims (e.g., bodies of the appended claims) are generally intended to be "open" terms (e.g., the term "including" should be construed as "including but not limited to," the term "having" should be construed as "having at least," the term "comprising" should be construed as "including but not limited to," etc.). It will be further understood by those with skill in the art that if a specific number of an introduced claim recitation is intended, such an intent will be explicitly recited in the claim, and in the absence of such recitation no such intent is present. For example, as an aid to understanding, the following appended claims may contain usage of the introductory phrases "at least one" and "one or more" to introduce claim(s). However, the use of such phrases should not be construed to imply that the introduction of a claim recitation by the indefinite articles "a" or "an" limits any particular claim containing such introduced claim recitation to claims containing only one such recitation, even when the same claim includes the introductory phrases "one or more" or "at least one" and indefinite articles such as "a" or "an" (e.g., "a" and/or "an" should typically be interpreted to mean "at least one" or "one or more"); the same holds true for the use of definite articles used to introduce claim recitations.

In addition, even if a specific number of an introduced claim recitation is explicitly recited, those skilled in the art will recognize that such recitation should typically be interpreted to mean at least the recited number (e.g., the bare recitation of "two recitations," without other modifiers, typically means at least two recitations, or two or more recitations). Moreover, in those instances where a convention analogous to "at least one of A, B and C, etc." is used, in general such a construction is intended in the sense one having skill in the art would understand the convention (e.g., "a system having at least one of A, B and C" would include but not be limited to systems having only a, only B, only C, A and B together, a and C together, B and C together, and/or A, B and C together, etc.). In those instances where a convention analogous to "A, B or at least one of C, etc." is used, in general such a construction is intended in the sense one having skill in the art would understand the convention (e.g., "a system having at least one of A, B or C" shall include but not be limited to systems having only a, only B, only C, A and B together, a and C together, B and C together, and/or A, B and C together, etc.). It will be further understood by those within the art that, in general, unless the context indicates otherwise, disjunctive words and/or phrases presenting two or more alternative terms in the detailed description, claims, or drawings should be understood to encompass the possibility of including one of the terms, either of the terms, or both. For example, the phrase "a or B" will generally be understood to include the possibility of "a" or "B" or "a and B".

For the purposes of the appended claims, those skilled in the art will understand that the operations recited therein can generally be performed in any order. Additionally, while various operational flow diagrams are set forth in one or more sequences, it should be understood that various operations may be performed in other sequences than the illustrated sequences, or may be performed concurrently. Examples of such alternative ordering may include overlapping, staggered, interrupted, reordered, incremental, preparatory, supplemental, simultaneous, reverse, or other altered ordering unless the context dictates otherwise. Moreover, unless the context dictates otherwise, terms such as "responsive to," "related to," or other past-type adjectives are generally not intended to exclude such variants.

It is worth mentioning that any reference to "an aspect", "an example" means that a particular feature, structure or characteristic described in connection with the aspect is included in at least one aspect. Thus, the appearances of the phrases "in one aspect," "in an example," and "in an example" in various places throughout this specification are not necessarily all referring to the same aspect. Furthermore, the particular features, structures, or characteristics may be combined in any suitable manner in one or more aspects.

Any patent application, patent, non-patent publication, or other disclosure material referred to in this specification and/or listed in any application data sheet is incorporated herein by reference, as if the incorporated material was not inconsistent herewith. Accordingly, and to the extent necessary, the disclosure as explicitly set forth herein supersedes any conflicting material incorporated herein by reference. Any material, or portion thereof, that is said to be incorporated by reference herein, but which conflicts with existing definitions, statements, or other disclosure material set forth herein will only be incorporated to the extent that no conflict arises between that incorporated material and the existing disclosure material.

In summary, many of the benefits resulting from employing the concepts described herein have been described. The foregoing detailed description of one or more forms has been presented for purposes of illustration and description. It is not intended to be exhaustive or to limit the invention to the precise form disclosed. Modifications or variations of the present invention are possible in light of the above teachings. One or more of the forms selected and described are chosen to illustrate principles and practical application to thereby enable one of ordinary skill in the art to utilize various forms and various modifications as are suited to the particular use contemplated. The claims filed herewith are intended to define the full scope.

Claims (22)

1. A method of distributing data among members of a surgical team, the method comprising:

receiving, by a modular control tower, imaging data from a plurality of imaging devices;

receiving, by the modular control tower, device-related data from each of a plurality of intelligent surgical instruments;

Associating, by the modular control tower, a display device with a member of the surgical team;

defining, by the modular control tower, a functional role of the member of the surgical team; and

Displaying, by the modular control tower, an augmented reality display via the display device, wherein the augmented reality display on the display device includes a virtual object based on the imaging data, the device-related data, the functional role of the member of the surgical team, and a surgical activity of the member of the surgical team.

2. The method of claim 1, wherein receiving device-related data by the modular control tower comprises receiving data defining a member of the surgical team by the modular control tower, the member of the surgical team controlling one or more of the plurality of intelligent surgical instruments.

3. The method of claim 2, further comprising displaying, by the modular control tower, virtual objects associated with control functions of the one or more of the plurality of intelligent surgical instruments by the member of the surgical team on an augmented reality display associated with the member of the surgical team controlling the one or more of the plurality of intelligent surgical instruments.

4. A method according to claim 3, further comprising:

determining, by the modular control tower, in-situ and in-vitro aspects of one or more of the plurality of intelligent surgical instruments controlled by the member of the surgical team; and

Displaying, by the modular control tower, virtual objects associated with the in-situ aspect of the one or more of the plurality of intelligent surgical instruments on the display device associated with the member of the surgical team controlling the one or more of the plurality of intelligent surgical instruments.

5. The method of claim 4, further comprising associating, by the modular control tower, the in-situ aspect of the one or more of the plurality of intelligent surgical instruments with the in-vitro aspect of the one or more of the plurality of intelligent surgical instruments.

6. The method of claim 3, further comprising altering, by the modular control tower, the virtual object displayed on the augmented reality display associated with the member of the surgical team controlling the one or more of the plurality of intelligent surgical instruments when the member of the surgical team relinquishes control of the one or more of the plurality of intelligent surgical instruments.

7. The method of claim 6, further comprising altering, by the modular control tower, an augmented reality display on a display device associated with a member of the surgical team receiving control of the one or more of the plurality of intelligent surgical instruments.

8. The method of claim 1, further comprising causing, by a first member of the surgical team, a display device associated with a second member of the surgical team to display an augmented reality display associated with the first member of the surgical team.

9. The method of claim 1, further comprising causing, by a first member of the surgical team, a display device associated with the first member of the surgical team to display an augmented reality display associated with a second member of the surgical team.

10. The method of claim 1, further comprising adjusting, by the member of the surgical team, the virtual object of the augmented reality display associated with the member of the surgical team.

11. The method of claim 1, further comprising adjusting, by the member of the surgical team, an aspect of the virtual object displayed on an augmented reality display associated with the member of the surgical team.

12. The method of claim 11, wherein adjusting aspects of the virtual objects displayed on the augmented reality display comprises adjusting a position of one or more of the virtual objects on the augmented reality display.

13. An automated surgical system, comprising:

A modular control tower;

a plurality of imaging devices in data communication with the modular control tower;

A plurality of intelligent surgical instruments; and

A plurality of display devices in data communication with the modular control tower;

Wherein each display device of the plurality of display devices is associated with one or more members of a surgical team through the modular control tower and each member of the one or more members of the surgical team is defined by a functional role, wherein the modular control tower comprises a controller in data communication with one or more memory components configured to be capable of storing instructions that, when executed by the controller, cause the controller to:

receiving imaging data from the plurality of imaging devices;

receiving device-related data from each of the plurality of intelligent surgical instruments; and

Displaying an augmented reality display on each of the plurality of display devices, wherein the augmented reality display on a designated display device includes a virtual object based on the imaging data, the device-related data, the functional role of a designated member of the surgical team associated with the designated display device, and a surgical activity of the designated member of the surgical team.

14. The system of claim 13, wherein the augmented reality display on each of the plurality of display devices is the same.

15. The system of claim 13, wherein an augmented reality display of the designated display device is customizable by the designated member of the surgical team associated with the designated display device.

16. The system of claim 13, wherein the virtual object of the augmented reality display is dependent on a type of the one or more display devices.

17. The system of claim 13, wherein the virtual object of the augmented reality display on the designated display device is dependent on one or more intelligent surgical instruments controlled by the designated member of the surgical team.

18. The system of claim 17, wherein the virtual object of the augmented reality display changes when the designated member of the surgical team relinquishes control of the one or more intelligent surgical instruments.

19. The system of claim 17, wherein the virtual object of the augmented reality display on the designated display device comprises a prediction of a second surgical activity by the designated member of the surgical team.

20. The system of claim 17, wherein the virtual object of the augmented reality display is dependent on a distance of the one or more intelligent surgical instruments from critical anatomy of a patient.

21. The system of claim 17, wherein the virtual object of the augmented reality display indicates that the one or more intelligent surgical instruments controlled by the designated member of the surgical team are improperly used or are not correct for the surgical activity.

22. The system of claim 17, wherein the virtual object of the augmented reality display depicts a view of the one or more intelligent surgical instruments that is not visible to the designated member of the surgical team.