JP2024514885A - Mixed reality feedback systems working together to enhance efficient perception of composite data feeds - Google Patents

Abstract

An augmented reality display system and method for use during a surgical procedure is disclosed. An imaging device captures real images of a surgical area during a surgical procedure and generates a first data feed. A sensor device generates a second data feed. An augmented reality device including an augmented reality display generates an augmented overlay based on the first data feed and the second data feed. The augmented overlay includes a visual portion and a non-visual portion. A processor receives the first data feed and the second data feed and combines the first data feed and the second data feed to generate the augmented overlay.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS
This application claims the benefit under 35 U.S.C. §119(e) to U.S. Provisional Patent Application No. 63/174,674, entitled "HEADS UP DISPLAY," filed on April 14, 2021, and U.S. Provisional Patent Application No. 63/284,326, entitled "INTRAOPERAIVE DISPLAY FOR SURGICAL SYSTEMS," filed on November 30, 2021, the disclosures of each of which are incorporated by reference in their entireties herein.

The present disclosure relates to devices, systems, and methods for providing augmented reality interactive experiences during surgical procedures. During surgical procedures, objects present in the real world are sometimes enhanced by overlaying computer-generated sensory information across multiple sensory modalities, including visual, auditory, tactile, somatosensory, and olfactory senses. It would be desirable to provide an augmented reality interactive experience of a real world environment. In the context of this disclosure, images of the surgical field and surgical instruments and other objects appearing in the surgical field may include computer-generated visual, auditory, tactile, somatosensory, olfactory, or other sensory information about the surgical field and Enhanced by overlaying onto real-world images of instruments or other objects appearing in the surgical field. Images may be streamed in real time or may be static images.

Real-world surgical instruments include a variety of surgical devices. Energy-based surgical devices include, among others, but not limited to, radio frequency (RF)-based monopolar and bipolar electrosurgical instruments, ultrasonic surgical instruments, combinations of RF electrosurgical instruments and ultrasonic instruments, Includes a combination of an RF electrosurgical stapler and a mechanical stapler. Surgical stapler devices are surgical instruments used to cut and staple tissue in a variety of surgical procedures, including bariatric, thoracic, colorectal, obstetrics and gynecology, urology, and general surgery.

In various examples, the present disclosure provides an augmented reality display system for use during a surgical procedure. The augmented reality display system includes an imaging device that captures real images of a surgical field during a surgical procedure and generates a first data feed. The sensor device generates a second data feed. An augmented reality device with an augmented reality display generates an augmented overlay based on the first data feed and the second data feed. The enhanced overlay includes a visual portion and a non-visual visual portion. A processor receives the first data feed, receives the second data feed, and combines the first data feed and the second data feed to generate an enhanced overlay.

In various examples, the present disclosure provides a method for presenting an augmented overlay during a surgical procedure. According to the method, an imaging device captures a real image of a surgical field during a surgical procedure. The imaging device generates a first data feed based on the captured real image. The sensor device generates a second data feed. An augmented reality device with an augmented reality display presents an augmented overlay based on the first data feed and the second data feed. The augmented overlay includes a visual portion and a non-visual portion.

The various aspects described herein, both as to construction and methods of operation, together with further objects and advantages thereof, may be best understood by reference to the following description in conjunction with the accompanying drawings, in which: FIG.
1 is a block diagram of a computer-implemented interactive surgical system according to one aspect of the present disclosure. FIG. 1 is an illustration of a surgical system used to perform a surgical procedure within an operating room, according to one aspect of the present disclosure; FIG. 1 is a surgical hub paired with a visualization system, a robotic system, and an intelligent instrument, according to one aspect of the present disclosure. Connecting a modular device located in one or more surgical sites of a medical facility, or any room within a medical facility with specialized equipment for surgical procedures, to the cloud according to an aspect of the present disclosure. FIG. 1 illustrates a surgical data network including configured modular communication hubs. 1 illustrates a computer-implemented interactive surgical system according to one aspect of the present disclosure; FIG. FIG. 3 illustrates a surgical hub including multiple modules coupled to a modular control tower, according to one aspect of the present disclosure. FIG. 2 illustrates an augmented reality (AR) system including an intermediate signal combiner disposed in a communication path between an imaging module and a surgical hub display, according to an aspect of the present disclosure. FIG. 2 illustrates an augmented reality (AR) system including an intermediate signal combiner disposed in a communication path between an imaging module and a surgical hub display, according to an aspect of the present disclosure. FIG. 3 illustrates an augmented reality (AR) device worn by a surgeon to communicate data to a surgical hub, according to one aspect of the present disclosure. 1 illustrates a system for augmenting surgical instrument information using an augmented reality display, according to one aspect of the present disclosure. FIG. 1 is a system diagram of an operating room with a surgical monitor having an intraoperative data display of a surgical field, in accordance with one aspect of the present disclosure; FIG. A live feed of a surgical field visualized through a laparoscopic camera during a minimally invasive surgical procedure showing poorly captured tissue between the jaws of a surgical instrument end effector as a tissue aspect, according to an aspect of the present disclosure. This is an expanded image. 3 is an expanded image of a live feed of a surgical field visualized through a laparoscopic camera during a minimally invasive surgical procedure displayed on an intraoperative data display, according to one aspect of the present disclosure; FIG. FIG. 3 illustrates an augmented reality system including an intermediate signal combiner disposed in a communication path between an imaging module and a surgical hub display, according to an aspect of the present disclosure. FIG. 3 illustrates a method of presenting an expansion overlay during a surgical procedure, according to one aspect of the present disclosure. FIG. 3 is a diagram illustrating a timeline of a context-aware surgical procedure, according to one aspect of the present disclosure.

Corresponding reference characters indicate corresponding parts throughout the figures. The examples set forth herein are indicative of various disclosed embodiments, and such illustrations should not be construed as limiting the scope in any way.

The applicant of the present application owns the following concurrently filed U.S. patent applications, the disclosures of each of which are incorporated herein by reference in their entirety:
- U.S. patent application entitled "METHOD FOR INTRAOPERATORY DISPLAY FOR SURGICAL SYSTEMS"; Attorney Docket No. END9352USNP1/210120-1M;
- U.S. patent application entitled "Utilization of surgical data values and situational awareness to control the overlay in surgical field view"; Attorney Docket No. END9352USNP2/210120-2;
- U.S. patent application entitled "Selective and adjustable mixed reality overlay in surgical field view"; Attorney Docket No. END9352USNP3/210120-3;
- U.S. patent application entitled "Risk based prioritization of display aspects in surgical field view"; Attorney Docket No. END9352USNP4/210120-4;
- U.S. patent application entitled "SYSTEMS AND METHODS FOR CONTROLLING SURGICAL DATA OVERLAY"; Attorney Docket No. END9352USNP5/210120-5;
- U.S. patent application entitled "SYSTEMS AND METHODS FOR CHANGING DISPLAY OVERLAY OF SURGICAL FIELD VIEW BASED ON TRIGGERING EVENTS"; Attorney Docket No. END9352USNP6/210120-6;
- U.S. patent application entitled "CUSTOMIZATION OF OVERLAID DATA AND CONFIGURATION"; attorney docket number END9352USNP7/210120-7;
- U.S. patent application entitled "INDICATION OF THE COUPLE PAIR OF REMOTE CONTROLS WITH REMOTE DEVICES FUNCTIONS"; attorney docket number END9352USNP8/210120-8;
- U.S. patent application entitled "COOPERATORY OVERLAYS OF INTERACTING INSTRUMENTS WHICH RESULTS IN BOTH OVERLAYS BEING EFFECTED"; Attorney Docket No. END9352USNP9/210120-9;
- U.S. patent application entitled "ANTICIPATION OF INTERACTIVE UTILIZATION OF COMMON DATA OVERLAYS BY DIFFERENT USERS"; Attorney Docket No. END9352USNP10/210120-10;
U.S. patent application entitled "MIXING DIRECTLY VISUALIZED WITH RENDERED ELEMENTS TO DISPLAY BLENDED ELEMENTS AND ACTIONS HAPPENING ON-SCREEN AND OFF-SCREEN"; Attorney Docket No. END9352USNP11/210120-11;
- U.S. patent application entitled "SYSTEM AND METHOD FOR TRACKING A PORTION OF THE USER AS A PROXY FOR NON-MONITORED INSTRUMENT"; Attorney Docket No. END9352USNP12/210120-12;
- U.S. patent application entitled "UTILIZING CONTEXTUAL PARAMETERS OF ONE OR MORE SURGICAL DEVICES TO PREDICT A FREQUENCY INTERVAL FOR DISPLAYING SURGICAL INFORMATION"; Attorney Docket No. END9352USNP13/210120-13;
- U.S. patent application entitled "COOPERATION AMONG MULTIPLE DISPLAY SYSTEMS TO PROVIDE A HEALTHCARE USER CUSTOMIZED INFORMATION"; Attorney Docket No. END9352USNP14/210120-14;
- U.S. patent application entitled "INTRAOPERAIVE DISPLAY FOR SURGICAL SYSTEMS"; Attorney Docket No. END9352USNP15/210120-15; and
- U.S. patent application entitled "ADAPTATION AND ADJUSTABILITY OR OVERLAID INSTRUMENT INFORMATION FOR SURGICAL SYSTEMS"; Attorney Docket No. END9352USNP16/210120-16.

The applicant of the present application owns the following U.S. patent applications, the disclosures of each of which are incorporated herein by reference in their entirety:
U.S. Patent Application No. 16/209,423, entitled "METHOD OF COMPRESSING TISSUE WITHIN A STAPLING DEVICE AND SIMULTANEOUSLY DISPLAYING THE LOCATION OF THE TISSUE WITHIN THE JAWS" (now U.S. Patent Application Publication No. 2019/0200981-A1);
- U.S. Patent Application No. 16/209,453, entitled "METHOD FOR CONTROLLING SMART ENERGY DEVICES" (now U.S. Patent Application Publication No. 2019/0201046-A1).

Before describing in detail various aspects of surgical devices and generators, the illustrative examples are limited to details of the construction and arrangement of parts, in applications or applications, that are illustrated in the accompanying drawings and description. Please note that this is not the case. The example embodiments may be implemented or incorporated into other aspects, variations, and modifications and may be practiced or carried out in various ways. Furthermore, unless otherwise stated, the terms and expressions used herein have been chosen for the convenience of the reader to describe the illustrative embodiments and are not intended to be limiting. Additionally, one or more of the aspects, expressions of aspects, and/or examples described below may be combined with any one or more of the other aspects, expressions of aspects, and/or examples described below. It should be understood that it can be combined with

Various aspects are directed to on-screen displays for surgical systems for various energy and surgical stapler based medical devices. Energy based medical devices include, but are not limited to, radio frequency (RF) based monopolar and bipolar electrosurgical instruments, ultrasonic surgical instruments, combinations of RF electrosurgical instruments and ultrasonic instruments, combinations of RF electrosurgical staplers and mechanical staplers, among others. Surgical stapler devices include surgical staplers in combination with electrosurgical devices and/or ultrasonic devices. Aspects of ultrasonic surgical devices can be configured, for example, to transect and/or coagulate tissue during a surgical procedure. Aspects of electrosurgical devices can be configured, for example, to transect, coagulate, seal, weld and/or desiccate tissue during a surgical procedure. Aspects of surgical stapler devices can be configured to transect and staple tissue during a surgical procedure, and in some aspects, surgical stapler devices can be configured to deliver RF energy to tissue during a surgical procedure. Electrosurgical devices are configured to deliver therapeutic and/or non-therapeutic RF energy to tissue. The elements of the surgical stapler, electrosurgical device, and ultrasonic device may be used in combination in a single surgical instrument.

In various aspects, the present disclosure provides an on-screen display of real-time information to the OR team during a surgical procedure. In accordance with various aspects of the present disclosure, a number of new unique on-screen displays are provided for displaying various visual information feedback on-screen to the OR team. According to this disclosure, visual information may include one or more of various visual media with or without sound. Generally, visual information includes still photographs, moving photographs, video or audio recordings, graphic art, visual aids, models, displays, visual representation services, and support processes. Visual information can be communicated on any number of display options, such as, for example, the primary OR screen, the energy or surgical stapler device itself, a tablet, augmented reality glasses, among others.

In various aspects, the present disclosure provides a long list of potential options for communicating visual information to the OR team in real time without overwhelming the OR team with too much visual information. For example, in various aspects, the present disclosure provides an on-screen display of visual information that allows the surgeon, or other members of the OR team, to selectively activate on-screen displays, such as icons surrounding the screen options, to manage the wealth of visual information. One or a combination of factors can be used to determine the active display, which may include, among others, the energy-based (e.g., electrosurgery, ultrasound) or mechanical-based (e.g., stapler) surgical device being used, the estimated risk associated with a given display, the surgeon's experience level, and the surgeon's choice. In other aspects, the visual information may include a wealth of data overlaid or superimposed on the surgical field to manage the visual information. In various aspects described below, this includes superimposed images that require video analysis and tracking to properly overlay the data. The visual information data communicated in this manner can provide additional useful visual information to the OR team in a more concise and understandable manner, as opposed to static icons.

In various aspects, the present disclosure provides techniques for selectively activating on-screen displays, such as icons that surround the screen, to manage visual information during a surgical procedure. In other aspects, the present disclosure provides techniques for determining an active display using one or a combination of factors. In various aspects, techniques according to the present disclosure may include, among other things, selecting an energy-based or mechanical-based surgical device in use as the active display, estimating the risk associated with a given display, and utilizing the experience level of the surgeon or OR team to make the selection.

In other aspects, techniques according to the present disclosure may include overlaying or superimposing rich data onto the surgical field to manage visual information. Some display arrangements described by this disclosure involve overlaying various visual representations of surgical data onto a live stream of the surgical field. As used herein, the term overlay includes translucent overlays, partial overlays, and/or moving overlays. Graphical overlays may be in the form of transparent, translucent, or opaque graphics, or a combination of transparent, translucent, and opaque elements or effects. Additionally, the overlay may be placed on or at least partially on or near objects in the surgical field, such as, for example, end effectors and/or critical surgical structures. The particular display arrangement may include displaying one or more of the overlays based on changes in the display priority value, including changes in color, size, shape, display time, display location, display frequency, highlighting, or a combination thereof. May include changes in elements. A graphical overlay is rendered on top of the active display monitor and conveys critical information to the OR team quickly and efficiently.

In other aspects, techniques according to this disclosure may include overlaying images that require video analysis and tracking to properly overlay visual information data. In other aspects, techniques according to the present disclosure communicate rich visual information, as opposed to simple static icons, to provide additional visual information to the OR team in a more concise and easy-to-understand manner. can include. In other aspects, visual overlays can be used in combination with auditory and/or somatosensory overlays, such as thermal, chemical, and mechanical devices, and combinations thereof.

The following description generally relates to devices, systems, and methods for providing augmented reality (AR) interactive experiences during surgical procedures. In this context, images of the surgical field and surgical instruments and other objects that appear in the surgical field may be used to convey computer-generated visual, auditory, tactile, somatosensory, olfactory, or other sensory information to the surgical field. by overlaying onto real-world images of instruments and/or other objects that appear in the image. Images may be streamed in real time or may be static images. Augmented reality is a technology for rendering and displaying virtual or "augmented" virtual objects, data, or visual effects that are overlaid onto a real environment. The real environment may include a surgical field. Virtual objects overlaid on the real environment may be represented in a fixed or set position relative to one or more aspects of the real environment. In a non-limiting example, when a real world object exits the field of view of the real environment, a virtual object fixed to the real world object also exits the field of view of the augmented reality.

Some display arrangements described by this disclosure involve overlaying various visual representations of surgical data onto a live stream of the surgical field. As used herein, the term overlay includes translucent overlays, partial overlays, and/or moving overlays. Additionally, the overlay may be placed on or at least partially on or near objects in the surgical field, such as, for example, end effectors and/or critical surgical structures. The particular display arrangement may include displaying one or more of the overlays based on changes in the display priority value, including changes in color, size, shape, display time, display location, display frequency, highlighting, or a combination thereof. May include changes in elements.

As described herein, AR is an augmented version of the real physical world achieved through the use of digital visual elements, sound, or other sensory stimuli delivered via technology. Virtual reality (VR) is a computer-generated environment with scenes and objects that appear to be real, making the user feel immersed in their surroundings. This environment is perceived through a device known as a virtual reality headset or helmet. Although mixed reality (MR) and AR are both considered immersive technologies, they are not the same. MR is an extension of mixed reality that allows real and virtual elements to interact within an environment. While AR adds digital elements to the live view, often by using cameras, MR experiences combine elements of both AR and VR, where real-world and digital objects interact.

In an AR environment, one or more computer-generated virtual objects may be displayed along with one or more real (i.e., so-called "real-world") elements. For example, real-time images or videos of the surrounding environment may be shown on a computer screen display along with one or more overlaid virtual objects. Such virtual objects may provide supplemental information about the environment, or may generally enhance the user's perception of and engagement with the environment. Conversely, real-time images or videos of the surrounding environment may additionally or alternatively enhance the user's engagement with the virtual objects shown on the display.

Apparatus, systems, and methods in the context of the present disclosure enhance images received from one or more imaging devices during a surgical procedure. Imaging devices may include various scopes used during non-invasive and minimally invasive surgical procedures, AR devices, and/or cameras that provide images during open surgical procedures. Images may be streamed in real time or may be static images. Apparatus, systems, and methods provide an augmented reality interactive experience by enhancing the image of a real-world surgical environment by overlaying virtual objects or data and/or representations of real objects onto the real-world surgical environment. do. The augmented reality experience may be viewed on a display and/or AR device that allows a user to view virtual objects overlaid on a real-world surgical environment. The display may be located within the operating room or may be remote from the operating room. AR devices are worn on the head of a surgeon or other operating room personnel and typically include two stereoscopic display lenses or screens, including one for each eye of the user. Natural light can pass through two transparent or translucent display lenses so that aspects of the real environment are visible while projecting light to make virtual objects visible to a user of the AR device.

Two or more displays and AR devices may be used in coordination, for example, with a first display or AR device controlling one or more additional displays or AR devices in the system with defined roles. For example, when activating a display or AR device, a user may select a role (e.g., surgeon during a surgical procedure, surgical assistant, nurse, etc.) and the display or AR device may display information related to that role. For example, a surgical assistant may have a virtual representation of the instruments the surgeon needs to perform for the next step in the surgical procedure displayed. The surgeon's focus on the current step may see different displayed information than a surgical assistant.

While there are many known on-screen displays and alerts, the present disclosure provides many new and unique augmented reality interactive experiences during surgical procedures. Such augmented reality interactive experiences include visual, auditory, tactile, somatosensory, olfactory, or other sensory feedback information to the surgical team inside or outside the operating room. Virtual feedback information overlaid on the real-world surgical environment can, for example, improve personnel within the OR, including, but not limited to, operating surgeons, surgeon's assistants, scrub wearers, anesthetists, and outgoing nurses, among others. may be provided to the operating room (OR) team, including: Virtual feedback information can be communicated on any number of display options, such as primary OR screen displays, AR devices, energy or surgical stapler instruments, tablets, augmented reality glasses, devices, etc.

1 illustrates a computer-implemented interactive surgical system 1 including one or more surgical systems 2 and a cloud-based system 4. The cloud-based system 4 may include a remote server 13 coupled to a remote storage 5. Each surgical system 2 includes at least one surgical hub 6 in communication with the cloud 4. For example, the surgical system 2 may include a visualization system 8, a robotic system 10, and a handheld intelligent surgical instrument 12, each configured to communicate with each other and/or with the hub 6. In some aspects, the surgical system 2 may include M hubs 6, N visualization systems 8, O robotic systems 10, and P handheld intelligent surgical instruments 12, where M, N, O, and P are integers equal to or greater than 1. The computer-implemented interactive surgical system 1 may be configured to provide an augmented reality interactive experience during a surgical procedure, as described herein.

FIG. 2 shows an example of a surgical system 2 for performing a surgical procedure on a patient lying on an operating table 14 in a surgical room 16. A robotic system 10 is used as part of the surgical system 2 in the surgical procedure. The robotic system 10 includes a surgeon's console 18, a patient side cart 20 (surgical robot), and a surgical robot hub 22. The patient side cart 20 can manipulate at least one detachably coupled surgical tool 17 through a minimally invasive incision in the patient's body while the surgeon views the surgical site through the surgeon's console 18 or an augmented reality (AR) device 66 worn by the surgeon. Images of the surgical site during the minimally invasive procedure (e.g., still images or live images streamed in real time) can be acquired by a medical imaging device 24. The patient side cart 20 can manipulate and orient the imaging device 24. Images of the incisional surgical procedure can be acquired by a medical imaging device 96. The robotic hub 22 processes the images of the surgical site for subsequent display on the surgeon's console 18 or an AR device 66 worn by the surgeon or other person in the surgical room 16.

The optical components of the imaging device 24, 96 or AR device 66 may include one or more illumination sources and/or one or more lenses. The one or more illumination sources may be directed to illuminate a portion of the surgical field. The one or more image sensors may receive light reflected or refracted from tissue and instruments in the surgical field.

In various aspects, imaging device 24 is configured for use in minimally invasive surgical procedures. Examples of imaging devices suitable for use with the present disclosure include arthroscopes, angioscopes, bronchoscopes, cholangioscopes, colonoscopes, cystoscopes, duodenoscopes, enteroscopes, esophagogastroduodenoscopes (gastroscopes), These include, but are not limited to, endoscopes, laryngoscopes, nasopharyngeal-pyeloscopes, sigmoidoscopes, thoracoscopes, and ureteroscopes. In various aspects, imaging device 96 is configured for use in open (invasive) surgical procedures.

In various aspects, visualization system 8 includes one or more imaging sensors strategically positioned relative to the sterile field, one or more image processing devices, one or more storage arrays, and one or more storage arrays. Equipped with a display. In one aspect, visualization system 8 includes interfaces for HL7, PACS, and EMR. In one aspect, imaging device 24 may employ multispectral monitoring to distinguish between topography and underlying structure. Multispectral images capture image data within specific wavelength ranges in the electromagnetic spectrum. The wavelengths are separated by filters or by instruments with sensitivity to specific wavelengths, including light from frequencies beyond the visible light range, such as the IR and ultraviolet. Spectral imaging can extract information invisible to the human eye. Multispectral monitoring can reposition the surgical field after the surgical task is completed to perform tests on the treated tissue.

2 shows a primary display 19 positioned in the sterile field for viewing by an operator at the operating table 14. The visualization tower 11 includes a first non-sterile display 7 and a second non-sterile display 9 positioned outside the sterile field and facing away from each other. A visualization system 8 guided by the hub 6 is configured to utilize the displays 7, 9, 19 to coordinate information flow to operators inside and outside the sterile field. For example, the hub 6 can cause the visualization system 8 to display AR images of the surgical site recorded by the imaging devices 24, 96 through the non-sterile displays 7, 9 or the AR device 66 while maintaining a live video of the surgical site on the primary display 19 or the AR device 66. The non-sterile displays 7, 9 can, for example, enable a non-sterile operator to perform diagnostic steps related to a surgical procedure.

3 shows a hub 6 in communication with a visualization system 8, a robotic system 10, and a handheld intelligent surgical instrument 12. The hub 6 includes a hub display 35, an imaging module 38, a generator module 40, a communication module 30, a processor module 32, a storage array 34, and an operating room mapping module 33. The hub 6 further includes a smoke evacuation module 26 and/or an aspiration/irrigation module 28. In various aspects, the imaging module 38 includes an AR device 66, and the processor module 32 includes an integrated video processor and an augmented reality modeler (e.g., as shown in FIG. 10). The modular light source can be adapted for use with a variety of imaging devices. In various examples, multiple imaging devices can be positioned at different locations of the surgical field to provide multiple views (e.g., non-invasive, minimally invasive, invasive, or open surgical procedures). The imaging module 38 can be configured to switch between imaging devices to provide an optimal view. In various aspects, the imaging module 38 can be configured to integrate images from different imaging devices to provide an augmented reality interactive experience during a surgical procedure as described herein.

4 shows a surgical data network 51 including a modular communication hub 53 configured to connect modular devices located at one or more surgical sites/rooms of a medical facility to a cloud-based system. The cloud 54 can include a remote server 63 (FIG. 5) coupled to a storage device 55. The modular communication hub 53 includes a network hub 57 and/or a network switch 59 in communication with a network router 61. The modular communication hub 53 is coupled to a local computer system 60 for processing data. Modular devices 1a-1n at the surgical site can be coupled to the modular communication hub 53. The network hub 57 and/or the network switch 59 can be coupled to the network router 61 to connect the devices 1a-1n to the cloud 54 or the local computer system 60. Data associated with the devices 1a-1n can be transferred through the router to a cloud-based computer for remote data processing and manipulation. The surgical site devices 1a-1n can be connected to the modular communication hub 53 via wired or wireless channels. The surgical data network 51 environment may be employed to provide an augmented reality interactive experience during a surgical procedure, as described herein, and in particular to provide an augmented image of the surgical field to one or more remote displays 58.

Figure 5 illustrates a computer-implemented interactive surgical system 50. The computer-implemented interactive surgical system 50 is similar in many respects to the computer-implemented interactive surgical system 1. The computer-implemented interactive surgical system 50 includes one or more surgical systems 52 that are similar in many respects to the surgical system 2. Each surgical system 52 includes at least one surgical hub 56 that communicates with a cloud 54 that may include a remote server 63. In one aspect, the computer-implemented interactive surgical system 50 includes a modular control tower 23 connected to multiple surgical field devices, such as, for example, intelligent surgical instruments, robots, and other computerized devices located within the surgical field. As shown in Figure 6, the modular control tower 23 includes a modular communication hub 53 coupled to a computer system 60.

Returning to FIG. 5, the modular control tower 23 includes an imaging module 38 coupled to an endoscope 98, a generator module 27 coupled to an energy device 99, a smoke evacuator module 76, an aspiration/irrigation module 78, and a communication The module 13 is coupled to a processor module 15, a storage array 16, a smart device/appliance 21 optionally coupled to a display 39, and a sensor module 29. The surgical site devices are coupled to cloud computing resources such as servers 63, data storage 55, and displays 58 via modular control tower 23. Robotic hub 72 may also be connected to modular control tower 23 and servers 63, data storage 55, and displays 58. Among other things, devices/appliances 21, visualization systems 58 may be coupled to modular control tower 23 via wired or wireless communication standards or protocols, as described herein. Modular control tower 23 is configured to display the received augmented images, including overlaid virtual objects on the real surgical world, received from imaging module 38, device/instrument display 39, and/or other visualization system 58. The hub display 65 (eg, monitor, screen) may also be coupled to the hub display 65 (eg, monitor, screen). Hub display 65 may also display data received from devices connected to modular control tower 23 along with images and overlay images.

6 shows a surgical hub 56 including multiple modules coupled to a modular control tower 23. The modular control tower 23 includes a modular communication hub 53, e.g., a network-connected device, and a computer system 60, e.g., for local processing, visualization, and imaging of extended surgical information. The modular communication hub 53 may be connected in a hierarchical configuration to expand the number of modules (e.g., devices) that may be connected to the modular communication hub 53 and transfer data associated with the modules to the computer system 60, cloud computing resources, or both. Each of the network hubs/switches 57/59 in the modular communication hub 53 may include three downstream ports and one upstream port. The upstream network hubs/switches 57, 59 are connected to a processor 31 to provide communication connections to cloud computing resources and a local display 67. Communication to the cloud 54 may be via either wired or wireless communication channels.

Computer system 60 includes a processor 31 and a network interface 37. Processor 31 is coupled to communication module 41, storage 45, memory 46, non-volatile memory 47, and input/output interface 48 via a system bus. The system bus may be any of several types of bus structures, including a memory bus or memory controller, a peripheral or external bus, and/or a local bus using a variety of available bus architectures. .

The processor 31 includes an augmented reality modeler (e.g., as shown in FIG. 10) and may be implemented as a single-core or multi-core processor such as those known under the trade name ARM Cortex by Texas Instruments. In one aspect, the processor may be, for example, an LM4F230H5QR ARM Cortex-M4F processor core available from Texas Instruments. The processor core includes on-chip memory of 256 KB of single-cycle flash memory or other non-volatile memory up to 40 MHz, a pre-fetch buffer to improve performance beyond 40 MHz, 32 KB of single-cycle serial random access memory (SRAM), internal read-only memory (ROM) with StellarisWare® software, 2 KB of electrically erasable programmable read-only memory (EEPROM), and/or one or more pulse width modulation (PWM) modules, one or more quadrature encoder input (QEI) analogs, one or more 12-bit analog-to-digital converters (ADCs) with 12 analog input channels, details of which are available in the product data sheet.

System memory includes volatile and nonvolatile memory. The basic input/output system (BIOS), containing the basic routines for transferring information between elements within a computer system, such as during start-up, is stored in nonvolatile memory. For example, nonvolatile memory may include ROM, programmable ROM (PROM), electrically programmable ROM (EPROM), EEPROM or flash memory. Volatile memory includes random access memory (RAM), which acts as external cache memory. In addition, RAM is available in many forms, such as SRAM, dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), SyncLink DRAM (SLDRAM), and direct RAMbus RAM (DRRAM).

The computer system 60 also includes removable/non-removable volatile/non-volatile computer storage media, such as disk storage. Disk storage devices include, but are not limited to, devices such as magnetic disk drives, floppy disk drives, tape drives, Jaz drives, Zip drives, LS-60 drives, flash memory cards, or memory sticks. In addition, the disk storage devices can include the above storage media, either independently or in combination with other storage media. Other storage media include, but are not limited to, optical disk drives, such as compact disk ROM drives (CD-ROM), compact disk recordable drives (CD-R drives), compact disk rewriteable drives (CD-RW drives), or digital versatile disk ROM drives (DVD-ROM). Removable or non-removable interfaces may be used to facilitate connection of the disk storage devices to the system bus.

In various aspects, computer system 60 of FIG. 6, imaging module 38 and/or visualization system 58 of FIGS. 4-6, and/or processor module 15 may include an image processor, image processing engine, image processing unit (GPU). , a media processor, or any dedicated digital signal processor (DSP) used to process digital images. Image processors can use parallel computing using single instruction multiple data (SIMD) or multiple instruction multiple data (MIMD) techniques to increase speed and efficiency. Digital image processing engines can perform a variety of tasks. The image processor may be a system on a chip with a multi-core processor architecture.

7 shows an augmented reality system 263 including an intermediate signal combiner 64 disposed in the communication path between the imaging module 38 and the surgical hub display 67. The signal combiner 64 combines audio and/or image data received from the imaging module 38 and/or the AR device 66. The surgical hub 56 receives the combined data from the combiner 64 and overlays the provided data on the display 67, where the overlaid data is displayed. The imaging device 68 may be a digital video camera and the audio device 69 may be a microphone. The signal combiner 64 may include a wireless heads-up display adapter for coupling to the AR device 66 disposed in the communication path of the display 67 to the console that allows the surgical hub 56 to overlay the data on the display 67.

FIG. 8 shows an augmented reality (AR) system that includes an intermediate signal combiner positioned in a communication path between an imaging module and a surgical hub display. FIG. 8 shows an AR device 66 worn by a surgeon 73 to communicate data to the surgical hub 56. The peripheral information of the AR device 66 does not include active video. Rather, the peripheral information includes only signals that do not have the same requirements for device settings or refresh rates. The interaction may enhance the surgeon's 73 information based on linking with preoperative computed tomography (CT) or other data linked within the surgical hub 56. The AR device 66 can identify structures, for example, can ask if the instrument is touching a nerve, blood vessel, or adhesion. AR device 66 may include preoperative scan data, optical views, tissue investigation characteristics acquired throughout the procedure, and/or processing at surgical hub 56 that is used to provide answers. Surgeon 73 may write notes on AR device 66 to be saved with patient data in hub storage 45 for later use in reporting or follow-up.

An AR device 66 worn by the surgeon 73 links to the surgical hub 56 with audio and visual information to avoid the need for overlays and allows customization of information displayed around the periphery of the field of view. Make it. AR device 66 provides signals from devices (eg, fixtures) and answers queries regarding device settings or location information linked with video to identify quadrants or locations. AR device 66 has audio control and audio feedback from AR device 66 . The AR device 66 can interact with other systems within the surgical site and can have feedback and interaction available wherever the surgeon 73 looks. For example, AR device 66 may receive audio or gesture initiation commands and queries from a surgeon, and AR device 66 may provide feedback in the form of one or more modalities including audio, visual, or tactile touch. Good too.

FIG. 9 shows a surgeon 73 and a patient 74 wearing an AR device 66 and may include a camera 96 within the operating room 75. AR device 66 worn by surgeon 73 may be used to present virtual objects to surgeon 73 that are overlaid on real-time images of the surgical field, either through augmented reality display 89 or through hub-connected display 67. The real-time image may include a portion of the surgical instrument 77. The virtual object may not be visible to others in the operating room 75 (eg, surgical assistants or nurses), although they may also be wearing the AR device 66. Even if another person is viewing the operating room 75 using an AR device 66, that person may not be able to see the virtual objects or in the augmented reality shared with the surgeon 73. It may be possible to view the virtual object, or it may be possible to view a modified version of the virtual object (eg, according to customization specific to the surgeon 73), or it may be possible to view a different virtual object.

The virtual objects and/or data may be captured on a portion of the surgical instrument 77 or by the imaging module 38, the imaging device 68 during a minimally invasive surgical procedure, and/or the camera 96 during an open surgical procedure. It may be configured to appear within the field of view. In the illustrated example, imaging module 38 is a laparoscopic camera that provides live video of the surgical field during a minimally invasive surgical procedure. The AR system may present virtual objects that are fixed to real objects regardless of the perspective of one or more viewers of the AR system (eg, surgeon 73). For example, the virtual object may be visible to a viewer of the AR system within the operating room 75 and may not be visible to a viewer of the AR system outside the operating room 75. The virtual object may be displayed to a viewer outside the operating room 75 when the viewer enters the operating room 75. The augmented image may be displayed on surgical hub display 67 or augmented reality display 89.

The AR device 66 may include one or more screens or lenses, such as a single screen or two screens (e.g., one for each eye of the user). The screens may allow light to pass through them so that aspects of the real environment are visible while displaying the virtual objects. The virtual objects may be made visible to the surgeon 73 by projecting light onto them. The virtual objects may appear to have a degree of transparency or may be opaque (i.e., blocking aspects of the real environment).

The AR system may be visible to one or more viewers and may include differences between views available to one or more viewers while retaining some aspects common between the views. . For example, a heads-up display may change between two views, but virtual objects and/or data may be fixed to real objects or regions in both views. Aspects such as object color, lighting, or other changes may be made between views without changing the fixed position of the at least one virtual object.

A user may view virtual objects and/or data presented within an AR system as opaque or as containing some level of transparency. In one example, a user can interact with a virtual object, such as by moving the virtual object from a first position to a second position. For example, the user may move the object with his/her hand. This may include one or more devices, which may be mounted on the AR device 66 (such as an AR device camera 79 or a separate 96), which may be static, or may be controlled to move. This may be done virtually in an AR system by determining (using a camera) that the hand has moved into a position coincident with or adjacent to an object and moving the object accordingly. Virtual aspects may include virtual representations of real-world objects or may include visual effects such as lighting effects. AR systems may include rules to govern the behavior of virtual objects, such as subjecting them to gravity or friction, or otherwise negating real-world physical constraints (e.g., floating objects, perpetual motion, etc.) may include predefined rules. AR device 66 may include a camera 79 on AR device 66 (not to be confused with camera 96, which is separate from AR device 66). The AR device camera 79 or camera 96 may include an infrared camera, an infrared filter, a visible light filter, multiple cameras, a depth camera, and the like. AR device 66 may project virtual items onto a representation of the real environment that can be viewed by a user.

The AR device 66 may be used, for example, in the operating room 75 during a surgical procedure performed by the surgeon 73 on the patient 74. The AR device 66 may project or display virtual objects, such as virtual objects during the surgical procedure, to augment the surgeon's vision. The surgeon 73 may view the virtual objects using the AR device 66, a remote controller for the AR device 66, or may interact with the virtual objects, for example, using his or her hands to "interact" with the virtual objects or gestures recognized by the camera 79 of the AR device 66. The virtual objects may augment surgical tools, such as the surgical instrument 77. For example, the virtual objects may appear (to the surgeon 73 viewing the virtual objects through the AR device 66) to be coupled with or remain a fixed distance from the surgical instrument 77. In another example, the virtual objects may be used to guide the surgical instrument 77 and may appear fixed to the patient 74. In certain examples, the virtual objects may respond to the movement of other virtual or real-world objects in the surgical field. For example, a virtual object may change when a surgeon manipulates a surgical tool in proximity to the virtual object.

Augmented reality display system imaging device 38 captures a real image of the surgical field during the surgical procedure. Augmented reality displays 89, 67 present an overlay of the operating aspects of surgical instrument 77 on the real image of the surgical field. Surgical instrument 77 includes communication circuitry 231 for communicating operating aspects and functional data from surgical instrument 77 to AR device 66 via communication circuitry 233 on AR device 66. Although surgical instrument 77 and AR device 66 are shown in RF wireless communication between circuits 231, 233 as indicated by arrows B, C, other communication techniques (e.g., wired, ultrasonic, infrared, etc.) may be employed. The overlay relates to the operating aspects of surgical instrument 77 that are actively visualized. The overlay combines aspects of tissue interaction in the surgical field with functional data from surgical instrument 77. The processor portion of the AR device 66 is configured to receive operational aspects and functional data from the surgical instrument 77, determine an overlay related to the operation of the surgical instrument 77, and combine tissue aspects within the surgical field with the functional data from the surgical instrument 77. The augmented images show alerts for device performance considerations, incompatible use alerts, and incomplete capture alerts. Incompatible use includes out of range tissue conditions and tissue incorrectly balanced within the jaws of the end effector. Additional augmented images provide indications of contingencies, including an indication of tissue tension and an indication of foreign body detection. Other augmented images show device status overlays and instrument indications.

10 illustrates a system 83 for augmenting an image of a surgical field with information using an AR display 89, according to at least one aspect of the present disclosure. The system 83 may be used to perform the techniques described below, for example, by using a processor 85. The system 83 includes an aspect of an AR device 66 that can communicate with a database 93. The AR device 66 includes a processor 85, a memory 87, an AR display 89, and a camera 79. The AR device 66 may include a sensor 90, a speaker 91, and/or a haptic controller 92. The database 93 may include an image storage 94 or a preoperative planning storage 95.

Processor 85 of AR device 66 includes an augmented reality modeler 86 . Augmented reality modeler 86 may be used by processor 85 to create an augmented reality environment. For example, augmented reality modeler 86 may receive images of instruments within the surgical field, such as from camera 79 or sensor 90, and create an augmented reality environment to fit within the displayed image of the surgical field of view. In another example, physical objects and/or data may be overlaid on the surgical field of view and/or surgical instrument images, and augmented reality modeler 86 uses the physical objects and data to create virtual objects and /Or an augmented reality display of data may be presented within an augmented reality environment. For example, augmented reality modeler 86 uses or detects instruments at a patient's surgical site and displays virtual objects and/or data on the surgical instruments and/or images of the surgical site within the surgical field of view captured by camera 79. May be presented. The AR display 89 may display an AR environment overlaid on the real environment. Display 89 may show virtual objects and/or data using AR device 66, such as at a fixed location within the AR environment.

AR device 66 may include a sensor 90, such as an infrared sensor. Camera 79 or sensor 90 may be used to detect movements, such as gestures by a surgeon or other user, which are determined by processor 85 to indicate an attempted or intended interaction by the user with a virtual target. may be interpreted as Processor 85 may identify objects in the real environment, such as by processing information received using camera 79. In other aspects, sensors 90 may be tactile, audible, chemical, or thermal sensors to generate corresponding signals that may be combined with various data feeds to create an augmented environment. Sensors 90 may include binaural audio sensors (spatial sounds), inertial measurement (accelerometers, gyroscopes, magnetometers) sensors, environmental sensors, depth camera sensors, hand and eye tracking sensors, and voice command recognition capabilities.

The AR display 89 can display virtual images corresponding to physical features hidden by the patient's anatomy, such as within the surgical field, while allowing the surgical field to be viewed through the AR display 89 during a surgical procedure. Features may also be presented. The virtual feature may have a virtual location or orientation that corresponds to a first physical location or orientation of the physical feature. In one example, the virtual position or orientation of the virtual feature may include an offset from the first physical position or orientation of the physical feature. The offset may include a predetermined distance from the augmented reality display, a relative distance from the augmented reality display to the anatomical feature, etc.

In one example, the AR device 66 can be an individual AR device. In one aspect, the AR device 66 can be a HoloLens 2 AR device manufactured by Microsoft of Redmond, Washington. This AR device 66 includes a visor with lenses and binaural audio features (spatial sound), inertial measurements (accelerometers, gyroscopes, magnetometers), environmental sensors, a depth camera, a video camera, hand and eye tracking, and voice command recognition. It provides an improved field of view with high resolution by using mirrors to direct the waveguide in front of the wearer's eyes. The image can be magnified by changing the angle of the mirror. It also provides eye tracking to recognize the user and adjust the lens width for the particular user.

In another example, AR device 66 may be a Snapchat Spectacles 3 AR device. This AR device provides the ability to capture paired images, recreate 3D depth mapping, add virtual effects, and play 3D videos. The AR device includes two HD cameras to capture 3D photos and videos at 60fps, while four built-in microphones record immersive high-fidelity audio. Images from both cameras are combined to build a geometric map of the real world around the user, providing a new sense of depth perception. Photos and videos can be wirelessly synced to an external display device.

In yet another example, AR device 66 may be a Glass 2 AR device by Google. This AR device provides inertial measurement (accelerometer, gyroscope, magnetometer) information overlaid on the lens (outside the field of view) to supplement the information.

In another example, AR device 66 may be an Echo Frames AR device by Amazon. This AR device has no camera/display. The microphone and speaker are linked to Alexa. This AR device has fewer features than a head-up display.

In yet another example, AR device 66 may be a Focals AR device by North (Google). This AR device offers notification pusher/smartwatch analog, inertial measurements, screen overlays of information (weather, calendar, messages), and voice control (Alexa) integration. This AR device provides basic head-up display functionality.

In another example, AR device 66 may be a Nreal AR device. This AR device includes spatial sound, two environmental cameras, a photo camera, IMU (accelerometer, gyroscope), ambient light sensor, and proximity sensor functionality. nebula projects application information onto the lens.

In various other examples, the AR device 66 is any one of the following commercially available AR devices: Magic Leap 1, Epson Moverio, Vuzix Blade AR, ZenFone AR, Microsoft AR Glasses Prototype, EyeTap. This creates light that is collinear with the light of the environment directly on the retina. A beam splitter, for example, makes the same visible light available to a computer for processing and overlaying information. AR visualization systems include HUDs, contact lenses, glasses, virtual reality (VR) headsets, virtual retinal displays, operating room displays, and/or smart contact lenses (bionic lenses).

The multi-user interface for the AR device 66 includes virtual retinal displays such as raster displays that are drawn directly on the retina rather than on a screen in front of the eyes, spatial displays such as smart TVs, smartphones, and/or Sony spatial display systems. including.

Other AR technologies may include, for example, AR capture devices and software applications, AR creation devices and software applications, and AR cloud devices and software applications. AR capture devices and software applications include, for example, Apple Polycam app, Ubiquity 6 (Mirrorworld using Display.land app), in which the user scans a 3D image of the real world (to create a 3D model); can be obtained. AR creation devices and software applications include, for example, Adobe Aero, Vuforia, ARToolKit, Google ARCore, Apple ARKit, MAXST, Aurasma, Zappar, Blippar. AR cloud devices and software applications include, for example, Facebook, Google (world geometry, object recognition, predictive data), Amazon AR Cloud (commerce), Microsoft A Including zure, Samsung Project Where, Niantic, and Magic Leap.

One aspect of the following disclosure describes the overlay of various operating aspects or functions of surgical instruments onto a live video stream of a surgical field visualized through the surgical field of a laparoscopic camera during a minimally invasive surgical procedure. The overlay relates to the operational aspect of one of the surgical instruments and devices that is being actively visualized. The overlay combines aspects of tissue/organ interaction with functional data received from surgical instruments used in a surgical procedure. Surgical instruments may include graspers, clamps, staplers, ultrasound, RF, or combinations of each of these instruments. For graspers and clamps, aspects of tissue parameters can include incomplete capture of tissue as well as clamp condition or clamp size. For surgical staplers, aspects of tissue parameters may include tissue capture location, tissue compression, clamping, or sufficient firing of the surgical stapler. For advanced energy devices such as ultrasound or RF devices, aspects of tissue parameters may include impedance, ablation status, hemorrhage size, and aspects of instrument functionality include, for example, energy level, timing, clamp pressure, among others. obtain. The augmented images shown in FIGS. 11-13 below can be viewed on a local display, remote display, and/or AR device, as described above in connection with FIGS. 1-10. Although enhanced images are described as being visualized through a laparoscopic camera during a minimally invasive surgical procedure, the images may be non-invasive and invasive (e.g. , open) may be captured during a surgical procedure. These aspects will be explained below.

Figures 11-13 show various expanded images visualized through a laparoscopic camera during a minimally invasive surgical procedure. Augmented reality display systems are used during surgical procedures. The augmented reality display system includes: an imaging device for capturing a real image of a surgical area during a surgical procedure; and an augmented reality display for presenting an overlay of operational aspects of a surgical instrument on the real image of the surgical area. A processor. The overlay relates to the operational aspects of the surgical instrument that are being actively visualized. The overlay combines aspects of tissue interaction in the surgical field with functional data from the surgical instruments. The processor is configured to receive operational aspects and functional data from the surgical instrument, determine an overlay associated with the operation of the surgical instrument, and combine aspects of the tissue within the surgical field with the functional data from the surgical instrument. Ru. The enhanced image shows alerts regarding device performance considerations, non-conforming usage alerts, and incomplete acquisition alerts. Incompatible use includes out-of-range tissue conditions and tissue that is incorrectly balanced within the jaws of the end effector. Additional enhanced images provide an indication of concomitant events, including an indication of tissue tension and an indication of foreign body detection. Other enhanced images show device status overlays and instrument indications.

FIGS. 11-13 also illustrate a functional overlay of key instrument operation or parameters to clearly depict aspects of the surgical stapler, energy device, or interaction thereof. In one aspect, the overlaid data is adjusted by aspects detected by the surgical hub to simply modify the overlay from information detected by the source instrument to add context. In another aspect, the display may be adjusted or modified by the user, resulting in a modification of the operation of the surgical instrument being monitored as well.

11-13 illustrate an intraoperative display system for use during a surgical procedure. The system includes a surgical monitor having an intraoperative data display of a surgical area. An advanced energy generator is coupled to an advanced energy surgical instrument. The advanced energy surgical instrument uses radio frequency (RF) energy and ultrasonic energy during a surgical procedure on a patient. A surgical hub is coupled to the advanced energy generator and the surgical monitor. The surgical hub provides a live feed of the surgical area to the surgical monitor for displaying the live feed of the surgical area through the intraoperative data display. The intraoperative data display displays a view of the surgical area including the advanced energy surgical instrument gripping tissue and a panel overlay displaying information specific to the advanced energy surgical instrument.

In one aspect, the intraoperative data display shows an end effector of a surgical instrument grasping tissue and a panel overlay displaying case information, system notifications, or device panels, or any combination thereof, and includes a live surgical feed. is overlaid on top of. The position, opacity, size, and placement of panel overlays are customized. Panel overlays are configured to be turned on and off individually or as a group. The panel overlay is further configured to dynamically change to indicate a change in status, such as device activation or power level adjustment. The panel overlay shows optimal device performance (ODP) guide images or other instructions for use (IFU)/information sources.

In various embodiments, panel overlays can be used to store, recall, or edit capital equipment, generators, injectors, smoke evacuators, electronic health records, laparoscopes, computers, surgical devices, wired and wireless connectivity devices, and surgeons who may save, recall, or edit the panel overlays. including at least one of data input information from profile preferences, or any combination thereof. The panel overlay may include case information, including at least one of patient name, surgeon name, case time, or instrument activation, or a combination thereof. The panel overlay may include system notifications including at least one of connected equipment status, minor error alert, medium error alert, or major error alert, or any combination thereof. The panel overlay may include information associated with surgical instruments connected to the system to provide advanced hemostasis. The panel overlay may include a visible patient panel overlay. The panel overlay may include a device panel overlay comprising at least one of a device name, device settings, or device supplementary characteristics, or any combination thereof. The panel overlay may include multiple panel overlays in a stacked configuration. The panel overlay may include multiple panel overlays in an expanded configuration. A panel overlay may display display device troubleshooting information. The panel overlay may display at least one of alerts, warnings, device information, or device characteristics, or any combination thereof.

In another aspect, the intraoperative data display includes a secondary configurable panel. The secondary configurable panel dynamically changes based on a selected customized laparoscopic overlay field displayed within the surgical field of view of the live surgical feed area of the intraoperative data display. The customized laparoscopic overlay field comprises at least one of a bottom edge panel, a top left corner panel, a top center panel, or a side edge panel, or any combination thereof.

FIG. 11 is a system diagram 3000 of an operating room with a surgical monitor having an intraoperative data display 3002 of the surgical area. An advanced energy generator 3004 is coupled to a surgical hub 3006 and an advanced energy surgical instrument 3008. The advanced energy surgical instrument 3008 uses RF and ultrasonic energy during a surgical procedure on a patient 3010. The surgical hub 3006 provides a live feed 3014 of the surgical area that is displayed by the intraoperative data display 3002. The intraoperative data display 3002 displays a view of the surgical area including the advanced energy surgical instrument 3008 gripping tissue and a panel overlay 3012 that displays information specific to the advanced energy surgical instrument 3008.

12 is an augmented image 300 of a live feed of a surgical field 324 visualized through a laparoscopic camera during a minimally invasive surgical procedure, showing tissue 322 as a tissue aspect insufficiently captured between the jaws 318 of a surgical instrument end effector 320. The laparoscopic view 302 of the surgical field 324 shows the surgical instrument end effector 320 grasping tissue 322 with the jaws 318 of the end effector 320. The augmented image 300 shows a graphical alert overlay 304 superimposed on the image of the surgical field 324 to indicate insufficiently captured tissue 322 against the edge of the cut at the jaws 318 of the end effector 320.

The augmented image 300 also includes a first sub-image 308 showing a graphic image 306 of general anatomy superimposed on or adjacent to the surgical view 302, and a reference frame 310 of actual anatomy superimposed on or adjacent to the surgical view 302. The augmented image 300 also includes a second sub-image 312 showing the type of surgical instrument being used, the energy level if applicable, and the current surgical procedure. The second sub-image 312 may be superimposed on or located adjacent to the surgical view 302. The augmented image 300 shows an ultrasonic surgical instrument being used in the surgical procedure with the energy level set to a maximum of 5 to achieve advanced hemostasis. A graphic image 316 of the surgical instrument is shown superimposed on the graphic image 314 of the incomplete tissue capture alert overlay 304. Thus, the augmented image 300 provides several virtual objects to inform the OR team of insufficiently captured tissue 322 relative to the edge of the cut. The superimposed incomplete tissue capture alert overlay 304 is applied to energy-based surgical instruments and surgical stapler instruments, etc.

FIG. 13 is an expanded image 3120 of a live feed of the surgical field visualized through a laparoscopic camera during a minimally invasive surgical procedure displayed on an intraoperative data display 3122. 3108 illustrates an end effector 3108 of a surgical instrument 3008 grasping tissue 3110 and a screen with a visible patient panel overlay 3124 in accordance with at least one aspect of the present disclosure. Visible patient panel overlay 3124 may require additional applications to display the content. Intraoperative data display 3122 also displays case information panel overlay 3102 and device panel overlay 3106. In the illustrated example, system notification panel overlay 3104 is hidden.

The following description provides alternative or collaborative augmented reality communications for providing an intuitive or data-dense feed of information to surgeons or other OR personnel. In one aspect, the present disclosure provides mixed reality, augmented reality, and/or augmented reality feedback systems and methods that work together to increase efficient perception of complex data feeds. In one aspect, first and second augmented data feeds are provided, and at least one of the first or second data feeds generates an overlay that is not part of the visual display. In another aspect, the visual display portion of the data feed or overlay may include multiple augmented reality display systems operating in series with each other or independently positioned. In another aspect, the non-visual communication of data may be through auditory, somatosensory, tactile, chemical (including odor), or thermal perception by the user, alone or in combination. The disclosure herein describes collaborative augmented reality, mixed reality, or AR communications, including a collaborative combination of one or more audible overlays and/or somatosensory overlays in cooperation with one or more visual overlays. Explain. Each of these cooperative overlays is described below.

14 illustrates an augmented reality system 5400 including an intermediate signal combiner 64 disposed in a communication path between the imaging module 38 and the surgical hub display 67. The signal combiner 64 combines sensor and/or image data received from the imaging module 38 and/or the AR device 66. The surgical hub 56 receives the combined data from the combiner 64 and overlays the data provided to the display 67, where the overlaid data is displayed. The AR device 66 combines data feeds from the imaging device 68 and the sensor device 5402. The combined data feed can be provided from the AR device 66 to the combiner 64 for further overlaying with the data feed from the imaging module 38. In one aspect, the visual portion of the augmented overlay includes multiple cooperating image displays. In one aspect, the multiple cooperating image displays function in series with one another. In another aspect, the multiple cooperating image displays are independently positioned. In various aspects, the data feed can be provided by the sensor 90, speaker 91, and haptic controller 92 portions of the AR device 66.

Imaging device 68 may be a digital video camera. Signal combiner 64 is a wireless head-up display adapter for coupling to an AR device 66 located within the communication path of display 67 to the console that allows surgical hub 56 to overlay data onto display 67. may include. In various aspects, the AR device 66 transmits a combination of images, audible sounds, and/or somatosensory signals from the imaging module 38, the imaging device 68, and/or the sensor device 5402 to the surgeon as described below. It may be displayed in the form of an overlay as part of the provided view. In various aspects, the display 67 displays a combination of images, audible sounds, and/or somatosensory signals from the imaging module 38, the imaging device 68, and/or the sensor device 5402, generally in an OR as described below. It may be displayed in the form of an overlay as part of the provided view. In one aspect, sensor device 5402 may be coupled to AR device 66 via filter 5404. In other aspects, an amplifier 5406 may be placed between the filter 5404 and the AR device 66 to amplify the signal from the sensor device 5402.

The sensor device 5402 may be an audio device, a somatosensory device, and/or a combination thereof. Somatosensory devices include, but are not limited to, thermal, chemical, and mechanical devices as described below. In one aspect, the sensor device 5402 may be configured to sense various audible input signals, such as sound 5408, biomarkers 5410, heartbeat/rhythm 5412, among others. The audible signals may be filtered by a filter 5404 and amplified by an amplifier 5406. In one aspect, the surgeon 73 or other OR personnel may receive stimulation input from various somatosensory stimuli, such as thermal stimuli 5414, chemical stimuli 5416, mechanical stimuli 5418, etc. The audio input may be overlaid with an image received from the imaging device 68 and/or imaging module 38. Similarly, the somatosensory stimulation input may be overlaid with an image received from the imaging device 68 and/or imaging module 38. The audio and somatosensory overlays may be displayed on the AR device 66 and/or the display 67.

In one aspect, the present disclosure provides an audible overlay based on an audible signal generated by a sensor device 5402. The audible signal may be filtered by filter 5404 to exclude or amplify certain OR audio relative to others and amplified by amplifier 5405, for example. Filtering may increase or decrease the attention span of the overlay to control the amount of user attention to the overlay. In one aspect, sensor device 5402 receives voice commands and converts the voice commands into electrical signals. In one aspect, sensor device 5402 includes a speech-to-text converter. Filter 5404 may be configured to filter specific commands to the OR assistant, such as positioning the manual handle up/down or left/right for assistance to surgeon 73 at the robot console. In another aspect, the filtering or amplification may be based on keywords that the surgeon 73 can speak during the surgical procedure.

The audible overlay provides an alternative method to help verify where the surgeon 73 is in the treatment plan. For example, if the augmented reality system 5400 knows that the next step in a surgical procedure requires a grasper to move an organ for access, the surgeon may say the word "grasper". The augmented reality system 5400 can then verify that the surgeon 73 and the surgical procedure are tracking properly. This additional control may be needed when the surgeon 73 deviates slightly from the initial surgical procedure. For example, if a surgeon 73 requests a surgical instrument such as a surgical stapler, the augmented reality system 5400 will recognize the word "surgical stapler" or simply "stapler" and display the surgeon's 73 main screen 67 or AR device 66 as Adjust with a specific combination of visual and audible overlays for placement and firing of digitally connected surgical instruments. Augmented reality system 5400 further provides pre-use checks of surgical instruments and communications before surgeon 73 fires the surgical instrument. The volume of the audible overlay portion may be increased or decreased based on the severity of the situation. Augmented reality system 5400 may be configured to reduce the volume of all background noise (eg, radio, telephone, etc.) if the current surgical procedure situation is deemed critical or high risk. Upon learning that a particular user is deaf, the augmented reality system 5400 responds by increasing the volume or adjusting the pitch to help the surgeon 73 hear clearly. be able to.

In another aspect, surgical procedure specific filtering can be used to separate audio to a particular surgical procedure. This type of filtering can be determined based on a risk benefit analysis and an assessment of the historical risks of the particular surgical procedure. For example, if the particular surgical procedure is a cholecystectomy, there is a relatively low need for the surgeon 73 to adjust to the patient's heart rate and blood pressure. Given the short surgical procedure time combined with a low risk of intraoperative complications, the augmented reality system 5400 can conclude after a risk assessment calculation that there is no reason for the surgeon 73 to require an audible overlay. However, the surgeon 73 can disable the augmented reality system 5400 commanding the presence of the audible overlay and other collaborative overlays.

In another aspect, the audible overlay may include overlaying audibly correlated feedback to specific patient biomarker 5410 data. The biomarker 5410 data may be the patient's heartbeat, with a corresponding audible overlay of the patient's heartbeat allowing the surgeon 73 to hear the patient's heart as if using a stethoscope. An overlay of sensed nerve stimulation may be employed to determine nerve proximity and overload. This may be done by increasing or decreasing both the volume and frequency of the audible overlay, allowing the surgeon 73 to correlate the location of the electrical device with critical nerves.

In yet another aspect, the audible overlay may include overlaying a predetermined heartbeat/rhythm 5412 to allow the surgeon to keep pace with the physical response. The heartbeat/rhythm 5412 can also be tuned to a surgical procedure or a key rhythm remote from the patient. Audible indication of unwanted tissue contact from robotic surgical devices outside the surgical field of view.

In one aspect, the present disclosure provides a somatosensory overlay based on one or more somatosensory signals detected by either the sensor device 5402 or the surgeon 73. In some aspects, the somatosensory signals may be received, for example, by the sensor device 5402, filtered by a filter 5404, and amplified by an amplifier 5406. The somatosensory signals may be employed as a somatosensory overlay in cooperation with any of the audible and/or image overlays described herein.

In one aspect, somatosensory signals primarily encode absolute and relative changes in temperature, within a non-noxious range, from specific thermoreceptors, or other non-specialized sensory receptors, or sensory neurons. may be a thermal signal received by the surgeon 73 to directly stimulate the receiving portion of the . The change in temperature of the surgical instrument handle or portion of the handle allows the surgical instrument handle to be used as a surrogate for the temperature of associated components of the surgical instrument. For example, referring also to FIG. 9, in certain powered surgical staplers 77, the motor and gearbox are located within the handle of the surgical stapler 77. This is the same area that surgeon 73 uses to hold and actuate surgical stapler 77. This area experiences an increase in temperature when surgical stapler 77 is used. Temperature is directly related to the work that surgical stapler 77 needs to perform during a surgical procedure. The surgeon 73 can feel this temperature increase during the surgical procedure. Surgeon 73 can use this physical temperature data input as a proxy for how surgical instrument 77 is operating and continues to operate. The handle becoming too hot to grip is a clear indication that the surgical instrument 77 is operating beyond normal use. Motor heating reduces the optimal performance of the system, and this reduction can directly impact the outcome of the surgical procedure. For example, surgical staplers may not be able to cut/staple tissue within the tightened jaws, thereby complicating the surgical procedure.

In another aspect, the somatosensory signals may stimulate certain chemical receptors that are primarily responsive to chemical stimuli in the OR environment. These may be sensed by the surgeon's 73 sense of taste and smell, for example, the surgeon 73 may smell burning electronics and indicate the result and to turn off the surgical instruments as necessary. In some aspects, the somatosensory signals may be detected by the sensor device 5402 and may then be used to generate a somatosensory overlay in cooperation with any one of the audible and/or image overlays.

In another aspect, somatosensory signals may stimulate specific mechanoreceptors that primarily respond to touch, tactile, or vibrational stimuli, among others. In one aspect, the mechanical vibrations of the surgical instrument vibration may be detected by either the surgeon 73 or the sensor device 5402. The sensed mechanical vibrations are used by the augmented reality system 5400 to indicate that the current movement or orientation of the surgical instrument 77 has a non-optimal result associated therewith and therefore requires correction. Augmented reality system 5400 may be configured to indicate that the drive was "out of normal" at the end of the current drive/cut stroke. This may be an indication that the clamping force of the surgical stapler 77 jaws was out of range or that the firing force was higher than expected. These conditions may be indicated by a series of tactile buzzers to distinguish between different indications. In one example, vibrations at the end of a stroke of surgical stapler 77 may indicate that surgical stapler 77 cannot move further in the indicated direction. In another example, vibrations in the handle may indicate that the "hot blade" of the energy-based surgical instrument is contacting secondary tissue and thus attempting to avoid critical structures. Certain types of vibrations may indicate that the robot arm is at maximum extension. Augmented reality system 5400 may be configured to provide a tactile pulse sequence to alert surgeon 73 that the maximum value has been reached.

In another aspect, the mechanoreceptor may respond to variations in the surgical stapler's actuation force threshold. The variation in actuation force provides feedback to the user that it is not desirable to actuate surgical stapler 77 at this particular time. For example, while initially clamping the jaws to tissue using the surgical stapler 77, the surgeon 73 can physically feel how tightly the tissue is clamped within the jaws. This direct physical input, with the sensor reading indicating the "measured" value, provides two different inputs for this value.

In another aspect, a mechanoreceptor may respond to an expandable stimulation element to indicate that its control is undesirable to use. The extensible portion may be an extensible pattern that provides a different "feel" rather than just a post or cleat. For example, the knife of the surgical stapler 77 is partially extended, the surgeon 73 attempts to release the closure system, and the extensible element is actuated on the release button to indicate that it cannot be actuated at this time or in this sequence. The extensible portion may be an extensible pattern that provides a different "feel" rather than just a post or cleat.

In another aspect, the mechanoreceptors can respond to force feedback to disable or inhibit an action from being performed. For example, when the surgeon 73 attempts to fire the surgical stapler 77, the surgical hub detects a foreign object currently located within the jaws. If the surgeon 73 then attempts to pull the firing trigger, the device pushes the trigger back, preventing the surgeon from depressing it.

In another aspect, a combination of multiple somatosensory outputs may be used simultaneously to communicate interrelated data feeds. One skilled in the art will appreciate the need to distinguish between two separate indications, both having the same standard feedback mode. In one aspect, one system feedback may indicate that it is unavailable based on the state of another system. In another aspect, a backlit LED may be placed in the control device to indicate a lack of functionality. The LED should be configured to clearly indicate two separate faults or conditions. Additionally, the LED system is configured to resolve conflicts between multiple similar indicators activated simultaneously. Force sensing may be provided to the OR assistant when inserting a circular stapler or rectal sizer.

In another aspect, a display 67 within the OR may be used to indicate a fault or indication. Overlays are displayed without interfering with key display information. In one aspect, overlay information is displayed around a perimeter border of main display 67 to reduce interference and can change color along with tissue identification information.

In another aspect, the somatosensory overlay can detect, e.g., instrument collision, notification of impending inadvertent tissue contact, hot instrument contact with adjacent tissue, and vibration to ensure that the output is clearly transmitted. Alternative feedback for device interaction based on AR device 66 haptic feedback may be included, such as combined heated gloves and/or high energy device completion of cycles. Yu et al. (Nature 575, 473-479; 2019) describes a wearable skin-integrated technology that can attach to the skin and vibrate. These devices include 1.4g, 12-18mm sized actuators that are wirelessly powered and controlled.

In another aspect, the somatosensory overlay may include visual feedback. A visual feedback somatosensory overlay may be used to indicate completion of advanced energy device cycling, incompatible system assembly, and disabled devices in the current configuration. Additional overlays include audible feedback through a speaker. Safety overlays for handheld and robotic surgical instruments may be provided.

FIG. 15 shows a method 5500 of presenting an augmented overlay during a surgical procedure. Referring also to FIGS. 10 and 14, according to method 5500, imaging device 68 captures an actual image of a surgical field during a surgical procedure (5502). A first data feed is generated (5504) based on the captured real images. A first data feed is provided to AR device 66. Sensor device 5402 generates a second data feed (5506). A second data feed is also provided to AR device 66. AR device 66 with augmented reality display 89 presents an augmented overlay based on the first and second data feeds (5508). The enhanced overlay includes a visual portion and a non-visual visual portion.

In one aspect, according to method 5500, sensor device 5402 may receive tactile, audible, chemical, or thermal signals from one or more sources, such as audio 5408, biomarkers 5410, heartbeat/rhythm 5412, thermal stimuli 5414, chemical stimuli 5416, or mechanical stimuli 5418. The tactile, audible, chemical, or thermal signals, or any combination thereof, are coupled to a non-visual portion of the augmentation overlay.

In one aspect, according to method 5500, filter 5404 can filter the signal received by sensor device 5402. Amplifier 5406 amplifies the filtered signal.

In one aspect, according to method 5500, display 67 coupled to AR device 66 displays an enhanced overlay. Imaging module 38 generates a third data feed, which is combined with the enhanced overlay, and the combined enhanced overlay is displayed on display 67. A combiner 64 combines the third data feed with the enhancement overlay. Surgical hub 56 communicates the enhanced overlay to display 67 .

Situational awareness is the ability of some aspects of a surgical system to determine or infer information relevant to a surgical procedure from data received from a database and/or instruments. The information may include the type of procedure being performed, the type of tissue being operated on, or the body cavity being treated. According to the context information regarding the surgical procedure, the surgical system may, for example, control modular devices (e.g., robotic arms and/or robotic surgical tools) connected to it, and may be used by the surgeon during the course of the surgical procedure. can be improved in a manner that provides contextualized information or suggestions.

FIG. 16 shows a timeline of situational awareness surgical procedures. FIG. 71 illustrates an example surgical procedure timeline 5200 and contextual information that the surgical hub 5104 may derive from data received from the data source 5126 at each step of the surgical procedure. Timeline 5200 illustrates the general steps that nurses, surgeons, and other medical personnel would take during the course of a segmental lung resection procedure, beginning with surgical site setup and ending with transporting the patient to the postoperative recovery room. The following steps are shown. Situational-aware surgical hub 5104 transmits data to data source 5126, including data generated each time a medical personnel uses a modular device 5102 paired with surgical hub 5104, throughout the course of a surgical procedure. Receive from. Surgical hub 5104 receives this data from paired modular devices 5102 and other data sources 5126 and receives new data, such as which steps of a procedure are being performed at any given time. Once performed, inferences (i.e., contextual information) regarding the ongoing procedure can be continuously derived. The situational awareness system of the surgical hub 5104 may, for example, record data regarding a procedure to generate reports, verify steps being taken by medical personnel, record data that may be related to a particular procedure step, or Providing a prompt (e.g., via a display screen), adjusting the modular device 5102 based on the context (e.g., activating a monitor, adjusting the FOV of a medical imaging device, or an ultrasonic surgical instrument) or changing the energy level of the RF electrosurgical instrument) and any other such operations described above.

First, at 5202, hospital personnel retrieves the patient's EMR from the hospital's EMR database. Based on the patient data selected in the EMR, surgical hub 5104 determines that the procedure being performed is thoracic surgery.

Second, 5204, personnel scan the incoming medical supplies for the procedure. The surgical hub 5104 cross-references the scanned supplies with a list of supplies utilized in various types of procedures and verifies that the mix of supplies is compatible with a thoracic procedure. Additionally, the surgical hub 5104 may also determine that the procedure is not a wedge resection (either because the incoming supplies do not include the specific supplies required for a thoracic wedge resection or are not otherwise compatible with a thoracic wedge resection).

In a third 5206, the medical personnel scans the patient band via a scanner 5128 communicatively connected to the surgical hub 5104. The surgical hub 5104 can then verify the patient's identity based on the scanned data.

Fourth, 5208, the medical personnel turns on the auxiliary device. The auxiliary equipment utilized may vary according to the type of surgical procedure and the technique used by the surgeon, but in this illustrative case includes a smoke evacuator, an insufflator, and a medical imaging device. Once activated, the ancillary device that is the modular device 5102 can automatically pair with a surgical hub 5104 located within a particular proximity of the modular device 5102 as part of its initialization process. . Surgical hub 5104 can then derive contextual information regarding the surgical procedure by detecting the type of modular device 5102 with which it is paired during this preoperative or initialization phase. In this particular example, surgical hub 5104 determines that the surgical procedure is a VATS procedure based on this particular combination of paired modular devices 5102. Based on a combination of data from the patient's EMR, the list of medical supplies used in the procedure, and the type of modular device 5102 that connects to the hub, the surgical hub 5104 generally estimates the specific procedure that the surgical team will perform. can do. Once the surgical hub 5104 knows what particular procedure is being performed, the surgical hub 5104 then reads the steps of that procedure from memory or from the cloud and then reads the steps of that procedure from the connected data source 5126 (e.g. Data subsequently received from the modular device 5102 and the patient monitoring device 5124) can be cross-referenced to estimate which steps of the surgical procedure the surgical team is performing.

In the fifth 5210, personnel attach EKG electrodes and other patient monitoring devices 5124 to the patient. EKG electrodes and other patient monitoring devices 5124 can be paired with surgical hub 5104. When surgical hub 5104 begins receiving data from patient monitoring device 5124, surgical hub 5104 verifies that the patient is at the surgical site.

Sixth 5212, the medical personnel induces anesthesia in the patient. Surgical hub 5104 determines whether the patient is under anesthesia based on data from modular device 5102 and/or patient monitoring device 5124, including, for example, EKG data, blood pressure data, ventilator data, or a combination thereof. You can guess that. Upon completion of the sixth step 5212, the preoperative portion of the lung segmentectomy surgery is completed and the surgical department begins.

Seventh, 5214, the patient's lung being operated on is collapsed (while ventilation is switched to the contralateral lung). The surgical hub 5104 can infer from the ventilator data that the patient's lung has been collapsed. Because the surgical hub 5104 can compare the detection of the patient's lung being collapsed to the expected steps of the procedure (which may be accessed or read in advance), it can infer that the surgical portion of the procedure has begun, thereby determining that collapsing the lung is the first surgical step in this particular procedure.

In the eighth 5216, the medical imaging device 5108 (e.g., a scope) is inserted and video footage from the medical imaging device is initiated. The surgical hub 5104 receives the medical imaging device data (i.e., still image data or real-time live streaming video) through a connection to the medical imaging device. Upon receiving the medical imaging device data, the surgical hub 5104 can determine that the laparoscopic portion of the surgical procedure has begun. Additionally, the surgical hub 5104 can determine that the particular procedure being performed is a segmentectomy as opposed to a lobectomy (note that a wedge resection has not already been taken into account by the surgical hub 5104 based on the data received in the second step 5204 of the procedure). Data from the medical imaging device 124 (FIG. 2) can be used to determine contextual information regarding the type of procedure being performed in a variety of ways, such as by determining the angle at which the medical imaging device is oriented relative to visualization of the patient's anatomy, by monitoring the number or medical imaging devices being utilized (i.e., activated and paired with the surgical hub 5104), and by monitoring the type of visualization equipment being utilized.

For example, one technique for performing a VATS lobectomy places the camera above the diaphragm in the anteroinferior corner of the patient's thoracic cavity, whereas one technique for performing a VATS segmentectomy involves The camera is placed in an intercostal position anterior to the segmental fissure. The situational awareness system can be trained to recognize the location of the medical imaging device according to the visualization of the patient's anatomy, using, for example, pattern recognition techniques or machine learning techniques. As another example, one technique for performing a VATS lobectomy utilizes a single medical imaging device, while another technique for performing a VATS segmentectomy utilizes multiple cameras. . As yet another example, one technique for performing a VATS segmentectomy utilizes an infrared light source (which can be communicatively coupled to a surgical hub as part of the visualization system) to visualize the segmental cleft. However, this is not utilized in VATS lobectomy. By tracking any or all of this data from the medical imaging device 5108, the surgical hub 5104 determines the specific type of surgical procedure being performed and/or used for the specific type of surgical procedure. It is possible to determine the technology being used.

At ninth 5218, the surgical team begins the cutting step of the procedure. The surgical hub 5104 receives data from an RF or ultrasound generator indicating that an energy instrument is being fired, so it can be inferred that the surgeon is in the process of dissecting and separating the patient's lungs. . The surgical hub 5104 cross-references the received data with the readout steps of the surgical procedure to determine whether the energy instrument being fired at this point in the process (i.e., after the procedure steps described above are completed) It can be determined that the step corresponds to the incision step.

In a tenth step 5220, the surgical team proceeds to the ligation step of the procedure. Surgical hub 5104 receives data from the surgical stapling and cutting instrument indicating that the instrument is being fired, so it can assume that the surgeon is ligating the artery and vein. Similar to the previous step, the surgical hub 5104 can derive this estimate by cross-referencing the receipt of data from the surgical stapling and cutting instruments with the steps in the process that have been read.

In an eleventh step 5222, the segmentectomy portion of the procedure is performed. The surgical hub 5104 estimates that the surgeon is transecting parenchymal tissue based on data from the surgical instrument, including data from the staple cartridge. The cartridge data may correspond to, for example, the size or type of staples being fired by the instrument. The cartridge data may indicate the type of tissue being stapled and/or transected for different types of staples being applied to different types of tissue. The type of staples being fired is applied to parenchymal tissue or other similar tissue types and the surgical hub 5104 can estimate that a segmentectomy procedure is being performed.

Then, in a twelfth step 5224, a nodule dissection step is performed. The surgical hub 5104 may infer that the surgical team is dissecting the nodule and performing a leak test based on data received from the generator indicating that the RF or ultrasound instrument is being fired. . For this particular procedure, the RF or ultrasound instruments utilized after the parenchymal tissue has been transected are compatible with the node dissection step, which allows the surgical hub 5104 to make this estimate. Because different instruments are better suited to specific tasks, surgeons may choose between surgical stapling/cutting instruments and surgical energy (i.e., RF or ultrasound) instruments depending on the particular step during the procedure. Note that the Thus, the particular sequence in which the stapling/cutting instruments and surgical energy instruments are used can indicate which step of the procedure the surgeon is performing. Upon completion of the twelfth step 5224, the incision is closed and the post-operative portion of the procedure begins.

In a thirteenth step 5226, the patient is removed from anesthesia. Surgical hub 5104 may estimate that the patient is emerging from anesthesia based on, for example, ventilator data (ie, the patient's breathing rate begins to increase).

Finally, at fourteenth 5228, the medical personnel removes the various patient monitoring devices 5124 from the patient. Therefore, the surgical hub 5104 can assume that the patient is being transferred to the recovery room when the hub loses the EKG, BP, and other data from the patient monitoring device 5124. According to data received from various data sources 5126 communicatively coupled to surgical hub 5104, surgical hub 5104 can determine or estimate when each step of a given surgical procedure is occurring. .

As shown in the first step 5202 of the timeline 5200 shown in FIG. 16, in addition to utilizing patient data from the EMR database(s) to estimate the type of surgical procedure to be performed, the patient data can also be utilized by the situational awareness surgical hub 5104 to generate control adjustments of the paired modular devices 5102.

Various additional aspects of the subject matter described herein are illustrated in the numbered examples below.

Example 1: An augmented reality display system for use during a surgical procedure, the system comprising: an imaging device that captures real images of a surgical field during the surgical procedure and generates a first data feed; a sensor device that generates a data feed; an augmented reality device that includes an augmented reality display that generates an augmented overlay that includes a visual portion and a non-visual visual portion based on the first data feed and the second data feed; an augmented reality processor that generates a first data feed, receives a second data feed, and combines the first data feed and the second data feed to generate an augmented overlay. display system.

Example 2: An augmented reality display system as described in Example 1, in which the visual portion of the augmented overlay comprises multiple cooperating image displays.

Example 3: The augmented reality display system of Example 2, wherein the multiple cooperative image displays function in series with each other.

Example 4: An augmented reality display system as described in any one of Examples 2 to 3, in which the multiple cooperating image displays are independently positioned.

Example 5: An augmented reality display system according to any one of Examples 1 to 4, wherein the non-visual portion of the augmentation overlay is tactile, audible, chemical, or thermal, or any combination thereof.

Example 6: The augmented reality display system according to any one of Examples 1 to 4, comprising a filter between the sensor device and the augmented reality device.

Example 7: The augmented reality display system as described in Example 6, comprising an amplifier between the filter and the augmented reality device.

Example 8: The augmented reality device as in any one of Examples 1-7, comprising a sensor, speaker, or haptic controller, or a combination thereof, that generates a data feed for the non-visual portion of the augmented overlay. Augmented reality display system described.

Example 9: An augmented reality display system according to any one of Examples 1 to 8, comprising a display coupled to an augmented reality device.

Example 10: The augmented reality display system of Example 9, comprising an imaging module that generates a third data feed, the third data feed being combined with an augmented overlay and displayed on the display.

Example 11: The augmented reality display system of Example 10, comprising a combiner that combines the third data feed with the augmented overlay.

Example 12: The augmented reality display system of Example 11, comprising a surgical hub between the coupler and the display, the surgical hub communicating the augmented overlay to the display.

Example 13: A method of presenting an augmented overlay during a surgical procedure, the method comprising: capturing, by an imaging device, a real image of a surgical field during the surgical procedure; generating, by the imaging device, a first data feed based on the captured real image; generating, by a sensor device, a second data feed; and presenting, by an augmented reality device having an augmented reality display, an augmented overlay including a visual portion and a non-visual portion based on the first data feed and the second data feed.

Example 14: The method of example 13, comprising receiving a tactile, audible, chemical, or thermal signal, or any combination thereof, from a sensor device, and coupling the tactile, audible, chemical, or thermal signal, or any combination thereof, to a non-visual portion of the augmentation overlay.

Example 15: The method of any one of Examples 13-14, comprising filtering the signal received by the sensor device by a filter.

Example 16: The method of Example 15 comprising amplifying the filtered signal with an amplifier.

Example 17: The method of any one of Examples 13 to 17, comprising displaying an augmented overlay on a display coupled to the augmented reality device.

Example 18: Generating a third data feed by an imaging module, combining the third data feed with an enhanced overlay, and displaying the combined enhanced overlay on a display. Method as described in Example 17.

Example 19: The method of example 18, including combining a third data feed with the augmentation overlay by a combiner.

Example 20: The method of Example 19 comprising communicating the augmented overlay to the display by the surgical hub.

While several embodiments have been shown and described, it is not the intention of the applicant to restrict or limit the scope of the appended claims to such details. Numerous modifications, variations, changes, substitutions, combinations, and equivalents of these embodiments can be implemented and will occur to those skilled in the art without departing from the scope of the present disclosure. Moreover, the structure of each element associated with the described embodiments can alternatively be described as a means for providing the function performed by that element. Also, although materials are disclosed with respect to specific components, other materials may be used. It is therefore to be understood that the above description and the appended claims are intended to cover all such modifications, combinations, and variations as falling within the scope of the disclosed embodiments. The appended claims are intended to cover all such modifications, variations, changes, substitutions, modifications, and equivalents.

The detailed descriptions above have described various forms of apparatus and/or processes using block diagrams, flow diagrams, and/or example embodiments. To the extent that such block diagrams, flow diagrams and/or examples include one or more features and/or operations, it will be understood by those skilled in the art that such block diagrams, flow diagrams and/or examples include one or more features and/or operations. Each included feature and/or operation may be implemented individually and/or collectively by a variety of hardware, software, firmware, or virtually any combination thereof. Those skilled in the art will appreciate that some aspects of the forms disclosed herein can be implemented, in whole or in part, as one or more computer programs running on one or more computers (e.g., as one or more computer programs running on one or more computers). (as one or more programs running on a computer system); as one or more programs running on one or more processors (e.g., as one or more programs running on one or more microprocessors); Designing circuits and/or writing software and/or firmware code that can equivalently be implemented on an integrated circuit as firmware, or as virtually any combination thereof, utilizes the present disclosure. It will be understood that it is within the skill of those skilled in the art. In addition, those skilled in the art will appreciate that the features of the subject matter described herein can be distributed in a variety of forms as one or more program products, The specific form of the subject matter applies without regard to the particular type of signal-bearing medium used to actually carry out the distribution.

The instructions used to program the logic to implement the various disclosed aspects may be stored in memory within the system, such as dynamic random access memory (DRAM), cache, flash memory, or other storage. Additionally, the instructions may be distributed over a network or by other computer-readable media. Thus, a machine-readable medium may include any mechanism for storing or transmitting information in a form readable by a machine (e.g., a computer), including floppy diskettes, optical disks, compact disks, read-only memory (CDs), etc. - ROM), as well as magneto-optical disks, read only memory (ROM), random access memory (RAM), erasable programmable read only memory (EPROM), electrically erasable programmable read only memory (EEPROM), magnetic or optical cards, Flash memory or other tangible machine-readable storage used to transmit information over the Internet via electrical, optical, acoustic, or other forms of propagating signals (e.g., carrier waves, infrared signals, digital signals, etc.) but not limited to. Thus, non-transitory computer-readable media includes any type of tangible machine-readable medium suitable for storing or transmitting electronic instructions or information in a form readable by a machine (eg, a computer).

As used in any aspect of this specification, the term "control circuitry" may refer to, for example, hardwired circuitry, programmable circuitry (e.g., a computer processor, processing unit, processor, microcontroller, microcontroller unit, controller, digital signal processor (DSP), programmable logic device (PLD), programmable logic array (PLA), or field programmable gate array (FPGA) that includes one or more individual instruction processing cores), state machine circuitry, firmware that stores instructions executed by the programmable circuitry, and any combination thereof. The control circuitry may be embodied, collectively or individually, as circuits that form part of a larger system, such as, for example, an integrated circuit (IC), an application specific integrated circuit (ASIC), a system on a chip (SoC), a desktop computer, a laptop computer, a tablet computer, a server, a smartphone, etc. Thus, as used herein, "control circuitry" includes, but is not limited to, electrical circuits having at least one discrete electrical circuit, electrical circuits having at least one integrated circuit, electrical circuits having at least one application specific integrated circuit, electrical circuits forming a general purpose computing device configured by a computer program (e.g., a general purpose computer configured by a computer program to at least partially execute the processes and/or devices described herein, or a microprocessor configured by a computer program to at least partially execute the processes and/or devices described herein), electrical circuits forming a memory device (e.g., a form of random access memory) and/or electrical circuits forming a communication device (e.g., a modem, a communication switch, or an optical-electrical facility). Those skilled in the art will recognize that the subject matter described herein may be implemented in analog or digital form, or some combination thereof.

As used in any aspect of this specification, the term "logic" may refer to an application, software, firmware, and/or circuitry configured to perform any of the operations described above. Software may be embodied as a software package, code, instructions, instruction sets, and/or data recorded on a non-transitory computer-readable storage medium. Firmware may be embodied as code, instructions, or instruction sets, and/or hard-coded (e.g., non-volatile) data in a memory device.

As used in any aspect herein, the terms "component," "system," "module," etc. refer to control circuitry, computer-related entities, hardware, combinations of hardware and software, software, or any running software.

As used in any aspect herein, an "algorithm" refers to a self-consistent sequence of steps leading to a desired result, where a "step" refers to, but is not required to include, the storage, transfer, combination, etc. Refers to the manipulation of physical quantities and/or logical states, which can take the form of electrical or magnetic signals, which can be compared, compared, and otherwise manipulated. It is common practice to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, or the like. These and similar terms may be associated with the appropriate physical quantities or are simply convenient labels applied to these quantities and/or conditions.

The network may include a packet-switched network. The communication devices may communicate with each other using a selected packet-switched network communication protocol. One exemplary communication protocol may include an Ethernet communication protocol that may enable communication using Transmission Control Protocol/Internet Protocol (TCP/IP). The Ethernet protocol may conform to or be compatible with the Ethernet standard entitled "IEEE 802.3 Standard" issued in December 2008 by the Institute of Electrical and Electronics Engineers (IEEE), and/or later versions of this standard. Alternatively or additionally, the communication devices may communicate with each other using the X.25 communication protocol. The T.25 communication protocol may conform to or be compatible with standards promulgated by the International Telecommunications Union-Telecommunications Standardization Sector (ITU-T). Alternatively or additionally, the communication devices may communicate with each other using a frame relay communication protocol. The frame relay communication protocol may conform to or be compatible with standards promulgated by the Consultative Committee for International Telegraph and Telephone (CCITT) and/or the American National Standards Institute (ANSI). Alternatively or additionally, the transceivers may be capable of communicating with each other using an asynchronous transfer mode (ATM) communication protocol. The ATM communications protocol may conform to or be compatible with the ATM standard published by the ATM Forum in August 2001, entitled "ATM-MPLS Network Interworking 2.0," and/or any later version of this standard. Of course, different and/or later developed connection-oriented network communications protocols are equally contemplated herein.

Unless explicitly stated otherwise, as is clear from the foregoing disclosure, the terms "processing," "computing," and "calculating" are used throughout the foregoing disclosure. , ``determining,'' ``displaying,'' and other terms, refers to the use of terms such as The actions and processing of a computer system or similar electronic computing device that manipulate and convert such information into other data similarly represented as physical quantities in memories or registers or other such information storage, transmission, or display devices. I hope you understand what I'm referring to.

One or more components are defined herein as "configured to," "configurable to," "operable/operating." (operable/operative to), “adapted/adaptable,” “able to,” “conformable/conformed to.” ” may be mentioned. Those skilled in the art will understand that "configured to" generally refers to active components and/or inactive components and/or standby components, unless the context requires otherwise. It will be understood that elements can be included.

The terms "proximal" and "distal" are used herein with reference to the clinician manipulating the handle portion of the surgical instrument. The term "proximal" refers to the part closest to the clinician, and the term "distal" refers to the part located away from the clinician. It will be further understood that for convenience and clarity, spatial terms such as "vertical," "horizontal," "above," and "below" may be used herein with respect to the drawings. However, surgical instruments may be used in many orientations and positions, and these terms are not intended to be limiting and/or absolute.

Those skilled in the art will generally understand that the terms used herein, and particularly in the appended claims (e.g., the text of the appended claims), are generally intended as "open" terms. (For example, the term "including" should be interpreted as "including but not limited to," and the term "having" should be interpreted as "including but not limited to.") should be interpreted as “having at least” and the term “includes” should be interpreted as “includes but is not limited to.” should be done, etc.). Further, if a specific number is intended in an introduced claim recitation, such intent will be clearly stated in the claim, and if no such statement is present, no such intent exists. will be understood by those skilled in the art. For example, as an aid to understanding, the following appended claims incorporate the introductory phrases "at least one" and "one or more" into claim statements. may be included in order to However, the use of such phrases does not apply when the claim is introduced by the indefinite article "a" or "an," even if the introductory phrase "one or more" or "at least one" appears within the same claim. and the indefinite article "a" or "an," any particular claim containing such an introduced claim statement is limited to a claim containing only one such statement. (e.g., "a" and/or "an" should generally be interpreted to mean "at least one" or "one or more") ). The same applies when introducing claim statements using definite articles.

In addition, those skilled in the art will recognize that even if a specific number is specified in an introduced claim description, such a description should typically be interpreted to mean at least the number recited (e.g., a description of "two items" without other qualifiers generally means at least two items, or more than two items). Furthermore, when a notation similar to "at least one of A, B, and C, etc." is used, such syntax is generally intended in the sense that a person skilled in the art would understand the notation (e.g., "a system having at least one of A, B, and C" includes, but is not limited to, systems having only A, only B, only C, both A and B, both A and C, both B and C, and/or all of A, B, and C, etc.). When a notation similar to "at least one of A, B, or C, etc." is used, such syntax is generally intended in the sense that one of ordinary skill in the art would understand the notation (e.g., "a system having at least one of A, B, or C" includes, but is not limited to, systems having only A, only B, only C, both A and B, both A and C, both B and C, and/or all of A, B, and C, etc.). Furthermore, one of ordinary skill in the art will understand that any disjunctive word and/or phrase expressing two or more alternative terms should typically be understood to contemplate the possibility of including one of those terms, either of those terms, or both of those terms, unless the context requires otherwise, whether in the specification, claims, or drawings. For example, the phrase "A or B" will typically be understood to include the possibility of "A" or "B" or "A and B."

With reference to the appended claims, those skilled in the art will appreciate that the acts recited herein may generally be performed in any order. Additionally, while flow diagrams of various operations are shown in sequence(s), it is understood that the various operations may be performed in an order other than that shown, or may be performed simultaneously. It should be. Examples of such alternative orderings include overlapping, interleaving, interleaving, reordering, incremental, preliminary, additional, simultaneous, reverse or other different orderings, unless the context requires otherwise. May include. Additionally, terms such as "responsive to," "related to," or other past tense adjectives generally do not mean otherwise, unless the context requires otherwise. It is not intended that such variations be excluded.

It is worth noting that any reference to "one embodiment," "an embodiment," "an example," "an example," or the like means that the particular feature, structure, or characteristic described in connection with that embodiment is included in at least one embodiment. Thus, the appearances of the phrases "in one embodiment," "in an embodiment," "in an example," and "in one example" in various places throughout this specification do not necessarily all refer to the same embodiment. Furthermore, the particular features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

Any patent application, patent, non-patent publication, or other disclosure material referenced herein and/or listed in any Application Data Sheet is incorporated herein by reference to the extent that the incorporated material is not inconsistent with this specification. As such, and to the extent necessary, the disclosure explicitly set forth in this specification shall take precedence over any conflicting statements incorporated herein by reference. Any content, or portions thereof, that is referred to as being incorporated herein by reference but that conflicts with current definitions, views, or other disclosure content set forth herein shall be incorporated only to the extent that no conflict arises between the incorporated content and the current disclosure content.

In summary, many benefits have been described that result from using the concepts described herein. The above description of one or more forms has been presented for purposes of illustration and description. It is not intended to be exhaustive or limited to the precise form disclosed. Modifications or variations are possible in light of the above teachings. The one or more embodiments are intended to be illustrative of the principles and practical application, thereby enabling those skilled in the art to utilize the various embodiments, with various modifications, as appropriate for the particular application contemplated. It has been selected and written in order to It is intended that the claims presented herewith define the overall scope.

[Embodiment]
(1) An augmented reality display system for use during a surgical procedure, comprising:
an imaging device that captures actual images of a surgical area during the surgical procedure and generates a first data feed;
a sensor device generating a second data feed;
an augmented reality device having an augmented reality display that generates an augmented overlay, the augmented overlay including a visual portion and a non-visual portion, based on the first data feed and the second data feed;
1. A processor comprising:
receiving the first data feed;
receiving the second data feed;
a processor that combines the first data feed and the second data feed to generate the augmented overlay.
2. The augmented reality display system of claim 1, wherein the visual portion of the augmented overlay comprises a plurality of cooperating image displays.
3. The augmented reality display system of claim 2, wherein the multiple cooperating image displays function in series with one another.
(4) The augmented reality display system of claim 2, wherein the multiple cooperating image displays are independently positioned.
5. The augmented reality display system of claim 1, wherein the non-visual portion of the augmentation overlay is tactile, audible, chemical, or thermal, or any combination thereof.

(6) The augmented reality display system of claim 1, further comprising a filter between the sensor device and the augmented reality device.
(7) The augmented reality display system of claim 6, further comprising an amplifier between the filter and the augmented reality device.
8. The augmented reality display system of claim 1, wherein the augmented reality device comprises a sensor, a speaker, or a haptic controller, or a combination thereof, that generates a data feed for the non-visual portion of the augmented overlay.
9. The augmented reality display system of claim 1, further comprising a display coupled to the augmented reality device.
10. The augmented reality display system of claim 9, further comprising an imaging module that generates a third data feed, the third data feed being combined with the augmentation overlay and displayed on the display.

11. The augmented reality display system of embodiment 10, comprising a combiner that combines the third data feed with the augmented overlay.
12. The augmented reality display system of embodiment 11, comprising a surgical hub between the coupler and the display, the surgical hub communicating the augmented overlay to the display.
(13) A method of presenting an expansion overlay during a surgical procedure, the method comprising:
capturing with an imaging device a real image of the surgical area during the surgical procedure;
generating a first data feed based on the captured real image by the imaging device;
generating a second data feed by the sensor device;
presenting, by an augmented reality device with an augmented reality display, an augmented overlay including a visual portion and a non-visual visual portion based on the first data feed and the second data feed.
(14) From the sensor device,
tactile signals,
audible signal,
receiving a chemical signal or a thermal signal;
14. The method of embodiment 13, comprising coupling the tactile signal, the audible signal, the chemical signal, or the thermal signal, or any combination thereof, to the non-visual portion of the augmented overlay.
15. The method of embodiment 13, comprising filtering the signal received by the sensor device with a filter.

16. The method of claim 15, further comprising amplifying the filtered signal with an amplifier.
17. The method of claim 13, further comprising displaying the augmented overlay on a display coupled to the augmented reality device.
(18) generating, by the imaging module, a third data feed;
combining the third data feed with the augmentation overlay;
and displaying the combined augmentation overlay on the display.
19. The method of claim 18, further comprising combining the third data feed with the augmentation overlay by a combiner.
20. The method of claim 19, further comprising communicating the augmentation overlay to the display by a surgical hub.

Claims (20)

An augmented reality display system for use during a surgical procedure, the system comprising:
an imaging device that captures real images of the surgical field during the surgical procedure and generates a first data feed;
a sensor device that generates a second data feed;
an augmented reality device comprising an augmented reality display that generates an augmented overlay that includes a visual portion and a non-visual visual portion based on the first data feed and the second data feed;
A processor,
receiving the first data feed;
receiving the second data feed;
an augmented reality display system comprising: a processor that combines the first data feed and the second data feed to generate the augmented overlay. The augmented reality display system of claim 1, wherein the visual portion of the augmented overlay comprises multiple cooperative image displays. 3. The augmented reality display system of claim 2, wherein the plurality of cooperative image displays function in series with each other. 3. The augmented reality display system of claim 2, wherein the plurality of cooperative image displays are independently located. 2. The augmented reality display system of claim 1, wherein the non-visual portion of the augmented overlay is tactile, audible, chemical, or thermal, or any combination thereof. The augmented reality display system of claim 1, comprising a filter between the sensor device and the augmented reality device. The augmented reality display system of claim 6, further comprising an amplifier between the filter and the augmented reality device. 2. The augmented reality display system of claim 1, wherein the augmented reality device comprises a sensor, speaker, or haptic controller, or a combination thereof, that generates a data feed for the non-visual portion of the augmented overlay. The augmented reality display system of claim 1, comprising a display coupled to the augmented reality device. 10. The augmented reality display system of claim 9, comprising an imaging module that generates a third data feed, the third data feed combined with the augmented overlay and displayed on the display. 11. The augmented reality display system of claim 10, comprising a combiner for combining the third data feed with the augmented overlay. The augmented reality display system of claim 11, comprising a surgical hub between the coupler and the display, the surgical hub communicating the augmented overlay to the display. 1. A method of presenting an augmentation overlay during a surgical procedure, comprising:
capturing an actual image of a surgical area during said surgical procedure with an imaging device;
generating a first data feed based on the captured actual image by the imaging device;
generating, by the sensor device, a second data feed;
and presenting, by an augmented reality device having an augmented reality display, an augmented overlay including a visual portion and a non-visual portion based on the first data feed and the second data feed. From the sensor device,
tactile signals,
audible signal,
receiving a chemical signal or a thermal signal;
14. The method of claim 13, comprising: coupling the tactile signal, the audible signal, the chemical signal, or the thermal signal, or any combination thereof to the non-visual portion of the augmented overlay. 14. The method of claim 13, comprising filtering the signal received by the sensor device by a filter. 16. The method of claim 15, comprising amplifying the filtered signal with an amplifier. 14. The method of claim 13, comprising displaying the augmented overlay on a display coupled to the augmented reality device. generating a third data feed by the imaging module;
combining the third data feed with the enhancement overlay;
18. The method of claim 17, comprising: displaying the combined enhanced overlay on the display. 19. The method of claim 18, further comprising combining the third data feed with the augmentation overlay by a combiner. 20. The method of claim 19, further comprising communicating the augmented overlay to the display by a surgical hub.

Images