JP2024513992A - System with a camera array deployable outside the channel of a tissue-penetrating surgical device - Google Patents

Abstract

A camera system integrated into the trocar. The camera system allows for a wide view of the internal surgical site and 3D mapping of fiducial markers during a laparoscopic procedure. Once inside the patient, the camera system is configured to deploy from a recessed position in the distal end of the trocar. In various aspects, the internal camera system is configured to keep the trocar port free for surgical instruments and provide surgical personnel with an expanded view of the surgical environment.

Description

(Cross reference to related applications)
This application is filed under 35 U.S.C. 119(e) in U.S. Provisional Patent Application No. 63/174,674, entitled "HEADS UP DISPLAY," filed on April 14, 2021, and filed on November 1, 2021. No. 63/284,326, entitled ``INTRAOPERATIVE DISPLAY FOR SURGICAL SYSTEMS,'' filed May 30, 2013, the disclosures of each of which are incorporated herein by reference in their entirety. incorporated into the book.

The present disclosure relates to devices, systems, and methods for providing augmented reality interactive experiences during surgical procedures. During surgical procedures, objects present in the real world are sometimes enhanced by overlaying computer-generated sensory information across multiple sensory modalities, including visual, auditory, tactile, somatosensory, and olfactory senses. It would be desirable to provide an augmented reality interactive experience of a real world environment. In the context of this disclosure, images of the surgical field and surgical instruments and other objects appearing in the surgical field may include computer-generated visual, auditory, tactile, somatosensory, olfactory, or other sensory information about the surgical field and Enhanced by overlaying onto real-world images of instruments or other objects appearing in the surgical field. Images may be streamed in real time or may be static images.

Real-world surgical instruments include a variety of surgical devices that include energy, staplers, or a combination of energy and staplers. Energy-based medical devices include, among others, but not limited to, radio frequency (RF)-based monopolar and bipolar electrosurgical instruments, ultrasound surgical instruments, combinations of RF electrosurgical instruments and ultrasound instruments, RF Includes a combination of an electrosurgical stapler and a mechanical stapler. Surgical stapler devices are surgical instruments used to cut and staple tissue in a variety of surgical procedures, including bariatric, thoracic, colorectal, obstetrics and gynecology, urology, and general surgery.

In various examples, the present disclosure provides a surgical system comprising a surgical device comprising an axial passage defining an outer diameter and an inner diameter, a proximal end, a distal end configured to penetrate tissue, a camera array with individual cameras connected in a ring configuration by elastic connections, and a detachable attachment trigger configured to extend the camera array from a first recessed position from the inner diameter of the distal end of the axial passage to a second deployed position in which the camera array is positioned circumferentially around the outer diameter of the distal end of the axial passage; an augmented reality (AR) device; and a surgical hub communicatively coupled to the camera array and the AR device, the surgical hub comprising control circuitry coupled to a memory, the control circuitry configured to receive a plurality of video feeds from the camera array, identify physical markers on the video feeds, and display the physical markers on the AR display.

In various examples, the present disclosure provides a surgical device comprising: a camera array including individual cameras connected in a ring configuration by resilient connections, the cameras being communicatively connectable to a surgical hub; an elongate penetrating member having an array, a proximal end, a distal end further comprising a tissue penetrating tip, and an axial passageway through the elongate penetrating member and the tissue penetrating tip, the axial passageway having an inner diameter; an axial passageway dimensioned to receive the camera array in the first recessed position; and an axial passageway dimensioned to receive the camera array in the first recessed position; a removable attachment trigger configured to extend the camera array to a second deployment position circumferentially positioned about the outer diameter of the distal end of the elongated penetrating member; , provides surgical devices.

In various examples, the present disclosure provides a method for displaying a surgical location inside a patient, the method including: receiving, by a surgical hub, a video feed from a camera located inside the patient; identifying, by the surgical hub, physical markers inside the patient; determining, by the surgical hub, a target location based on a relationship to the physical markers; generating, by the surgical hub, a virtual element corresponding to the target location; and displaying, by an augmented reality (AR) device coupled to the surgical hub, the virtual element overlaid on the video feed on an AR display.

The various aspects described herein, both as to construction and methods of operation, together with further objects and advantages thereof, may be best understood by reference to the following description in conjunction with the accompanying drawings, in which: FIG.
1 is a block diagram of a computer-implemented interactive surgical system according to one aspect of the present disclosure. FIG. 1 is an illustration of a surgical system used to perform a surgical procedure within an operating room, according to one aspect of the present disclosure; FIG. 1 is a surgical hub paired with a visualization system, a robotic system, and an intelligent instrument, according to one aspect of the present disclosure. Connecting a modular device located in one or more surgical sites of a medical facility, or any room within a medical facility with specialized equipment for surgical procedures, to the cloud according to an aspect of the present disclosure. FIG. 1 illustrates a surgical data network including configured modular communication hubs. 1 illustrates a computer-implemented interactive surgical system according to one aspect of the present disclosure; FIG. FIG. 3 illustrates a surgical hub including multiple modules coupled to a modular control tower, according to one aspect of the present disclosure. FIG. 2 illustrates an augmented reality (AR) system including an intermediate signal combiner disposed in a communication path between an imaging module and a surgical hub display, according to an aspect of the present disclosure. FIG. 2 illustrates an augmented reality (AR) system including an intermediate signal combiner disposed in a communication path between an imaging module and a surgical hub display, according to an aspect of the present disclosure. FIG. 3 illustrates an augmented reality (AR) device worn by a surgeon to communicate data to a surgical hub, according to one aspect of the present disclosure. 1 illustrates a system for augmenting surgical instrument information using an augmented reality display, according to one aspect of the present disclosure. FIG. FIG. 3 is a diagram illustrating a timeline of a context-aware surgical procedure, according to one aspect of the present disclosure. FIG. 3 illustrates a structured surface with a plurality of fiducial markers, according to an aspect of the present disclosure. FIG. 3 illustrates a process for surface aligning external structures of a patient with fiducial markers, according to one aspect of the present disclosure. FIG. 3 illustrates a process for surface matching internal structures of a patient with fiducial markers, according to one aspect of the present disclosure. FIG. 3 illustrates a stereotactic frame external surgical alignment instrument for assisting a surgeon in a surgical procedure, according to one aspect of the present disclosure. FIG. 3 illustrates a starfix platform external surgical alignment instrument for assisting a surgeon in a surgical procedure, according to one aspect of the present disclosure. FIG. 3 illustrates a microtable external surgical alignment instrument for assisting a surgeon in a surgical procedure, according to one aspect of the present disclosure. FIG. 3 is a flow diagram for identifying an object based on multiple alignment parameters, according to an aspect of the present disclosure. FIG. 3 is a flow diagram for classifying unknown surgical instruments based on partial information of known and unknown parameters, according to one aspect of the present disclosure. FIG. 3 illustrates a trocar with an internal camera system, according to one aspect of the present disclosure. FIG. 12 illustrates a reusable attachment tool inserted into the proximal end of a trocar and configured to deploy and retract the camera system around the outer diameter of the trocar, according to one aspect of the present disclosure. FIG. 3 is a diagram illustrating a plurality of fiducial markers tagged to a region of interest in a preoperative computed tomography (CT) scan, according to one aspect of the present disclosure. FIG. 3 illustrates a laparoscopic surgical procedure that utilizes multiple fiducial markers to assist a surgeon in locating a surgical site, in accordance with one aspect of the present disclosure. FIG. 3 illustrates a physical marker applied by injection into a patient's vasculature along with an indocyanine dye, according to one aspect of the present disclosure. FIG. 12 further illustrates exemplary tissue injected with dye and illuminated to show vasculature, according to one aspect of the present disclosure. FIG. 3 illustrates a system configured to monitor pressure or fluid changes within a body cavity according to impedance measurements by a probe, according to one aspect of the present disclosure. 1 illustrates an infrared (IR) thermal detection system comprising an IR camera system configured to direct IR light onto a treatment area of tissue and identify temperature differences within a surgical environment, according to an aspect of the present disclosure; It is a diagram. FIG. 3 illustrates a surgical procedure employing three end effectors configured to grasp and transect tissue in accordance with one aspect of the present disclosure. FIG. 6 illustrates a third end effector sliding along tissue from a first position to a second position, according to an aspect of the present disclosure. FIG. 6 illustrates a third end effector positioned adjacent to a second end effector, according to one aspect of the present disclosure. FIG. 3 illustrates a surgical procedure with three static clamps and a dynamic clamp configured to move tissue between the static clamps, according to one aspect of the present disclosure. FIG. 3 is a logical flow diagram of a process for displaying a surgical location within a patient, according to one aspect of the present disclosure.

Corresponding reference characters indicate corresponding parts throughout the figures. The examples set forth herein are indicative of various disclosed embodiments, and such illustrations should not be construed as limiting the scope in any way.

The applicant of this application owns the following concurrently filed US patent applications, each of which is incorporated herein by reference in its entirety.
- United States patent application entitled "METHOD FOR INTRAOPERATIVE DISPLAY FOR SURGICAL SYSTEMS"; Attorney docket number END9352USNP1/210120-1M;
・U.S. patent application entitled “Utilization of surgical data values and situational awareness to control the overlay in surgical field view”; Attorney docket number E ND9352USNP2/210120-2,
- U.S. patent application entitled "SELECTIVE AND ADJUSTABLE MIXED REALITY OVERLAY IN SURGICAL FIELD VIEW"; Attorney docket number END9352USNP3/210120-3;
- U.S. patent application entitled "RISK BASED PRIORITIZATION OF DISPLAY ASPECTS IN SURGICAL FIELD VIEW"; Attorney docket number END9352USNP4/210120-4;
- United States patent application entitled "SYSTEMS AND METHODS FOR CONTROLLING SURGICAL DATA OVERLAY"; Attorney docket number END9352USNP5/210120-5;
・United States patent application entitled "SYSTEMS AND METHODS FOR CHANGING DISPLAY OVERLAY OF SURGICAL FIELD VIEW BASED ON TRIGGERING EVENTS"; Attorney docket number END9352USNP6 /210120-6,
- United States patent application entitled "CUSTOMIZATION OF OVERLAID DATA AND CONFIGURATION"; Attorney docket number END9352USNP7/210120-7;
- U.S. patent application entitled "INDICATION OF THE COUPLE PAIR OF REMOTE CONTROLS WITH REMOTE DEVICES FUNCTIONS"; attorney docket number END9352USNP8/210120-8;
・U.S. patent application entitled "COOPERATIVE OVERLAYS OF INTERACTING INSTRUMENTS WHICH RESULT IN BOTH OVERLAYS BEING EFFECTED"; Attorney docket number END9352USNP9/21012 0-9,
- United States patent application entitled "ANTICIPATION OF INTERACTIVE UTILIZATION OF COMMON DATA OVERLAYS BY DIFFERENT USERS"; Attorney docket number END9352USNP10/210120-10;
・“MIXING DIRECTLY VISUALIZED WITH RENDERED ELEMENTS TO DISPLAY BLENDED ELEMENTS AND ACTIONS HAPPENING ON-SCREEN AND OFF-S CREEN”; Attorney Docket No. END9352USNP11/210120-11;
・United States patent application entitled "SYSTEM AND METHOD FOR TRACKING A PORTION OF THE USER AS A PROXY FOR NON-MONITORED INSTRUMENT"; Attorney docket number END9352USNP12/2101 20-12,
・“UTILIZING CONTEXTUAL PARAMETERS OF ONE OR MORE SURGICAL DEVICES TO PREDICT A FREQUENCY INTERVAL FOR DISPLAYING SURGICAL US Patent Application entitled ``INFORMATION''; Attorney Docket No. END9352USNP13/210120-13;
- United States patent application entitled "INTRAOPERATIVE DISPLAY FOR SURGICAL SYSTEMS"; Attorney docket number END9352USNP15/210120-15;
・ US patent application entitled "ADAPTATATION ADJUSTABILITY or OVERLAID Instrument Instrument Informention for Surgical Systems" / 210120-16, and
・U.S. Patent Application entitled "MIXED REALITY FEEDBACK SYSTEMS THAT COOPERATE TO INCREASE EFFICIENT PERCEPTION OF COMPLEX DATA FEEDS"; Attorney docket number END9352USN P17/210120-17.

The applicant of this application owns the following US patent applications, each of which is incorporated by reference in its entirety.
・“METHOD OF COMPRESSING TISSUE WITHIN A STAPLING DEVICE AND SIMULTANEOUSLY DISPLAYING THE LOCATION OF THE TISSUE WITHIN THE U.S. Patent Application No. 16/209,423 entitled “JAWS” (now U.S. Patent Application Publication No. 2019/0200981-A1) ,
- U.S. Patent Application No. 16/209,453 entitled "METHOD FOR CONTROLLING SMART ENERGY DEVICES" (now U.S. Patent Application Publication No. 2019/0201046-A1).

Before describing in detail various aspects of surgical devices and generators, the illustrative examples are limited to details of the construction and arrangement of parts, in applications or applications, that are illustrated in the accompanying drawings and description. Please note that this is not the case. The example embodiments may be implemented or incorporated into other aspects, variations, and modifications and may be practiced or carried out in various ways. Furthermore, unless otherwise stated, the terms and expressions used herein have been chosen for the convenience of the reader to describe the illustrative embodiments and are not intended to be limiting. Additionally, one or more of the aspects, expressions of aspects, and/or examples described below may be combined with any one or more of the other aspects, expressions of aspects, and/or examples described below. It should be understood that it can be combined with

Various aspects are directed to on-screen displays for surgical systems for various energy and surgical stapler-based medical devices. Energy-based medical devices include, among others, but not limited to, radio frequency (RF)-based monopolar and bipolar electrosurgical instruments, ultrasound surgical instruments, combinations of RF electrosurgical instruments and ultrasound instruments, RF Includes a combination of an electrosurgical stapler and a mechanical stapler. Surgical stapler devices include surgical staplers combined with electrosurgical devices and/or ultrasound devices. Aspects of the ultrasonic surgical device may be configured, for example, to transect and/or coagulate tissue during a surgical procedure. Aspects of the electrosurgical device may be configured, for example, to transect, coagulate, seal, weld, and/or dry tissue during a surgical procedure. Embodiments of the surgical stapler device can be configured to transect and staple tissue during a surgical procedure, and in some embodiments, the surgical stapler device applies RF to tissue during a surgical procedure. Can be configured to deliver energy. Electrosurgical devices are configured to deliver therapeutic and/or non-therapeutic RF energy to tissue. The elements of a surgical stapler, electrosurgical device, and ultrasound device may be used in combination in a single surgical instrument.

In various aspects, the present disclosure provides an on-screen display of real-time information to the OR team during a surgical procedure. In accordance with various aspects of the present disclosure, a number of new unique on-screen displays are provided for displaying various visual information feedback to the OR team on-screen. According to this disclosure, visual information may include one or more of various visual media with or without sound. Generally, visual information includes still photographs, moving photographs, video or audio recordings, graphic art, visual aids, models, displays, visual representation services, and support processes. Visual information can be communicated on any number of display options, such as, for example, the primary OR screen, the energy or surgical stapler device itself, a tablet, augmented reality glasses, among others.

In various aspects, this disclosure provides a long list of potential options for communicating visual information to the OR team in real time without overwhelming the OR team with too much visual information. For example, in various aspects, the present disclosure enables a surgeon, or other member of the OR team, to selectively activate on-screen displays, such as icons surrounding screen options, to manage rich visual information. Provides an on-screen display of visual information that enables. One or a combination of factors can be used to determine the active display, including the energy-based (e.g., electrosurgical, ultrasound) or mechanical-based (e.g., stapler) in use. It may include the surgical device, the estimated risk associated with a given indication, the surgeon's experience level, and the surgeon's selection. In other aspects, the visual information may include rich data overlaid or superimposed on the surgical field to manage the visual information. Various aspects described below involve overlapping images that require video analysis and tracking to properly overlay data. Visual information data communicated in this manner, as opposed to static icons, can provide additional useful visual information to the OR team in a more concise and understandable manner.

In various aspects, the present disclosure provides techniques for selectively activating on-screen displays, such as icons surrounding the screen, to manage visual information during a surgical procedure. In other aspects, the present disclosure provides techniques for determining active display using one or a combination of factors. In various aspects, techniques according to the present disclosure provide, among other things, selecting an energy-based or mechanical-based surgical device in use as an active display; estimating the risk associated with a given display; This may include taking advantage of the experience level of the surgeon or OR team.

In other aspects, techniques according to the present disclosure may include overlaying or superimposing rich data onto the surgical field to manage visual information. Some display arrangements described by this disclosure involve overlaying various visual representations of surgical data onto a live stream of the surgical field. As used herein, the term overlay includes translucent overlays, partial overlays, and/or moving overlays. Graphical overlays may be in the form of transparent, translucent, or opaque graphics, or a combination of transparent, translucent, and opaque elements or effects. Additionally, the overlay may be placed on or at least partially on or near objects in the surgical field, such as, for example, end effectors and/or critical surgical structures. The particular display arrangement may include changes in the color, size, shape, display time, display location, display frequency, highlighting, or combinations thereof of one or more of the overlays based on the change in the display priority value. May include changes in elements. A graphical overlay is rendered on top of the active display monitor and conveys critical information to the OR team quickly and efficiently.

In other aspects, techniques according to this disclosure may include overlaying images that require video analysis and tracking to properly overlay visual information data. In other aspects, techniques according to the present disclosure communicate rich visual information, as opposed to simple static icons, to provide additional visual information to the OR team in a more concise and easy-to-understand manner. can include. In other aspects, visual overlays can be used in combination with auditory and/or somatosensory overlays, such as thermal, chemical, and mechanical devices, and combinations thereof.

The following description generally relates to devices, systems, and methods for providing augmented reality (AR) interactive experiences during surgical procedures. In this context, images of the surgical field and of surgical instruments and other objects that appear in the surgical field can be used to convey computer-generated visual, auditory, tactile, somatosensory, olfactory, or other sensory information to the surgical field. by overlaying onto real-world images of instruments and/or other objects that appear in the image. Images may be streamed in real time or may be static images. Augmented reality is a technology for rendering and displaying virtual or "augmented" virtual objects, data, or visual effects that are overlaid onto a real environment. The real environment may include a surgical field. Virtual objects overlaid on the real environment may be represented in a fixed or set position relative to one or more aspects of the real environment. In a non-limiting example, when a real-world object exits the field of view of the real environment, a virtual object fixed to the real-world object also exits the field of view of the augmented reality.

Some display arrangements described by this disclosure involve overlaying various visual representations of surgical data onto a live stream of the surgical field. As used herein, the term overlay includes translucent overlays, partial overlays, and/or moving overlays. Additionally, the overlay may be placed on or at least partially on or near objects in the surgical field, such as, for example, end effectors and/or critical surgical structures. The particular display arrangement may include changes in the color, size, shape, display time, display location, display frequency, highlighting, or combinations thereof of one or more of the overlays based on the change in the display priority value. May include changes in elements.

As described herein, AR is an augmented version of the real physical world achieved through the use of digital visual elements, sound, or other sensory stimuli delivered via technology. Virtual reality (VR) is a computer-generated environment with scenes and objects that appear to be real, making the user feel immersed in their surroundings. This environment is perceived through a device known as a virtual reality headset or helmet. Although mixed reality (MR) and AR are both considered immersive technologies, they are not the same. MR is an extension of mixed reality that allows real and virtual elements to interact within an environment. While AR adds digital elements to the live view, often by using cameras, MR experiences combine elements of both AR and VR, where real-world and digital objects interact.

In an AR environment, one or more computer-generated virtual objects may be displayed together with one or more real-world (ie, so-called "real world") elements. For example, real-time images or video of the surrounding environment may be shown on a computer screen display with one or more overlaying virtual objects. Such virtual objects may provide supplementary information about the environment or generally enhance the user's perception and engagement with the environment. Conversely, real-time images or videos of the surrounding environment may additionally or alternatively enhance the user's engagement with the virtual objects shown on the display.

Apparatus, systems, and methods in the context of the present disclosure enhance images received from one or more imaging devices during a surgical procedure. Imaging devices may include various scopes used during non-invasive and minimally invasive surgical procedures, AR devices, and/or cameras that provide images during open surgical procedures. Images may be streamed in real time or may be static images. Apparatus, systems, and methods provide an augmented reality interactive experience by enhancing the image of a real-world surgical environment by overlaying virtual objects or data and/or representations of real objects onto the real-world surgical environment. do. The augmented reality experience may be viewed on a display and/or AR device that allows a user to view virtual objects overlaid on a real-world surgical environment. The display may be located within the operating room or may be remote from the operating room. AR devices are worn on the head of a surgeon or other operating room personnel and typically include two stereoscopic display lenses or screens, including one for each eye of the user. Natural light can pass through two transparent or translucent display lenses so that aspects of the real environment are visible while projecting light to make virtual objects visible to a user of the AR device.

Two or more displays and AR devices may be used in conjunction with, for example, a first display or AR device controlling one or more additional displays or AR devices in the system with a defined role. . For example, when activating a display or AR device, a user may select a role (e.g., surgeon during a surgical procedure, surgical assistant, nurse, etc.) and the display or AR device may select a role associated with that role. Information may be displayed. For example, a surgical assistant may display a virtual representation of instruments that the surgeon needs to perform for the next step in a surgical procedure. The surgeon's focus on the current process may see displayed information differently than the surgical assistant.

While there are many known on-screen displays and alerts, the present disclosure provides many new and unique augmented reality interactive experiences during surgical procedures. Such augmented reality interactive experiences include visual, auditory, tactile, somatosensory, olfactory, or other sensory feedback information to the surgical team inside or outside the operating room. Virtual feedback information overlaid on the real-world surgical environment can, for example, improve personnel within the OR, including, but not limited to, operating surgeons, surgeon's assistants, scrub wearers, anesthesiologists, and outgoing nurses, among others. may be provided to the operating room (OR) team, including: Virtual feedback information can be communicated on any number of display options, such as primary OR screen displays, AR devices, energy or surgical stapler instruments, tablets, augmented reality glasses, devices, etc.

FIG. 1 shows a computer-implemented interactive surgical system 1 that includes one or more surgical systems 2 and a cloud-based system 4 . Cloud-based system 4 may include a remote server 13 coupled to remote storage 5 . Each surgical system 2 comprises at least one surgical hub 6 in communication with the cloud 4. For example, surgical system 2 may include visualization system 8 , robotic system 10 , and hand-held intelligent surgical instrument 12 , each configured to communicate with each other and/or with hub 6 . There is. In some aspects, surgical system 2 may include M hubs 6 , N visualization systems 8 , O robotic systems 10 , and P hand-held intelligent surgical instruments 12 . Often M, N, O and P are integers greater than or equal to 1. Computer-implemented interactive surgical system 1 may be configured to provide an augmented reality interactive experience during a surgical procedure, as described herein.

FIG. 2 shows an example of a surgical system 2 for performing a surgical procedure on a patient lying on an operating table 14 in a surgical operating room 16. Robotic system 10 is used as part of surgical system 2 in a surgical procedure. Robotic system 10 includes a surgeon's console 18 , a patient cart 20 (surgical robot), and a surgical robot hub 22 . The patient-side cart 20 is removably coupled to at least one patient-side cart 20 through a minimally invasive incision in the patient's body while the surgeon views the surgical site through the surgeon's console 18 or an augmented reality (AR) device 66 worn by the surgeon. surgical tool 17 can be operated. Images of the surgical site during a minimally invasive procedure (eg, still images or live images streamed in real time) may be acquired by medical imaging device 24. Patient-side cart 20 can operate imaging device 24 to orient imaging device 24 . Images of the open surgical procedure may be acquired by medical imaging device 96. The robotic hub 22 processes images of the surgical site for subsequent display on the surgeon's console 18 or an AR device 66 worn by the surgeon or other person within the surgical operating room 16 .

The optical components of the imaging device 24, 96 or AR device 66 may include one or more illumination sources and/or one or more lenses. The one or more illumination sources may be directed to illuminate a portion of the surgical field. The one or more image sensors may receive light reflected or refracted from tissue and instruments in the surgical field.

In various aspects, imaging device 24 is configured for use in minimally invasive surgical procedures. Examples of imaging devices suitable for use with the present disclosure include arthroscopes, angioscopes, bronchoscopes, cholangioscopes, colonoscopes, cystoscopes, duodenoscopes, enteroscopes, esophagogastroduodenoscopes, endoscopic These include, but are not limited to, endoscopes, laryngoscopes, nasopharyngeal-pyeloscopes, sigmoidoscopes, thoracoscopes, and ureteroscopes. In various aspects, imaging device 96 is configured for use in open (invasive) surgical procedures.

In various aspects, visualization system 8 includes one or more imaging sensors strategically positioned relative to the sterile field, one or more image processing devices, one or more storage arrays, and one or more storage arrays. Equipped with a display. In one aspect, visualization system 8 includes interfaces for HL7, PACS, and EMR. In one aspect, imaging device 24 may employ multispectral monitoring to distinguish between topography and underlying structure. Multispectral images capture image data within specific wavelength ranges in the electromagnetic spectrum. The wavelengths are separated by filters or by instruments with sensitivity to specific wavelengths, including light from frequencies beyond the visible light range, such as the IR and ultraviolet. Spectral imaging can extract information invisible to the human eye. Multispectral monitoring can reposition the surgical field after the surgical task is completed to perform tests on the treated tissue.

FIG. 2 shows the primary display 19 placed in the sterile field so as to be visible to the operator at the operating table 14. The visualization tower 11 includes a first non-sterile display 7 and a second non-sterile display 9 located outside the sterile field and facing oppositely from each other. A visualization system 8 guided by the hub 6 is configured to utilize the displays 7, 9, 19 to coordinate the flow of information to operators inside and outside the sterile field. For example, the hub 6 may display an AR image of the surgical site recorded by the imaging device 24 , 96 on the visualization system 8 , while maintaining live footage of the surgical site on the primary display 19 or the AR device 66 , on a non-sterile display 7 , 9 or an AR device 66. The non-sterile displays 7, 9 may, for example, enable a non-sterile operator to perform diagnostic steps associated with a surgical procedure.

FIG. 3 shows a hub 6 in communication with a visualization system 8, a robotic system 10, and a hand-held intelligent surgical instrument 12. Hub 6 includes a hub display 35 , an imaging module 38 , a generator module 40 , a communication module 30 , a processor module 32 , a storage array 34 , and an operating room mapping module 33 . Hub 6 further includes a smoke evacuation module 26 and/or a suction/irrigation module 28. In various aspects, imaging module 38 includes an AR device 66 and processor module 32 includes an integrated video processor and augmented reality modeler (eg, as shown in FIG. 10). Modular light sources can be adapted for use with a variety of imaging devices. In various examples, multiple imaging devices can be placed at different locations in the surgical field to provide multiple views (eg, non-invasive, minimally invasive, invasive, or open surgical procedures). Imaging module 38 may be configured to switch between imaging devices to provide an optimal view. In various aspects, imaging module 38 can be configured to integrate images from different imaging devices and provide an augmented reality interactive experience during a surgical procedure as described herein.

FIG. 4 shows a surgical data network 51 that includes a modular communications hub 53 configured to connect modular devices located in one or more surgical sites/rooms of a medical facility to a cloud-based system. Cloud 54 may include a remote server 63 (FIG. 5) coupled to storage device 55. Modular communications hub 53 includes a network hub 57 and/or network switch 59 that communicates with network router 61 . Modular communication hub 53 is coupled to local computer system 60 for processing data. Modular devices 1a-1n at the surgical site may be coupled to a modular communication hub 53. Network hub 57 and/or network switch 59 may be coupled to network router 61 to connect devices 1a-1n to cloud 54 or local computer system 60. Data associated with devices 1a-1n may be transferred via a router to a cloud-based computer for remote data processing and manipulation. Surgical site devices 1a-1n may be connected to modular communication hub 53 via wired or wireless channels. Surgical data network 51 environment specifically provides augmented images of the surgical field to one or more remote displays 58 to provide an augmented reality interactive experience during a surgical procedure, as described herein. may be employed to do so.

Figure 5 illustrates a computer-implemented interactive surgical system 50. The computer-implemented interactive surgical system 50 is similar in many respects to the computer-implemented interactive surgical system 1. The computer-implemented interactive surgical system 50 includes one or more surgical systems 52 that are similar in many respects to the surgical system 2. Each surgical system 52 includes at least one surgical hub 56 that communicates with a cloud 54 that may include a remote server 63. In one aspect, the computer-implemented interactive surgical system 50 includes a modular control tower 23 connected to multiple surgical field devices, such as, for example, intelligent surgical instruments, robots, and other computerized devices located within the surgical field. As shown in Figure 6, the modular control tower 23 includes a modular communication hub 53 coupled to a computer system 60.

Returning to FIG. 5, the modular control tower 23 includes an imaging module 38 coupled to an endoscope 98, a generator module 27 coupled to an energy device 99, a smoke evacuator module 76, an aspiration/irrigation module 78, and a communication The module 13 is coupled to a processor module 15, a storage array 16, a smart device/appliance 21 optionally coupled to a display 39, and a sensor module 29. The surgical site devices are coupled to cloud computing resources such as servers 63, data storage 55, and displays 58 via modular control tower 23. Robotic hub 72 may also be connected to modular control tower 23 and servers 63, data storage 55, and displays 58. Among other things, devices/appliances 21, visualization systems 58 may be coupled to modular control tower 23 via wired or wireless communication standards or protocols, as described herein. Modular control tower 23 is configured to display the received augmented images, including overlaid virtual objects on the real surgical world, received from imaging module 38, device/instrument display 39, and/or other visualization system 58. The hub display 65 (eg, monitor, screen) may also be coupled to the hub display 65 (eg, monitor, screen). Hub display 65 may also display data received from devices connected to modular control tower 23 along with images and overlay images.

FIG. 6 shows a surgical hub 56 that includes multiple modules coupled to a modular control tower 23. As shown in FIG. Modular control tower 23 includes a modular communication hub 53, such as a network connection device, and a computer system 60, for example, for local processing, visualization, and imaging of advanced surgical information. Modular communications hubs 53 are connected in a layered configuration to expand the number of modules (e.g., devices) that may be connected to modular communications hubs 53 to transmit data associated with the modules to computer systems 60, It may be transferred to a cloud computing resource, or both. Each of the network hubs/switches 57/59 within modular communication hub 53 may include three downstream ports and one upstream port. Upstream network hubs/switches 57 , 59 are connected to processor 31 to provide communication connectivity to cloud computing resources and local display 67 . Communication to cloud 54 can occur via either wired or wireless communication channels.

Computer system 60 includes a processor 31 and a network interface 37. Processor 31 is coupled to communication module 41, storage 45, memory 46, non-volatile memory 47, and input/output interface 48 via a system bus. The system bus may be any of several types of bus structures, including a memory bus or memory controller, a peripheral or external bus, and/or a local bus using a variety of available bus architectures. .

Processor 31 includes an augmented reality modeler (eg, as shown in FIG. 10) and may be implemented as a single-core or multi-core processor, such as that known under the trade name ARM Cortex by Texas Instruments. In one aspect, the processor may be, for example, an LM4F230H5QR ARM Cortex-M4F processor core available from Texas Instruments. The processor core includes on-chip memory of 256 KB of single-cycle flash memory or other non-volatile memory up to 40 MHz, a prefetch buffer to improve performance above 40 MHz, 32 KB of single-cycle serial random access memory (SRAM), and StellarisWare Internal Read Only Memory (ROM) with ® software, 2 KB of Electrically Erasable Programmable Read Only Memory (EEPROM) and/or one or more Pulse Width Modulation (PWM) modules, one or more quadrature Encoder input (QEI) analog, including one or more 12-bit analog-to-digital converters (ADC) with 12 analog input channels. Further details are available in the product data sheet.

System memory includes volatile memory and non-volatile memory. The basic input/output system (BIOS), containing the basic routines for transferring information between elements within a computer system, such as during startup, is stored in non-volatile memory. For example, non-volatile memory may include ROM, programmable ROM (PROM), electrically programmable ROM (EPROM), EEPROM, or flash memory. Volatile memory includes random access memory (RAM), which functions as external cache memory. Furthermore, RAM can be divided into various types such as SRAM, dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), sink link DRAM (SLDRAM), and direct RAM bus RAM (DRRAM). Available in many forms.

Computer system 60 also includes removable/non-removable, volatile/non-volatile computer storage media, such as disk storage. Disk storage devices include, but are not limited to, devices such as magnetic disk drives, floppy disk drives, tape drives, Jaz drives, Zip drives, LS-60 drives, flash memory cards, or memory sticks. Additionally, a disk storage device can include any of the storage media described above, either independently or in combination with other storage media. Other storage media include compact disc ROM devices (CD-ROMs), compact disc recordable drives (CD-R drives), compact disc rewritable drives (CD-RW drives), or digital versatile disc ROM drives (DVD-ROM drives). Examples include, but are not limited to, optical disk drives such as ROM). Removable or non-removable interfaces may be used to facilitate connection of disk storage devices to the system bus.

In various aspects, computer system 60 of FIG. 6, imaging module 38 and/or visualization system 58 of FIGS. 4-6, and/or processor module 15 may include an image processor, image processing engine, image processing unit (GPU). , a media processor, or any dedicated digital signal processor (DSP) used to process digital images. Image processors can use parallel computing using single instruction multiple data (SIMD) or multiple instruction multiple data (MIMD) techniques to increase speed and efficiency. Digital image processing engines can perform a variety of tasks. The image processor may be a system on a chip with a multi-core processor architecture.

FIG. 7 shows an augmented reality system 263 that includes an intermediate signal combiner 64 disposed in the communication path between the imaging module 38 and the surgical hub display 67. Signal combiner 64 combines audio and/or image data received from imaging module 38 and/or AR device 66 . Surgical hub 56 receives the combined data from combiner 64 and overlays the data provided to display 67, where the overlaid data is displayed. Imaging device 68 may be a digital video camera and audio device 69 may be a microphone. Signal combiner 64 is a wireless head-up display adapter for coupling to an AR device 66 located within the communication path of display 67 to the console that allows surgical hub 56 to overlay data onto display 67. may include.

FIG. 8 shows an augmented reality (AR) system that includes an intermediate signal combiner positioned in a communication path between an imaging module and a surgical hub display. FIG. 8 shows an AR device 66 worn by a surgeon 73 to communicate data to the surgical hub 56. The peripheral information of the AR device 66 does not include active video. Rather, the peripheral information includes only signals that do not have the same requirements for device settings or refresh rates. The interaction may enhance the surgeon's 73 information based on linking with preoperative computed tomography (CT) or other data linked within the surgical hub 56. The AR device 66 can identify structures, for example, can ask if the instrument is touching a nerve, blood vessel, or adhesion. AR device 66 may include preoperative scan data, optical views, tissue investigation characteristics acquired throughout the procedure, and/or processing at surgical hub 56 that is used to provide answers. Surgeon 73 may write notes on AR device 66 to be saved with patient data in hub storage 45 for later use in reporting or follow-up.

An AR device 66 worn by the surgeon 73 links to the surgical hub 56 with audio and visual information to avoid the need for overlays and allows customization of information displayed around the periphery of the field of view. Make it. AR device 66 provides signals from devices (eg, fixtures) and answers queries regarding device settings or location information linked with video to identify quadrants or locations. AR device 66 has audio control and audio feedback from AR device 66 . The AR device 66 can interact with other systems within the surgical site and can have feedback and interaction available wherever the surgeon 73 looks. For example, AR device 66 may receive audio or gesture initiation commands and queries from a surgeon, and AR device 66 may provide feedback in the form of one or more modalities including audio, visual, or tactile touch. Good too.

FIG. 9 shows a surgeon 73 and a patient 74 wearing an AR device 66 and may include a camera 96 within the operating room 75. AR device 66 worn by surgeon 73 may be used to present virtual objects to surgeon 73 that are overlaid on real-time images of the surgical field, either through augmented reality display 89 or through hub-connected display 67. The real-time image may include a portion of the surgical instrument 77. The virtual object may not be visible to others in the operating room 75 (eg, surgical assistants or nurses), although they may also be wearing the AR device 66. Even if another person is viewing the operating room 75 using an AR device 66, that person may not be able to see the virtual objects or in the augmented reality shared with the surgeon 73. It may be possible to view the virtual object, or it may be possible to view a modified version of the virtual object (eg, according to customization specific to the surgeon 73), or it may be possible to view a different virtual object.

The virtual objects and/or data may be captured on a portion of the surgical instrument 77 or by the imaging module 38, the imaging device 68 during a minimally invasive surgical procedure, and/or the camera 96 during an open surgical procedure. It may be configured to appear within the field of view. In the illustrated example, imaging module 38 is a laparoscopic camera that provides live video of the surgical field during a minimally invasive surgical procedure. The AR system may present virtual objects that are fixed to real objects regardless of the perspective of one or more viewers of the AR system (eg, surgeon 73). For example, the virtual object may be visible to a viewer of the AR system within the operating room 75 and may not be visible to a viewer of the AR system outside the operating room 75. The virtual object may be displayed to a viewer outside the operating room 75 when the viewer enters the operating room 75. The augmented image may be displayed on surgical hub display 67 or augmented reality display 89.

AR device 66 may include one or more screens or lenses, such as a single screen or two screens (eg, one for each eye of the user). The screen may allow light to pass through the screen so that aspects of the real environment are visible while displaying the virtual object. The virtual object may become visible to the surgeon 73 by projecting light. Virtual objects may appear to have some degree of transparency, or may be opaque (ie, occlude aspects of the real environment).

The AR system may be visible to one or more viewers and may include differences between views that are available to one or more viewers while retaining some aspects common between the views. . For example, a heads-up display may change between two views, but virtual objects and/or data may be fixed to real objects or regions in both views. Aspects such as object color, lighting, or other changes may be made between views without changing the fixed position of the at least one virtual object.

A user may view virtual objects and/or data presented within an AR system as opaque or as containing some level of transparency. In one example, a user can interact with a virtual object, such as by moving the virtual object from a first position to a second position. For example, the user may move the object with his/her hand. This may include one or more cameras, which may be mounted on the AR device 66 (such as an AR device camera 79 or a separate 96), which may be static, or may be controlled to move. This may be done virtually in an AR system by determining (using a camera) that the hand has moved into a position coincident with or adjacent to an object and moving the object accordingly. Virtual aspects may include virtual representations of real-world objects or may include visual effects such as lighting effects. AR systems may include rules to govern the behavior of virtual objects, such as subjecting them to gravity or friction, or otherwise negating real-world physical constraints (e.g., floating objects, perpetual motion, etc.) may include predefined rules. AR device 66 may include a camera 79 on AR device 66 (not to be confused with camera 96, which is separate from AR device 66). The AR device camera 79 or camera 96 may include an infrared camera, an infrared filter, a visible light filter, multiple cameras, a depth camera, and the like. AR device 66 may project virtual items onto a representation of the real environment that can be viewed by a user.

AR device 66 may be used, for example, within operating room 75 during a surgical procedure performed on patient 74 by surgeon 73. AR device 66 may project or display virtual objects, such as virtual objects during a surgical procedure, to augment the surgeon's vision. The surgeon 73 may view the virtual object using the AR device 66 , a remote controller for the AR device 66 , or “interact” with the virtual object or gesture recognized by the camera 79 of the AR device 66 , for example. The hands may be used to interact with virtual objects. The virtual object can extend a surgical tool, such as surgical instrument 77. For example, the virtual object may appear to be coupled to or remain a fixed distance from the surgical instrument 77 (to the surgeon 73 viewing the virtual object through the AR device 66). good. In another example, virtual objects may be used to guide surgical instruments 77 and may appear fixed to patient 74. In certain examples, virtual objects may react to movement of other virtual or real-world objects in the surgical field. For example, a virtual object may be modified when a surgeon is manipulating a surgical instrument in close proximity to the virtual object.

Augmented reality display system imaging device 38 captures a real image of the surgical field during the surgical procedure. Augmented reality displays 89, 67 present an overlay of the operational aspects of surgical instrument 77 on the real image of the surgical field. Surgical instrument 77 includes communication circuitry 231 for communicating operational aspects and functional data from surgical instrument 77 to AR device 66 via communication circuitry 233 on AR device 66. Although surgical instrument 77 and AR device 66 are shown in RF wireless communication between circuits 231, 233 as indicated by arrows B, C, other communication techniques (e.g., wired, ultrasonic, infrared, etc.) may be employed. The overlay relates to the operational aspects of surgical instrument 77 that are actively visualized. The overlay combines aspects of tissue interaction in the surgical field with functional data from surgical instrument 77. The processor portion of the AR device 66 is configured to receive operational aspects and functional data from the surgical instrument 77, determine an overlay related to the operation of the surgical instrument 77, and combine aspects of tissue within the surgical field with the functional data from the surgical instrument 77. The augmented images show alerts for device performance considerations, incompatible use alerts, and incomplete capture alerts. Incompatible use includes out of range tissue conditions and tissue incorrectly balanced within the jaws of the end effector. Additional augmented images provide indications of contingencies, including an indication of tissue tension and an indication of foreign body detection. Other augmented images show device status overlays and instrument indications.

10 illustrates a system 83 for augmenting an image of a surgical field with information using an AR display 89, according to at least one aspect of the present disclosure. The system 83 may be used to perform the techniques described below, for example, by using a processor 85. The system 83 includes an aspect of an AR device 66 that can communicate with a database 93. The AR device 66 includes a processor 85, a memory 87, an AR display 89, and a camera 79. The AR device 66 may include a sensor 90, a speaker 91, and/or a haptic controller 92. The database 93 may include an image storage 94 or a preoperative planning storage 95.

Processor 85 of AR device 66 includes an augmented reality modeler 86 . Augmented reality modeler 86 may be used by processor 85 to create an augmented reality environment. For example, augmented reality modeler 86 may receive images of instruments within the surgical field, such as from camera 79 or sensor 90, and create an augmented reality environment to fit within the displayed image of the surgical field of view. In another example, physical objects and/or data may be overlaid on the surgical field of view and/or surgical instrument images, and augmented reality modeler 86 uses the physical objects and data to create virtual objects and /Or an augmented reality display of the data may be presented within an augmented reality environment. For example, augmented reality modeler 86 uses or detects instruments at a patient's surgical site and displays virtual objects and/or data on the surgical instruments and/or images of the surgical site within the surgical field of view captured by camera 79. May be presented. The AR display 89 may display an AR environment overlaid on the real environment. Display 89 may show virtual objects and/or data using AR device 66, such as at a fixed location within the AR environment.

AR device 66 may include a sensor 90, such as an infrared sensor. Camera 79 or sensor 90 may be used to detect movements, such as gestures by a surgeon or other user, which are determined by processor 85 to indicate an attempted or intended interaction by the user with a virtual target. may be interpreted as Processor 85 may identify objects in the real environment, such as by processing information received using camera 79. In other aspects, sensors 90 may be tactile, audible, chemical, or thermal sensors to generate corresponding signals that may be combined with various data feeds to create an augmented environment. Sensors 90 may include binaural audio sensors (spatial sounds), inertial measurement (accelerometers, gyroscopes, magnetometers) sensors, environmental sensors, depth camera sensors, hand and eye tracking sensors, and voice command recognition capabilities.

The AR display 89 may present virtual features corresponding to physical features obscured by the patient's anatomical features, such as within a surgical field, while allowing the surgical field to be viewed through the AR display 89 during a surgical procedure. The virtual features may have a virtual position or orientation corresponding to a first physical position or orientation of the physical features. In one example, the virtual position or orientation of the virtual features may include an offset from the first physical position or orientation of the physical features. The offset may include a predetermined distance from the augmented reality display, a relative distance from the augmented reality display to the anatomical features, etc.

In one example, AR device 66 may be an individual AR device. In one aspect, AR device 66 may be a HoloLens 2 AR device manufactured by Microsoft of Redmond, Washington. This AR device 66 includes a visor with lens and binaural audio features (spatial sounds), inertial measurements (accelerometer, gyroscope, magnetometer), environmental sensors, depth camera, video camera, hand and eye tracking. and a voice command recognition function. It provides improved vision with high resolution by using mirrors to direct the waveguide in front of the wearer's eyes. The image can be enlarged by changing the angle of the mirror. It also provides eye tracking to recognize users and adjust lens width for specific users.

In another example, AR device 66 may be a Snapchat Spectacles 3 AR device. This AR device provides the ability to capture paired images, recreate 3D depth mapping, add virtual effects, and play 3D videos. The AR device includes two HD cameras to capture 3D photos and videos at 60fps, while four built-in microphones record immersive high-fidelity audio. Images from both cameras are combined to build a geometric map of the real world around the user, providing a new sense of depth perception. Photos and videos can be wirelessly synced to an external display device.

In yet another example, AR device 66 may be a Glass 2 AR device by Google. This AR device provides inertial measurement (accelerometer, gyroscope, magnetometer) information overlaid on the lens (outside the field of view) to supplement the information.

In another example, AR device 66 may be an Echo Frames AR device by Amazon. This AR device has no camera/display. The microphone and speaker are linked to Alexa. This AR device has fewer features than a head-up display.

In yet another example, AR device 66 may be a Focals AR device by North (Google). This AR device offers notification pusher/smartwatch analog, inertial measurements, screen overlays of information (weather, calendar, messages), and voice control (Alexa) integration. This AR device provides basic head-up display functionality.

In another example, AR device 66 may be a Nreal AR device. This AR device includes spatial sound, two environmental cameras, a photo camera, an IMU (accelerometer, gyroscope), an ambient light sensor, and a proximity sensor function. nebula projects application information onto the lens.

In various other examples, the AR device 66 is any one of the following commercially available AR devices: Magic Leap 1, Epson Moverio, Vuzix Blade AR, ZenFone AR, Microsoft AR Glasses Prototype, EyeTap. This creates light that is collinear with the light of the environment directly on the retina. A beam splitter, for example, makes the same visible light available to a computer for processing and overlaying information. AR visualization systems include HUDs, contact lenses, glasses, virtual reality (VR) headsets, virtual retinal displays, operating room displays, and/or smart contact lenses (bionic lenses).

The multi-user interface for the AR device 66 includes virtual retinal displays such as raster displays that are drawn directly on the retina rather than on a screen in front of the eyes, spatial displays such as smart TVs, smartphones, and/or Sony Spatial Display Systems. including.

Other AR technologies may include, for example, AR capture devices and software applications, AR creation devices and software applications, and AR cloud devices and software applications. AR capture devices and software applications include, for example, Apple Polycam app, Ubiquity 6 (Mirrorworld using Display.land app), in which the user scans a 3D image of the real world (to create a 3D model); can be obtained. AR creation devices and software applications include, for example, Adobe Aero, Vuforia, ARToolKit, Google ARCore, Apple ARKit, MAXST, Aurasma, Zappar, Blippar. AR cloud devices and software applications include, for example, Facebook, Google (world geometry, object recognition, predictive data), Amazon AR Cloud (commerce), Microsoft A Including zure, Samsung Project Where, Niantic, and Magic Leap.

Situational awareness is the ability of some aspects of a surgical system to determine or infer information related to a surgical procedure from data received from a database and/or instruments. The information may include the type of procedure being performed, the type of tissue being operated on, or the body cavity being treated. According to the contextual information about the surgical procedure, the surgical system can, for example, control modular devices (e.g., robotic arms and/or robotic surgical tools) connected to it, and control the course of the surgical procedure. can be improved in a manner that provides contextualized information or suggestions to the surgeon.

FIG. 11 shows a timeline of situational awareness surgical procedures. FIG. 11 illustrates an exemplary surgical procedure timeline 5200 and contextual information that the surgical hub 5104 may derive from data received from the data source 5126 at each step of the surgical procedure. Timeline 5200 illustrates the general steps that nurses, surgeons, and other medical personnel would take during the course of a segmental lung resection procedure, beginning with surgical site setup and ending with transporting the patient to the postoperative recovery room. The process is shown below. Situational-aware surgical hub 5104 transmits data to data source 5126, including data generated each time a medical personnel uses a modular device 5102 paired with surgical hub 5104, throughout the course of a surgical procedure. Receive from. Surgical hub 5104 receives this data from paired modular devices 5102 and other data sources 5126 and receives new data, such as which steps of a procedure are being performed at any given time. Once performed, inferences (i.e., contextual information) regarding the ongoing procedure can be continuously derived. The situational awareness system of the surgical hub 5104 may, for example, record data regarding a procedure to generate reports, verify steps being taken by medical personnel, record data that may be related to a particular procedure step, or Providing a prompt (e.g., via a display screen), adjusting the modular device 5102 based on the context (e.g., activating a monitor, adjusting the FOV of a medical imaging device, or an ultrasonic surgical instrument) or changing the energy level of the RF electrosurgical instrument) and any other such operations described above.

First, at 5202, hospital personnel retrieves the patient's EMR from the hospital's EMR database. Based on the patient data selected in the EMR, surgical hub 5104 determines that the procedure being performed is thoracic surgery.

In the second 5204, personnel scan incoming medical supplies for treatment. Surgical hub 5104 cross-references the scanned supplies with a list of supplies utilized in various types of procedures to ensure that the mix of supplies corresponds to the thoracic procedure. Additionally, the surgical hub 5104 may also determine that the procedure is not a wedge resection (the incoming supplies do not include certain supplies required for a thoracic wedge resection, or the thoracic wedge is otherwise (because they are either not amenable to wedge resection).

Third 5206, the medical professional scans the patient band via a scanner 5128 communicatively connected to the surgical hub 5104. Surgical hub 5104 can then verify the patient's identity based on the scanned data.

Fourth, 5208, the medical personnel turns on the auxiliary device. The auxiliary equipment utilized may vary according to the type of surgical procedure and the technique used by the surgeon, but in this illustrative case includes a smoke evacuator, an insufflator, and a medical imaging device. Once activated, the ancillary device that is the modular device 5102 can automatically pair with a surgical hub 5104 located within a particular proximity of the modular device 5102 as part of its initialization process. . Surgical hub 5104 can then derive contextual information regarding the surgical procedure by detecting the type of modular device 5102 with which it is paired during this preoperative or initialization phase. In this particular example, surgical hub 5104 determines that the surgical procedure is a VATS procedure based on this particular combination of paired modular devices 5102. Based on a combination of data from the patient's EMR, the list of medical supplies used in the procedure, and the type of modular device 5102 that connects to the hub, the surgical hub 5104 generally estimates the specific procedure that the surgical team will perform. can do. Once the surgical hub 5104 knows what particular procedure is being performed, the surgical hub 5104 then reads the steps for that procedure from memory or from the cloud and then reads the steps of that procedure from the connected data source 5126 (e.g. Data subsequently received from the modular device 5102 and the patient monitoring device 5124) can be cross-referenced to estimate which steps of the surgical procedure the surgical team is performing.

In a fifth step 5210, personnel attach EKG electrodes and other patient monitoring devices 5124 to the patient. The EKG electrodes and other patient monitoring devices 5124 can be paired with the surgical hub 5104. Once the surgical hub 5104 begins receiving data from the patient monitoring devices 5124, the surgical hub 5104 confirms that the patient is at the surgical site.

In a sixth step 5212, medical personnel induce anesthesia in the patient. The surgical hub 5104 can infer that the patient is under anesthesia based on data from the modular device 5102 and/or the patient monitoring device 5124, including, for example, EKG data, blood pressure data, ventilator data, or a combination thereof. Once the sixth step 5212 is completed, the pre-operative portion of the lung segmentectomy surgery is complete and the surgical portion begins.

Seventh 5214, the patient's lung being operated on is collapsed (while ventilation is switched to the contralateral lung). The surgical hub 5104 can infer from the ventilator data that the patient's lung has collapsed. The surgical hub 5104 can compare the detection of the patient's lung collapse to the expected steps of the procedure (which can be accessed or read out in advance) so that the surgical portion of the procedure has begun. It can therefore be determined that collapsing the lung is the first surgical step in this particular procedure.

At the eighth 5216, a medical imaging device 5108 (eg, a scope) is inserted and video footage from the medical imaging device is initiated. Surgical hub 5104 receives medical imaging device data (ie, still image data or real-time live streaming video) through a connection to a medical imaging device. Upon receiving the medical imaging device data, the surgical hub 5104 can determine that the laparoscopic portion of the surgical procedure has begun. Further, the surgical hub 5104 may determine that the particular procedure being performed is a segmentectomy as opposed to a lobectomy (based on data received in the second step 5204 of the procedure). Note that wedge resection is already not taken into account by surgical hub 5104). Data from the medical imaging device 124 (FIG. 2) may be utilized in a variety of ways, for example, by determining the angle of the medical imaging device that is oriented relative to visualization of the patient's anatomy. , by monitoring the number or medical imaging devices being utilized (i.e., activated and paired with the surgical hub 5104), and by monitoring the types of visualization devices being utilized. , contextual information regarding the type of procedure being performed may be determined.

For example, one technique for performing a VATS lobectomy places the camera above the diaphragm in the anterior inferior corner of the patient's thoracic cavity, while one technique for performing a VATS segmentectomy places the camera in an intercostal position anterior to the segmental fissure. The situational awareness system can be trained to recognize the location of the medical imaging device according to the visualization of the patient's anatomy, for example, using pattern recognition or machine learning techniques. As another example, one technique for performing a VATS lobectomy utilizes a single medical imaging device, while another technique for performing a VATS segmentectomy utilizes multiple cameras. As yet another example, one technique for performing a VATS segmentectomy utilizes an infrared light source (which can be communicatively coupled to the surgical hub as part of a visualization system) to visualize the segmental fissure, which is not utilized in a VATS lobectomy. By tracking any or all of this data from the medical imaging device 5108, the surgical hub 5104 can determine the particular type of surgical procedure being performed and/or the technique being used for the particular type of surgical procedure.

In the ninth step 5218, the surgical team begins the incision step of the procedure. Surgical hub 5104 receives data from an RF or ultrasound generator indicating that an energy instrument is being fired, so it can be inferred that the surgeon is in the process of dissecting and separating the patient's lungs. . The surgical hub 5104 cross-references the received data with the read steps of the surgical procedure to determine whether the energy instrument being fired at this point in the process (i.e., after the steps of the procedure described above are completed) It can be determined that the process corresponds to the incision process.

In the tenth step 5220, the surgical team proceeds to the ligation step of the procedure. Surgical hub 5104 receives data from the surgical stapling and cutting instrument indicating that the instrument is being fired, so it can assume that the surgeon is ligating the artery and vein. Similar to the previous step, the surgical hub 5104 can derive this estimate by cross-referencing the receipt of data from the surgical stapling and cutting instruments with the steps in the process that have been read.

In the eleventh step 5222, the segmentectomy portion of the procedure is performed. Surgical hub 5104 estimates that the surgeon is transecting parenchymal tissue based on data from the surgical instruments, including data from the staple cartridge. Cartridge data may correspond, for example, to the size or type of staple being fired by the instrument. Cartridge data may indicate the type of tissue being stapled and/or transected for different types of staples utilized on different types of tissue. The type of staple being fired may be applied to parenchymal tissue or other similar tissue type, and the surgical hub 5104 may assume that a segmental ablation procedure is being performed.

Subsequently, in a twelfth step 5224, a nodule dissection step is performed. The surgical hub 5104 may infer that the surgical team is dissecting the nodule and performing a leak test based on data received from the generator indicating that the RF or ultrasound instrument is being fired. . For this particular procedure, the RF or ultrasound instruments utilized after the parenchymal tissue is transected are compatible with the nodule dissection step, which allows the surgical hub 5104 to make this estimate. Because different instruments are better suited to specific tasks, surgeons may choose between surgical stapling/cutting instruments and surgical energy (i.e., RF or ultrasound) instruments depending on the particular step during the procedure. Note that the Thus, the particular sequence in which the stapling/cutting instruments and surgical energy instruments are used can indicate which step of the procedure the surgeon is performing. Upon completion of the twelfth step 5224, the incision is closed and the post-operative portion of the procedure begins.

In the thirteenth step 5226, the patient is released from anesthesia. Surgical hub 5104 may estimate that the patient is emerging from anesthesia based on, for example, ventilator data (ie, the patient's breathing rate begins to increase).

Finally, at fourteenth 5228, the medical personnel removes the various patient monitoring devices 5124 from the patient. Therefore, the surgical hub 5104 can assume that the patient is being transferred to the recovery room when the hub loses the EKG, BP, and other data from the patient monitoring device 5124. According to data received from various data sources 5126 communicatively coupled to surgical hub 5104, surgical hub 5104 can determine or estimate when each step of a given surgical procedure is occurring. .

As shown in a first step 5202 of timeline 5200 shown in FIG. 11, patient data from the EMR database(s) is utilized to estimate the type of surgical procedure to be performed. Additionally, patient data can also be utilized by the situational awareness surgical hub 5104 to generate control adjustments for the paired modular device 5102.

The present disclosure describes methods and systems for tracking tissue, identifying marked regions of interest, and generating virtual elements representing regions of interest in an augmented reality environment.

Aligning Physical Spatial Parameters In various aspects, a patient may be virtually or physically tagged with fiducial markers to assist the surgeon in surgery. Surgical procedures may require the patient to undergo a preoperative fiducial marking process.

FIG. 12 shows an example of a structured surface 18000 that includes a plurality of fiducial markers. A plurality of points 18010a-d are identified and tagged on the structured surface 18000. A computing system, such as remote server 63 (FIG. 5) or surgical hub 56 (FIG. 6), evaluates the structural surface and assigns fiducial markers 18010a-d based on a target registration error (TRE) model 18008. The TRE model uses the sample data set to estimate the placement of fiducial markers 18010a-d. 18002 shows an initial view of structured surface 18000. 18004 shows a TRE model representation of the structural surface 18000. Transformation 18006 shows the resulting placement of fiducial markers 18010a-d on structured surface 18000.

FIG. 13 shows a process 18020 for surface aligning external structures of a patient with fiducial markers. A computing system, such as a remote server 63 (FIG. 5) or surgical hub 56 (FIG. 6) system, generates an initial mapping of the surface based on the digital representation of the surface (18022). The system uses facial recognition features such as the eyes and bridge of the nose to recognize (18024) multiple anatomical landmarks 18028 and map surfaces and distances between points. Based on the determined distances between anatomical landmarks 18028, the system generates fiducial markers 18030 (18026).

FIG. 14 shows a process 18040 for surface aligning a patient's internal structure 18044 with a fiducial marker 18042. The internal structure is displayed on the output display 18046 and allows the technician to tag areas of the internal structure 18044 with a stylus 18048. Fiducial marker 18042 may correspond to a route or location of a surgical procedure.

15-17 illustrate external surgical alignment instruments that assist a surgeon in a surgical procedure. 15 shows a stereotactic frame 18060, FIG. 16 shows a starfix platform 18080, and FIG. 17 shows a microtable 18100. Surgical hub 56 (FIG. 6) may register and catalog the dimensions of the external surgical aid and assign fiducial markers to multiple points on the surgical aid. Surgical hub 56 can then align the fiducial markers of the external surgical alignment instrument with the patient's surface markers.

In various aspects, surgical hub 56 (FIG. 6) is configured to track the location and position of surgical instruments 12 (FIGS. 1, 2), 21 (FIGS. 5, 6) within operating room 16 (FIG. 2). It is composed of The surgical instrument 12, 21 may include a plurality of fiducial markers strategically placed on the external housing of the instrument 12, 21 to convey specific parameters of the instrument 12, 21. The fiducial marker may be used by a passive tracking system in combination with an infrared (IR) light source. Additionally, the surgical instruments 12, 21 may include sensors that indicate when the surgical instruments 12, 21 are in use and when the surgical instruments 12, 21 are within the patient's body. In one aspect, the trocar can include one or more internal patient sensors that indicate when the device is inserted into the body cavity. Once the trocar determines that a surgical instrument 12, 21 is within a patient's body cavity, the trocar may automatically or manually detect and track the location of another surgical instrument 12, 21. The trocar may be equipped with an internal camera system that identifies fiducial markers on other surgical instruments 12,21. An internal camera system may receive commands to locate the end effector and tag the end effector with a marker associated with the tip of the trocar. The tags are used to monitor the end effector tip throughout the surgical procedure and provide alignment points that can be associated with virtual elements rendered on the AR display 89 of the AR device 66 (FIG. 10). In situations where the end effector tip may be outside of the field of view of the internal camera system, the virtual element may continuously display the tracked position of the end effector tip based on the tag.

Prior to a surgical procedure, all surgical instruments 12 (Figs. 1, 2), 21 (Figs. 5, 6) must be inspected according to their mass, size, length, shape, associated surgical procedure, Classified according to multiple parameters, including hand position. FIG. 18 shows a flowchart for identifying objects based on multiple alignment parameters. In various aspects, the surgical hub receives physical characteristics of objects from one or more object detection cameras within the operating room. The camera provides raw imaging data of the object (18122). The surgical hub performs image processing to remove the back of the image or frame and perform edge detection (18124). Once the surgical hub performs image processing, the surgical hub compares the detected objects to a catalog of objects (18126). The surgical hub reviews the characteristics of each object (18128) and determines the object that most closely matches the identified parameters of the image (18130).

FIG. 19 shows a flow diagram 18140 for classifying unknown surgical instruments based on partial information of known and unknown parameters. The object recognition system may be implemented by remote server 63 (FIG. 5) or surgical hub 56 (FIG. 6). Although the object recognition system may not be able to definitively determine the object, it may narrow the object down to multiple candidates. The system inputs (18142) partial object information and evaluates the object using known parameters. If the system determines the physical properties of the object, the system may perform a geometric uncertainty analysis (18144). If the system determines the geometric properties of the object, the system may perform a physical uncertainty analysis 18146 (18146). However, if the system does not have enough information, it requires manual identification and can classify the object as unknown (18148).

In various aspects, the surgical hub receives spatial and physical parameters associated with an operating room or external environment. Physical parameters may be registered to a particular room or environment. In various aspects, the external environment may be classified according to certain characteristics, such as sterile or non-sterile environment, pre-operative, OR, or post-operative room, and specific equipment in the room (eg, MRI, CT scanner).

A trocar with a camera system attached to the outer diameter keeps the port free for instruments and increases the field of view.
This disclosure further describes a camera system incorporated into a trocar. The camera system allows a wide view of the internal surgical site and 3D mapping of fiducial markers during laparoscopic procedures. Once inside the patient, the camera system is configured to be deployed from a recessed location at the distal end of the trocar. In various embodiments, the internal camera system is configured to keep the trocar port free of surgical instruments and provide surgical personnel with an expanded view of the surgical environment.

FIG. 20 shows a trocar 18160 with an internal camera system 18166. Internal camera system 18166 includes multiple cameras 18166a-n connected together by resilient members 18168. Resilient connection 18168 allows camera system 18166 to be folded together and fit through a passageway defined in the center of the trocar. In various aspects, camera system 18166 emits light in the non-visible spectrum that enables cameras 18166a-n to detect various types of fiducial markers (eg, IR fiducial markers). When internal camera system 18166 is deployed at 18160a, camera system 18166 is attached to outer diameter 18170 of distal end 18162 of trocar 18160. Internal camera system 18166 is in a retracted position 18160b when the trocar is inserted into and removed from a patient's body cavity. In the retracted position 18160b, a camera system 18166 is mounted within the inner diameter 18172 of the distal end 18162 of the trocar 18160.

20 and 21, FIG. 21 shows a reusable camera system 18166 configured to be inserted into the proximal end 18164 of a trocar 18160 and to deploy and retract a camera system 18166 around the outer diameter 18170 of the trocar 18160. An installation tool 18176 is shown. Camera system 18166 is communicatively coupled to surgical hub 56 (FIG. 6) via a wired or wireless connection. In a wireless configuration, each camera 18166a-n may have its own power source (e.g., a rechargeable battery) and the camera system 18166 has a single unit that connects to each camera 18166a-n via an elastically deformable wired connection. may have one external power source. In a wired configuration, a wired connection may be inserted outside of trocar 18160 while it is inserted, keeping the inner diameter of trocar 18160 free to surgical instruments. The camera system 18160 may be attached to the outer diameter of the trocar 18160 through the squeeze and friction of a resilient connection, a magnet in the case of a metal trocar, or a separate connection on the outer diameter of the trocar 18160.

FIG. 21 also shows a side view 18178a of attachment plunger 18174a in a fully depressed position. Conical distal end 18158 of trocar 18160 releases camera system 18166. Plunger 18174b is pulled proximally, causing conical distal end 18158 to force the camera system along outer diameter 18170 of trocar 18160. As plunger 18174c continues to retract proximally, camera system 18166 is attached to outer diameter 18170 of trocar 18160. Reusable attachment tool 18176 is removed so that the laparoscopic surgical procedure can begin.

Fiducial Marker-Based Pre-Operative Computed Tomography (CT) Scanning with Real-Time 3D Model Updates for Improved Partial Treatment Tracking The present disclosure further describes a system configured to generate a 3D model for a surgeon to navigate through a patient's internal tissue structures. The system identifies and marks target tissues or structures of interest in a pre-operative CT scan. The system generates an initial 3D model based on the CT scan that is used by the surgeon to aid in navigation of the internal structures. The 3D model can be continuously updated in real-time based on additional data points received intraoperatively. In one aspect, the system can determine proximity of distance from a surgical instrument and update the model to reflect tissue movement or changes in tissue position.

In various aspects, the system generates 3D renderings of internal tissue structures with virtual elements and displays the 3D model on an augmented reality display. The system may generate a live feed of the surgical environment or provide virtual elements overlaid on a real-world live feed of the surgical site. In various aspects, the 3D model indicates areas of interest, areas to avoid. Additionally, markers can indicate tissue that needs to be sealed or is difficult to find, such as the pulmonary veins and arteries.

FIG. 22 shows a plurality of fiducial markers 18180 tagged to a region of interest in a pre-operative CT scan. The fiducial markers may be placed to create a center of gravity at the structure of interest 18182. A centroid value is determined based on the relative distance between each of the fiducial markers in the set.

FIG. 23 illustrates a laparoscopic surgical procedure that utilizes multiple fiducial markers to assist the surgeon in locating the surgical site. Preoperative determination of critical structures 18184 is generally an approximation of the structure's location, but may not be an exact location. In various aspects, internal cameras may be used with fiducial markers to provide real-time updated locations of critical structures 18186 with updated models or updated fiducial markers. In various aspects, fiducial markers are placed on the surgical instrument 18190 and serve to provide updated locations 18186 based on relationships between other points. The surgical instrument may further include an integrated mapping sensor 18188.

Tracking Tissue Movement and Position with Physical Markers in Laparoscopic Surgery The present disclosure further describes various methods and systems for marking and tracking tissue movement using physical markers. The tracking system includes a camera system configured to detect and track physical markers. In various embodiments, the physical marker includes a magnetic ink, a visible ink in the visible light spectrum, an invisible ink in the invisible light spectrum, or other detectable ink by a camera system.

Figure 24 shows a physical marker applied by injecting indocyanine dye 18202 into the patient's vasculature. Dye 18202 is illuminated by light source 18204 that allows camera 18206 to capture and record the vasculature of tissue 18210. In one aspect, light source 18204 can be a fluorescent light source. Camera system 18206 may visualize dye 18202 using various optical frequencies or lasers. Camera system 18206 is further configured to identify various overlay paths of ink and display 3D renderings on output display 18208. In various aspects, the vasculature can be used like a fingerprint to uniquely identify a structure and track the structure as it moves. Additionally, preoperative CT imaging may be used to help the system generate a 3D map of the structure. This dye can also be used to track organs and alert surgical staff if they are attempting to grasp highly vascularized tissue. FIG. 25 further depicts exemplary tissue injected with dye and illuminated to show vasculature.

In various aspects, light source 18204 may emit light at wavelengths outside the visible spectrum, such as IR. Additionally, the dye 18202 can include magnetic ink used as a marker to distinguish between regions of interest inside and outside the field of view of the camera 18206. In one aspect, the dye 18202 can be splatter sprayed onto the surgical area in the non-visible spectrum so that the body can easily absorb the dye 18202. The splatter creates a unique pattern that allows camera 18206 to easily track the location and movement of tissue 18210.

Intraoperative Tracking of Unfixed Physical Markers for Measuring Anatomical or Surgical Events The present disclosure is further configured to track tissue or anatomical structures without physically fixed anatomical markers. Describe the system that was created. Although physical markers are typically used to track tissue or anatomical structures, there are situations that preclude the use of this method, such as recently sealed tissue. The system tracks tissue using non-fixed markers including temperature and impedance.

FIG. 26 shows a system 18300 configured to monitor pressure or fluid changes within a body cavity according to impedance measurements by a probe 18302. Probe 18302 is coupled to surgical instrument 18304 and measures impedance values based on pressure generated by fluid and/or gas within body cavity 18306. Surgical hub 56 (FIG. 6) may be coupled to surgical instrument 18304 and configured to determine whether there is a change in pressure within body cavity 18306. A change in pressure indicates that a leak exists within the body cavity 18306. Probe 18302 is configured to maintain a fixed gap to measure pressure changes at potential leak site 18308.

Additionally, surgical hub 56 (FIG. 6) may notify surgical personnel of detected leaks by generating alerts through AR content. In one aspect, the virtual element may be rendered on an AR display 89 of an AR device 66 (FIG. 10) that is fixed at the location of the detected event. Virtual elements may be identified with contrasting colors that are solid, flashing, or translucent. The virtual element may be accompanied by a text alert identifying the type of event and/or the severity of the event.

FIG. 27 shows an infrared (IR) thermal detection system 18310 that includes an IR camera system 18312 configured to direct IR light 18314 to a treatment area of tissue 18316 and identify temperature differences within a surgical environment 18304. In one aspect, the IR camera system 18312 may be configured to identify the location of a leak within the pressurized cavity due to changes in the temperature of the air surrounding the leak. The body cavity is pressurized with air at a cooler or warmer temperature than the fluid within the abdominal cavity. IR camera system 18312 observes leaks by looking at gas at a temperature that is either warmer or cooler than the cavity. In response to leak detection, surgical hub 56 may notify surgical personnel.

In one aspect, IR camera system 18312 may determine that the area of tissue has been recently sealed. The sealed tissue may be at a different temperature, allowing the IR camera system 18312 to distinguish the sealed tissue as a sensitive treatment area. Surgical hub 56 (FIG. 6) may render a virtual element overlying the treated area to indicate to the surgeon that the sealed tissue has been recently treated and is a sensitive area.

Sealed tissue is identified based on a predetermined tissue temperature threshold at which the tissue is sealed. Although the tissue temperature may cool slowly, the IR camera system 18312 may mark the region with non-fixed markers that are maintained even after the tissue temperature has fallen below the initial threshold temperature.

Motion tracking system configured to control and adjust surgical instruments to prevent excessive tension on tissue The present disclosure further provides a tissue tracking system configured to prevent excessive tension from being applied to tissue. explain. The system is configured to track markers applied to tissue at specific locations that indicate motion, force, and tension. Surgical hub 56 (FIG. 6) may continuously monitor tissue tension and motion parameters. Surgical hub 56 may determine that tissue tension at a particular location has reached a predetermined threshold and provide a notification to one or more AR devices 66 (FIG. 10).

FIG. 28 depicts a surgical procedure 18350 that employs three end effectors 18352, 18354, 18356 configured to grasp and transect tissue 18360. Tissue 18360 is initially grasped at two points by first end effector 18352 and second end effector 18356. AR device 66 (FIG. 10) may indicate an initial position at which end effector 18352, 18356 is to be positioned. An initial distance 18358 between the first end effector 18352 and the second end effector 18356 is determined based on the force applied to the tissue 18360. A second end effector 18356 is configured to traverse tissue 18360 and requires a third end effector 18354 to compensate for the increased tissue tension.

FIG. 29 shows the third end effector 18354 sliding along tissue from a first position 18354a to a second position 18354b. Surgical personnel monitor the position of the third end effector 18354 to ensure it is properly positioned to compensate for the increased tension. In various aspects, AR device 66 (FIG. 10) may highlight tissue 18360 when it exceeds a predetermined tension threshold. The surgeon may reposition the third end effector 18354 so that the tissue 18360 is no longer highlighted, indicating that the tissue tension is back within the predetermined threshold. In various aspects, the surgeon may receive feedback at the handle or joystick to indicate when repositioning is necessary and when the new position is satisfactory.

FIG. 30 shows a third end effector 18354 positioned adjacent to a second end effector 18356. Upon transecting the tissue 18360 by the second end effector, the tissue tension is within a predetermined threshold based on the initial distance 18358 (FIG. 28).

FIG. 31 shows a surgical procedure 18370 using three static clamps 18372, 18274, 18378 and a dynamic clamp 18376 configured to transfer tissue between the static clamps 18372, 18274, 18378. The first clamp 18372 is a stationary clamp configured to hold the edge of the tissue 18386 and prevent tension or pulling beyond the region of interest 18382. The second clamp 18374 and the third clamp 18378 are positioned according to a predetermined distance so that the tissue maintains a predetermined tension. Second clamp 18374 and third clamp 18378 are stationary, but may be opened and closed to pull new tissue 18386 during stationary distance 18380. The fourth clamp 18376 is a dynamic clamp that pulls the tissue 18386 between the second clamp 18374 and the third clamp 18378 to reduce the tension between the first clamp 18372 and the second clamp 18374. configured to reduce. Fourth clamp 18376 repositions tissue 18386 to reduce excess tension at 18384 as indicated by graphical highlighting. AR device 66 (FIG. 10) may provide similar highlighting to indicate excessive tissue tension.

FIG. 32 shows a logic flow diagram of a method 18400 for displaying a surgical location within a patient. According to method 18400, surgical hub 56 (FIG. 6) receives a video feed from one or more cameras located inside a patient (18402). Surgical hub 56 identifies one or more physical markers within the patient (18404). Surgical hub 56 determines the target location based on the relationship with one or more physical markers (18406). Surgical hub 56 generates a virtual element corresponding to the target location (18408). AR device 66 (FIG. 10) coupled to surgical hub 56 (FIG. 6) displays virtual elements overlaid on the video feed on augmented reality (AR) display 79 (FIG. 10) (18410).

In one aspect of method 18400, the video feed is a wide-angle view stitched together from at least two video feeds. In another aspect, according to method 18400, the one or more physical markers are visible under illumination of a light source outside the visible spectrum. In another aspect of method 18400, the one or more physical markers are fiducial markers assigned in a preoperative computed tomography (CT) scan. In yet another aspect of method 18400, the target location is continuously updated in real time on augmented reality (AR) device 66 (FIG. 10).

Various additional aspects of the subject matter described herein are illustrated in the numbered examples below.

Example 1: A surgical system comprising a surgical device having an axial passageway defining an outer diameter and an inner diameter, a proximal end, and a distal end configured to penetrate tissue. , a camera array comprising individual cameras connected in a ring configuration by resilient connections; a removable attachment trigger configured to extend to a second deployed position circumferentially around the outer diameter of the distal end of the surgical device; a surgical hub communicatively coupled to the camera array and the AR device, the surgical hub comprising a control circuit coupled to the memory, the control circuit communicatively coupled to the camera array and the AR device; A surgical system configured to receive a plurality of video feeds from a user, identify a physical marker on the video feed, and display the physical marker on an AR display.

Example 2: The surgical system of Example 1, wherein the video feed is a wide-angle view stitched together from each of the individual cameras.

Example 3: The surgical system according to any one of Examples 1 and 2, wherein the physical marker is visible under illumination of a light source outside the visible spectrum.

Example 4: A surgical system according to any one of Examples 1 to 3, in which the physical marker is a dye.

Example 5: The surgical system of any one of Examples 1-4, wherein the physical marker is a fiducial marker assigned in a preoperative computed tomography (CT) scan.

Example 6: A surgical system according to any one of Examples 1 to 5, wherein the physical marker is configured to indicate a target location for a surgical procedure.

Example 7: The surgical system of Example 6, wherein the target location is continuously updated in real time on the AR device.

Example 8: The surgical system of Example 7, wherein the target location is updated based on the relationship between the surgical instrument and the physical marker.

Example 9: A surgical device comprising: a camera array comprising individual cameras connected in a ring configuration by resilient connections, the camera array being communicatively connectable to a surgical hub; an elongate penetrating member having an end and a distal end further comprising a tissue penetrating tip; an axial passageway through the elongate penetrating member and the tissue penetrating tip, the inner diameter of the axial passageway having a distal end portion further comprising a tissue penetrating tip; an axial passageway dimensioned to receive the camera array in the first recessed location; a removable attachment trigger configured to extend to a second deployment position circumferentially positioned about the outer diameter of the distal end of the penetrating member.

Example 10: The surgical device of Example 9, wherein the camera array is communicatively coupleable with the surgical hub via a wireless communication protocol.

Example 11: The surgical device of Example 10, wherein the camera array is powered by a rechargeable battery.

Example 12: The surgical device of any one of Examples 9-11, wherein the camera array is communicatively connectable to the surgical hub via a wired communication protocol.

Example 13: The surgical device of Example 12, wherein the camera array is powered by a wired external power source comprising a wire extending along the outer diameter of the elongated penetrating member.

Example 14: The surgical device of any one of Examples 9-13, wherein the camera array is attached to the outer diameter of the distal end of the elongated penetrating member in a squeeze and friction configuration.

Example 15: The surgical device of any one of Examples 9-14, wherein the camera array does not occupy space in the inner diameter of the axial passageway in the second deployed position.

Example 16: A method for displaying a surgical location inside a patient, the method including: receiving, by a surgical hub, a video feed from a camera located inside the patient; identifying, by the surgical hub, physical markers inside the patient; determining, by the surgical hub, a target location based on a relationship to the physical markers; generating, by the surgical hub, a virtual element corresponding to the target location; and displaying, by an augmented reality (AR) device coupled to the surgical hub, the virtual element overlaid on the video feed on an AR display.

Example 17: The method of example 16, wherein the video feed is a wide-angle view stitched together from at least two video feeds.

Example 18: The method of any one of Examples 16 and 17, wherein the physical marker is visible only under illumination of a light source outside the visible spectrum.

Example 19: The method of any one of Examples 16-18, wherein the physical marker is a fiducial marker assigned in a preoperative computed tomography (CT) scan.

Example 20: The method of any one of Examples 16 to 19, wherein the target location is continuously updated in real time on an augmented reality (AR) device.

While several forms have been shown and described, it is not the intention of the applicant to limit or limit the scope of the appended claims to such details. Many modifications, variations, changes, permutations, combinations and equivalents of these forms may be implemented and will occur to those skilled in the art without departing from the scope of this disclosure. Furthermore, the structure of each element associated with the described form may alternatively be described as a means for providing the functionality performed by that element. Also, although materials are disclosed for particular components, other materials may be used. It is therefore intended that the foregoing description and appended claims cover all such modifications, combinations, and variations as falling within the scope of the disclosed forms. I want you to understand. The appended claims are intended to cover all such modifications, variations, changes, substitutions, modifications, and equivalents.

The detailed description above has described various forms of apparatus and/or processes using block diagrams, flow diagrams, and/or example embodiments. To the extent that such block diagrams, flow diagrams and/or examples include one or more features and/or operations, it will be understood by those skilled in the art that such block diagrams, flow diagrams and/or examples include one or more features and/or operations. Each included feature and/or operation may be implemented individually and/or collectively by a variety of hardware, software, firmware, or virtually any combination thereof. Those skilled in the art will appreciate that some aspects of the forms disclosed herein can be implemented, in whole or in part, as one or more computer programs running on one or more computers (e.g., as one or more computer programs running on one or more computers). (as one or more programs running on a computer system); as one or more programs running on one or more processors (e.g., as one or more programs running on one or more microprocessors); Designing circuits and/or writing software and/or firmware code that can equivalently be implemented on an integrated circuit as firmware, or as virtually any combination thereof, utilizes the present disclosure. It will be understood that it is within the skill of those skilled in the art. In addition, those skilled in the art will appreciate that the features of the subject matter described herein can be distributed in a variety of forms as one or more program products, The specific form of the subject matter applies without regard to the particular type of signal-bearing medium used to actually effectuate the distribution.

The instructions used to program the logic to implement the various disclosed aspects may be stored in memory within the system, such as dynamic random access memory (DRAM), cache, flash memory, or other storage. Additionally, the instructions may be distributed over a network or by other computer-readable media. Thus, a machine-readable medium may include any mechanism for storing or transmitting information in a form readable by a machine (e.g., a computer), including floppy diskettes, optical disks, compact disks, read-only memory (CDs), etc. - ROM), as well as magneto-optical disks, read only memory (ROM), random access memory (RAM), erasable programmable read only memory (EPROM), electrically erasable programmable read only memory (EEPROM), magnetic or optical cards, Flash memory or other tangible machine-readable storage used to transmit information over the Internet via electrical, optical, acoustic, or other forms of propagating signals (e.g., carrier waves, infrared signals, digital signals, etc.) but not limited to. Thus, non-transitory computer-readable media includes any type of tangible machine-readable medium suitable for storing or transmitting electronic instructions or information in a form readable by a machine (eg, a computer).

As used in any aspect herein, the term "control circuit" refers to, for example, hard-wired circuits, programmable circuits (e.g., computer processors including one or more individual instruction processing cores, processing units, , microcontroller, microcontroller unit, controller, digital signal processor (DSP), programmable logic device (PLD), programmable logic array (PLA), or field programmable gate array (FPGA)), state machine circuit, programmable circuit. can refer to firmware that stores instructions for use in a computer, and any combination thereof. Control circuits may be used, collectively or individually, in, for example, integrated circuits (ICs), application specific integrated circuits (ASICs), systems on chips (SoCs), desktop computers, laptop computers, tablet computers, servers, smartphones, etc. , may be implemented as a circuit that forms part of a larger system. Accordingly, as used herein, "control circuit" refers to an electrical circuit having at least one discrete electrical circuit, an electrical circuit having at least one integrated circuit, an electrical circuit having at least one application-specific integrated circuit. a circuit, a general-purpose computing device configured with a computer program (e.g., a general-purpose computer configured with a computer program that at least partially executes a process and/or device described herein, or a process described herein) and/or an electrical circuit forming a memory device (e.g. in the form of a random access memory) and/or a communication device (e.g. a modem). , communications switches, or optical-electrical equipment). Those skilled in the art will recognize that the subject matter described herein may be implemented in analog or digital form, or some combination thereof.

As used in any aspect herein, the term "logic" may refer to applications, software, firmware, and/or circuitry configured to perform any of the operations described above. Software may be embodied as a software package, code, instructions, instruction sets, and/or data recorded on a non-transitory computer-readable storage medium. Firmware may be embodied as code, instructions, or a set of instructions in a memory device, and/or hard-coded (eg, non-volatile) data.

As used in any aspect herein, the terms "component," "system," "module," etc. refer to control circuitry, computer-related entities, hardware, combinations of hardware and software, software, or any running software.

As used in any aspect herein, "algorithm" refers to a self-consistent sequence of steps leading to a desired result; Refers to the manipulation of physical quantities and/or logical states, which can take the form of electrical or magnetic signals, which can be compared, compared, and otherwise manipulated. It is common practice to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, or the like. These and similar terms may be associated with the appropriate physical quantities or are simply convenient labels applied to these quantities and/or conditions.

The network may include a packet switched network. The communication devices can communicate with each other using selected packet-switched network communication protocols. One example communication protocol may include the Ethernet communication protocol, which may enable communication using Transmission Control Protocol/Internet Protocol (TCP/IP). The Ethernet protocol conforms to or is compatible with the Ethernet standard published by the Institute of Electrical and Electronics Engineers (IEEE) entitled "IEEE 802.3 Standard" published December 2008, and/or any later version of this standard. It can be sexual. Alternatively or additionally, the communication device is an X. can communicate with each other using the X.25 communication protocol. X. The T.25 communication protocols may conform to or be compatible with standards promulgated by the International Telecommunication Union-Telecommunication Standardization Sector (ITU-T). Alternatively or additionally, communication devices may communicate with each other using a frame relay communication protocol. The Frame Relay communication protocol is a protocol developed by the Consultative Committee for International Telegraph and Telephone (CCITT) and/or the American National Standards Institute (AN SI) or may be compatible with the standards promulgated by SI. Alternatively or additionally, the transceivers may be capable of communicating with each other using an asynchronous transfer mode (ATM) communication protocol. The ATM communications protocol shall comply with or be compatible with the ATM standard published by the ATM Forum in August 2001 under the title "ATM-MPLS Network Interworking 2.0" and/or with later versions of this standard. obtain. Of course, different and/or later developed connection-oriented network communication protocols are equally contemplated herein.

Unless expressly specified otherwise, as will be apparent from the foregoing disclosure, discussions throughout the foregoing disclosure using terms such as "processing," "computing," "calculating," "determining," "displaying," and the like will be understood to refer to the actions and processing of a computer system or similar electronic computing device that manipulates and converts data represented as physical (electronic) quantities in the registers and memory of the computer system into other data similarly represented as physical quantities in the memory or registers of the computer system or other such information storage, transmission, or display devices.

One or more components are defined herein as "configured to," "configurable to," "operable/operating." (operable/operative to), “adapted/adaptable,” “able to,” “conformable/conformed to.” ” may be mentioned. Those skilled in the art will understand that "configured to" generally refers to active components and/or inactive components and/or standby components, unless the context requires otherwise. It will be understood that elements can be included.

The terms "proximal" and "distal" are used herein with reference to the clinician manipulating the handle portion of the surgical instrument. The term "proximal" refers to the part closest to the clinician, and the term "distal" refers to the part located away from the clinician. It will be further understood that for convenience and clarity, spatial terms such as "vertical," "horizontal," "above," and "below" may be used herein with respect to the drawings. However, surgical instruments may be used in many orientations and positions, and these terms are not intended to be limiting and/or absolute.

Those skilled in the art will generally understand that the terms used herein, and particularly in the appended claims (e.g., the text of the appended claims), are generally intended as "open" terms. (For example, the term "including" should be interpreted as "including but not limited to," and the term "having" should be interpreted as "including but not limited to.") should be interpreted as “having at least” and the term “includes” should be interpreted as “includes but is not limited to.” should be done, etc.). Further, if a specific number is intended in an introduced claim recitation, such intent will be clearly stated in the claim, and if no such statement is present, no such intent exists. will be understood by those skilled in the art. For example, as an aid to understanding, the following appended claims incorporate the introductory phrases "at least one" and "one or more" into claim statements. may be included in order to However, the use of such phrases does not apply when the claim is introduced by the indefinite article "a" or "an," even if the introductory phrase "one or more" or "at least one" appears within the same claim. and the indefinite article "a" or "an," any particular claim containing such an introduced claim statement is limited to a claim containing only one such statement. (e.g., "a" and/or "an" should generally be interpreted to mean "at least one" or "one or more") ). The same applies when introducing claim statements using definite articles.

Additionally, even if a particular number is specified in an introduced claim statement, such statement is typically to be construed to mean at least the recited number. will be recognized by those skilled in the art. ). Further, when a notation similar to "at least one of A, B, and C, etc." is used, such construction is generally intended in the sense that one of ordinary skill in the art would understand the notation (e.g. , "a system having at least one of A, B, and C" includes, but is not limited to, A only, B only, C only, both A and B, both A and C, B and C. and/or all of A, B, and C, etc.). When a notation similar to "at least one of A, B, or C, etc." is used, such construction is generally intended in the sense that a person skilled in the art would understand the notation (e.g., "A , B, or C includes, but is not limited to, only A, only B, only C, both A and B, both A and C, both B and C. and/or systems having all of A, B, C, etc.). Further, typically any disjunctive words and/or phrases representing two or more alternative terms, whether within the specification, unless the context otherwise requires; It should be understood that the possibility of including one, either, or both of these terms is intended, whether in the claims or in the drawings. Those skilled in the art will understand that. For example, the phrase "A or B" will typically be understood to include the possibilities "A" or "B" or "A and B."

With reference to the appended claims, those skilled in the art will appreciate that the acts recited herein may generally be performed in any order. Additionally, while flow diagrams of various operations are shown in sequence(s), it is understood that the various operations may be performed in an order other than that shown, or may be performed simultaneously. It should be. Examples of such alternative orderings include overlapping, interleaving, interleaving, reordering, incremental, preliminary, additional, simultaneous, reverse or other different orderings, unless the context requires otherwise. May include. Additionally, terms such as "responsive to," "related to," or other past tense adjectives generally do not mean otherwise, unless the context requires otherwise. It is not intended that such variations be excluded.

Any reference to "an aspect," "an aspect," "an example," "an example," or the like includes a particular feature, structure, or characteristic described in connection with that aspect that is included in at least one aspect. The meaning of this is worth noting. Thus, the phrases "in one aspect," "in an aspect," "in an example," and "in one example" appearing in various places throughout this specification do not necessarily all refer to the same aspect. Furthermore, the particular features, structures, or characteristics may be combined in any suitable manner in one or more aspects.

Any patent application, patent, non-patent publication, or other disclosure material referenced herein and/or listed in any application data sheet is incorporated by reference herein to the extent that the incorporated material is not inconsistent with this specification. , incorporated herein by reference. As such, and to the extent necessary, disclosure expressly set forth herein shall supersede any conflicting description incorporated herein by reference. Any material, or portion thereof, that is cited as being incorporated herein by reference but is inconsistent with current definitions, views, or other disclosures set forth herein shall be deemed to be incorporated by reference. They are incorporated only to the extent that they are consistent with existing disclosures.

In summary, many benefits have been described that result from using the concepts described herein. The above description of one or more forms has been presented for purposes of illustration and description. It is not intended to be exhaustive or limited to the precise form disclosed. Modifications or variations are possible in light of the above teachings. The one or more embodiments are intended to be illustrative of the principles and practical application, thereby enabling those skilled in the art to utilize the various embodiments, with various modifications, as appropriate for the particular application contemplated. It has been selected and written in order to It is intended that the claims presented herewith define the overall scope.

[Embodiment]
(1) A surgical system comprising:
1. A surgical device comprising:
an axial passage defining an outer diameter and an inner diameter;
A proximal end portion;
a distal end configured to penetrate tissue;
a camera array comprising individual cameras connected in a ring configuration by elastic connections;
a detachable attachment trigger configured to extend the camera array from a first recessed position from the inner diameter of the distal end of the axial passage to a second deployed position in which the camera array is positioned circumferentially around the outer diameter of the distal end of the axial passage; and
an augmented reality (AR) device;
a surgical hub communicatively coupled to the camera array and the AR device, the surgical hub comprising a control circuit coupled to a memory, the control circuit comprising:
receiving a plurality of video feeds from the camera array;
Identifying a physical marker on the video feed;
A surgical system configured to display the physical marker on the AR display.
(2) The surgical system of claim 1, wherein the video feed is a wide-angle view from each of the individual cameras stitched together.
(3) The surgical system of claim 1, wherein the physical marker is visible under illumination of a light source outside the visible spectrum.
(4) The surgical system of claim 1, wherein the physical marker is a dye.
(5) The surgical system of claim 1, wherein the physical markers are fiducial markers assigned in a preoperative computed tomography (CT) scan.

(6) The surgical system of embodiment 1, wherein the physical marker is configured to indicate a target location for a surgical procedure.
7. The surgical system of embodiment 6, wherein the target location is continuously updated in real time on the AR device.
(8) The surgical system of embodiment 7, wherein the target location is updated based on a relationship between a surgical instrument and the physical marker.
(9) A surgical device,
a camera array including individual cameras connected in a ring configuration by resilient connections, the camera array being communicatively connectable to a surgical hub;
an elongate penetrating member having a proximal end and a distal end further comprising a tissue penetrating tip;
an axial passageway through the elongate penetrating member and the tissue penetrating tip, the inner diameter of the axial passageway being dimensioned to accommodate the camera array in a first recessed position; A passage and
the camera array from a first recessed position from the inner diameter of the distal end of the elongate penetrating member, the camera array circumferentially about the outer diameter of the distal end of the elongate penetrating member; a removable attachment trigger configured to extend to a second deployment position positioned in the direction.
10. The surgical device of embodiment 9, wherein the camera array is communicatively coupled with the surgical hub via a wireless communication protocol.

11. The surgical device of embodiment 10, wherein the camera array is powered by a rechargeable battery.
12. The surgical device of embodiment 9, wherein the camera array is communicatively coupled to the surgical hub via a wired communication protocol.
13. The surgical device of claim 12, wherein the camera array is powered by a wired external power source comprising a wire extending along the outer diameter of the elongated penetrating member.
14. The surgical device of claim 9, wherein the camera array is attached to the outer diameter of the distal end of the elongate penetrating member in a squeeze and friction configuration.
15. The surgical device of embodiment 9, wherein the camera array does not occupy space in the inner diameter of the axial passageway in the second deployed position.

(16) A method for displaying a surgical location inside a patient, the method comprising:
receiving a video feed from a camera located inside the patient by the surgical hub;
identifying a physical marker within the patient with the surgical hub;
determining, by the surgical hub, a target location based on a relationship with the physical marker;
generating, by the surgical hub, a virtual element corresponding to the target location;
displaying the virtual element overlaid on the video feed on an AR display by an augmented reality (AR) device coupled to the surgical hub.
17. The method of embodiment 16, wherein the video feed is a wide-angle view stitched together from at least two video feeds.
18. The method of claim 16, wherein the physical marker is visible under illumination of a light source outside the visible spectrum.
19. The method of claim 16, wherein the physical marker is a fiducial marker assigned in a preoperative computed tomography (CT) scan.
20. The method of embodiment 16, wherein the target location is continuously updated in real time on an augmented reality (AR) device.

Claims (20)

A surgical system comprising:
A surgical device,
an axial passageway defining an outer diameter and an inner diameter;
a proximal end;
a distal end configured to penetrate tissue;
a camera array comprising individual cameras connected in a ring configuration by resilient connections;
the camera array from a first recessed position from the inner diameter of the distal end of the axial passageway; a removable attachment trigger configured to extend to a circumferentially positioned second deployment position;
Augmented reality (AR) device and
a surgical hub communicatively coupled to the camera array and the AR device, the surgical hub comprising a control circuit coupled to memory, the control circuit comprising:
receiving a plurality of video feeds from the camera array;
identifying a physical marker on the video feed;
A surgical system configured to display the physical marker on the AR display. The surgical system of claim 1, wherein the video feed is a wide-angle view stitched together from each of the individual cameras. The surgical system of claim 1, wherein the physical marker is visible under illumination of a light source outside the visible spectrum. The surgical system of claim 1, wherein the physical marker is a dye. The surgical system of claim 1, wherein the physical marker is a fiducial marker assigned in a preoperative computed tomography (CT) scan. The surgical system of claim 1, wherein the physical marker is configured to indicate a target location for a surgical procedure. The surgical system of claim 6, wherein the target location is continuously updated in real time on the AR device. The surgical system of claim 7, wherein the target location is updated based on a relationship between a surgical instrument and the physical marker. A surgical device,
a camera array including individual cameras connected in a ring configuration by resilient connections, the camera array being communicatively connectable to a surgical hub;
an elongate penetrating member having a proximal end and a distal end further comprising a tissue penetrating tip;
an axial passageway through the elongate penetrating member and the tissue penetrating tip, the inner diameter of the axial passageway being dimensioned to accommodate the camera array in a first recessed position; A passage and
the camera array from a first recessed position from the inner diameter of the distal end of the elongate penetrating member, the camera array circumferentially about the outer diameter of the distal end of the elongate penetrating member; a removable attachment trigger configured to extend to a second deployment position positioned in the direction. The surgical device of claim 9, wherein the camera array is communicatively coupleable with the surgical hub via a wireless communication protocol. The surgical device of claim 10, wherein the camera array is powered by a rechargeable battery. The surgical device of claim 9, wherein the camera array is communicatively coupleable to the surgical hub via a wired communication protocol. The surgical device of claim 12, wherein the camera array is powered by a wired external power source comprising a wire extending along the outer diameter of the elongated penetrating member. The surgical device of claim 9, wherein the camera array is attached to the outer diameter of the distal end of the elongate penetrating member in a squeeze and friction configuration. The surgical device of claim 9, wherein the camera array does not occupy space in the inner diameter of the axial passageway in the second deployed position. A method for displaying a surgical location within a patient, the method comprising:
receiving a video feed from a camera located inside the patient by the surgical hub;
identifying a physical marker within the patient with the surgical hub;
determining, by the surgical hub, a target location based on a relationship with the physical marker;
generating, by the surgical hub, a virtual element corresponding to the target location;
displaying the virtual element overlaid on the video feed on an AR display by an augmented reality (AR) device coupled to the surgical hub. 17. The method of claim 16, wherein the video feed is a wide-angle view stitched together from at least two video feeds. 17. The method of claim 16, wherein the physical marker is visible under illumination of a light source outside the visible spectrum. The method of claim 16, wherein the physical markers are fiducial markers assigned on a preoperative computed tomography (CT) scan. 17. The method of claim 16, wherein the target location is continuously updated in real time on an augmented reality (AR) device.

Images