EP4648702A1

EP4648702A1 - Surgical robotic system and method for navigating surgical instruments

Info

Publication number: EP4648702A1
Application number: EP24700493.0A
Authority: EP
Inventors: Tuvia C. Rappaport; Max L. BALTER; Paul M. Loschak
Original assignee: Covidien LP
Current assignee: Covidien LP
Priority date: 2023-01-13
Filing date: 2024-01-04
Publication date: 2025-11-19
Also published as: WO2024150088A1; CN120513063A

Abstract

A surgical robotic system includes a first surgical robotic arm having a first instrument, a second surgical robotic arm having a second instrument, and a surgeon console configured to receive user input to control navigation of the first instrument. The surgeon console includes a display and a processor. The processor is configured to detect a location of a first point within a surgical site corresponding to a navigated position of the first instrument, calculate a location of a second point within the surgical site based on the first point, calculate a trajectory for navigating the second instrument to the second point, display an overlay of the trajectory on a user interface, and cause the second instrument to navigate to the second point along the trajectory.

Description

SURGICAL ROBOTIC SYSTEM AND METHOD FOR NAVIGATING SURGICAL INSTRUMENTS

CROSS-REFERENCE TO RELATED APPLICATION

[0001] This application claims the benefit of U.S. Provisional Patent Application Serial No. 63/438,857, filed January 13, 2023, the entire content of which is incorporated herein by reference.

BACKGROUND

[0002] Surgical robotic systems are used in a variety of surgical procedures, including minimally invasive medical procedures. Some surgical robotic systems include a surgeon console controlling a surgical robotic arm and a surgical instrument having an end effector (e.g., forceps or grasping instrument) coupled to and actuated by the robotic arm. In operation, the robotic arm is moved to a position over a patient and then guides the surgical instrument into a small incision via a surgical port or a natural orifice of a patient to position the end effector of the surgical instrument at a work site within the patient’s body.

SUMMARY

[0003] Wristed surgical instruments are easier for a clinician to navigate within a surgical work site using a surgical robotic system than non-wristed instruments due the limited degrees of freedom of the end effectors of non-wristed instruments. The present disclosure provides for a surgical robotic system which enables a clinician to indirectly navigate an end effector of a non-wristed instrument to a specific point within a work site based on the clinician’s navigation of one or more wristed surgical instruments to one or more points.

[0004] According to one embodiment of the present disclosure, a surgical robotic system is disclosed. The surgical robotic system includes: a first surgical robotic arm including a wristed instrument; a second surgical robotic arm including a non-wristed instrument; and a surgeon console including a hand controller configured to receive user input to control navigation of the wristed instrument. The surgeon console includes a display configured to display a user interface, memory storing instructions, and a processor. The processor is configured to detect a location of a first point within the surgical site corresponding to a navigated position of the wristed instrument; calculate a location of a second point within the surgical site based on the first point; calculate a trajectory for navigating the non-wristed instrument to the second point; display an overlay of the trajectory for navigating the non- wristed instrument to the second point on the user interface; display a prompt for acceptance of the trajectory for navigating the non-wristed instrument to the second point on the user interface; and instruct the second robotic arm to move the non-wristed instrument to the second point along the trajectory for navigating the non-wristed instrument to the second point.

[0005] In an aspect, the processor may be configured to calculate the location of the second point based on an offset from the location of the first point. Additionally, or alternatively, the offset may be user-selectable.

[0006] In an aspect, the location of the second point may be the same as the location of the first point.

[0007] In an aspect, the processor may be configured to calculate a trajectory for retracting the wristed instrument from the first point, and display on the user interface an overlay of the trajectory for retracting the wristed instrument from the first point.

[0008] In an aspect, the processor may be configured to detect a location of a third point within the surgical site corresponding to a navigated position of a second wristed instrument, and calculate the location of the second point based on a first offset from the location of the first point and a second offset from the location of the third point. Additionally, or alternatively, the location of the third point may be between the first point and the second point. [0009] In an aspect, the processor may be configured to display the overlay of the trajectory for navigating the non-wristed instrument to the second point on the user interface by displaying an animation of a representation of the non-wristed instrument moving along the trajectory.

In another aspect of the present disclosure, a method for navigating to a target within a surgical site is provided. The method includes detecting a location of a first point within the surgical site corresponding to a navigated position of a first instrument; calculating a location of the target within the surgical site based on the first point; calculating a trajectory for navigating a second instrument to the target; displaying a prompt for acceptance of the trajectory for navigating the second instrument to the target; and instructing a robotic arm to move the second instrument to the target along the trajectory for navigating the second instrument to the target based on an acceptance of the prompt.

[0010] In an aspect, the method may further include displaying an overlay of the trajectory for navigating the second instrument to the target on a user interface of a display. [0011] In an aspect, displaying the overlay of the trajectory for navigating the second instrument to the target may include displaying an animation of a representation of the second instrument moving along the trajectory.

[0012] In an aspect, calculating a trajectory for navigating a second instrument to the target may include calculating the location of the target based on an offset from the location of the first point. Additionally, or alternatively, the offset may be user-selectable.

[0013] In an aspect, the location of the target may be the same as the location of the first point.

[0014] In an aspect, the method may further include calculating a trajectory for retracting the first instrument from the first point, and displaying an overlay of the traj ectory for retracting the first instrument from the first point.

[0015] In an aspect, the method may further include detecting a location of a third point within the surgical site, the third point corresponding to a navigated position of a third instrument. Additionally, or alternatively, calculating the location of the target within the surgical site based on the first point includes calculating the location of the target based on a first offset from the location of the first point and a second offset from the location of the third point. Additionally, or alternatively, the location of the target may be between the first point and the third point.

[0016] In another aspect, a surgical robotic system is provided and includes a first instrument, a second instrument, and a surgeon console including a hand controller configured to receive user input to control navigation of the first instrument within a surgical site. The surgeon console includes a display configured to display a user interface, memory storing instructions, and a processor. The processor is configured to detect a location of a first point within the surgical site corresponding to a navigated position of the first instrument; calculate a location of a target within the surgical site based on at least one of the location of the first point or an offset from the location of the first point; calculate a trajectory for navigating the second instrument to the target; display an overlay of the trajectory for navigating the second instrument to the target on the user interface by displaying an animation of a representation of the second instrument moving along the trajectory; display a prompt for acceptance of the trajectory for navigating the second instrument to the target; and move the second instrument to the target along the trajectory for navigating the second instrument to the target if the prompt is accepted. [0017] In an aspect, the processor may be configured to calculate a trajectory for retracting the first instrument from the first point, and display an overlay of the trajectory for retracting the first instrument from the first point.

[0018] In an aspect, the processor may be configured to determine whether the trajectory for retracting the first instrument from the first point is acceptable, and cause the second instrument to navigate to the target based on the trajectory for navigating the second instrument to the target if the prompt is accepted and it is determined that the trajectory for retracting the first instrument from the first point is acceptable.

BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Various embodiments of the present disclosure are described herein with reference to the drawings wherein:

[0020] FIG. 1 is a schematic illustration of a surgical robotic system including a control tower, a console, and one or more surgical robotic arms each disposed on a mobile cart according to an embodiment of the present disclosure;

[0021] FIG. 2 is a perspective view of a surgical robotic arm of the surgical robotic system of FIG. 1 according to an embodiment of the present disclosure;

[0022] FIG. 3 is a perspective view of a mobile cart having a setup arm with the surgical robotic arm of the surgical robotic system of FIG. 1 according to an embodiment of the present disclosure;

[0023] FIG. 4 is a schematic diagram of a computer architecture of the surgical robotic system of FIG. 1 according to an embodiment of the present disclosure;

[0024] FIG. 5 is a plan schematic view of the surgical robotic system of FIG. 1 positioned about a surgical table according to an embodiment of the present disclosure;

[0025] FIG. 6 is a view of a screen of a surgeon console displaying a first user interface according to an embodiment of the present disclosure;

[0026] FIG. 7 is a view of a screen of a surgeon console displaying a second user interface according to an embodiment of the present disclosure;

[0027] FIG. 8 is a view of a screen of a surgeon console displaying a third user interface according to an embodiment of the present disclosure;

[0028] FIG. 9 is a flow chart illustrating a method for navigating a surgical instrument to a target in connection with the first user interface of FIG. 7;

[0029] FIG. 10 is a flow chart illustrating a method for navigating a surgical instrument to a target in connection with the second user interface of FIG. 8; and [0030] FIG. 11 is a flow chart illustrating a method for navigating a surgical instrument to a target in connection with the third user interface of FIG. 9.

DETAILED DESCRIPTION

[0031] Embodiments of the presently disclosed surgical robotic system are described in detail with reference to the drawings, in which like reference numerals designate identical or corresponding elements in each of the several views.

[0032] As will be described in detail below, the present disclosure is directed to a surgical robotic system, which includes a surgeon console, a control tower, and one or more mobile carts having a surgical robotic arm coupled to a setup arm. The surgeon console receives user input through one or more interface devices, which are processed by the control tower as movement commands for moving the surgical robotic arm and an instrument and/or camera coupled thereto. Thus, the surgeon console enables teleoperation of the surgical arms and attached instruments/camera. The surgical robotic arm includes a controller, which is configured to process the movement commands and to generate torque commands for activating one or more actuators of the robotic arm, which would, in turn, move the robotic arm in response to the movement command.

[0033] With reference to FIG. 1, a surgical robotic system 10 includes a control tower 20, which is connected to all of the components of the surgical robotic system 10 including a surgeon console 30 and one or more mobile carts 60. Each of the mobile carts 60 includes a robotic arm 40 having a surgical instrument 50 removably coupled thereto. The robotic arms 40 also couple to the mobile carts 60. The robotic system 10 may include any number of mobile carts 60 and/or robotic arms 40.

[0034] The surgical instrument 50 is configured for use during minimally invasive surgical procedures. In embodiments, the surgical instrument 50 may be configured for open surgical procedures. In further embodiments, the surgical instrument 50 may be an electrosurgical forceps configured to seal tissue by compressing tissue between jaw members and applying electrosurgical current thereto. In yet further embodiments, the surgical instrument 50 may be a surgical stapler including a pair of jaws configured to grasp and clamp tissue while deploying a plurality of tissue fasteners, e.g., staples, and cutting stapled tissue. In yet further embodiments, the surgical instrument 50 may be a surgical clip applier including a pair of jaws configured apply a surgical clip onto tissue. While some surgical instruments 50 may have wristed distal portions, which are capable of articulating in multiple directions, some surgical instruments 50 are non-wristed. [0035] One of the robotic arms 40 may include a laparoscopic camera 51 configured to capture video of the surgical site. The laparoscopic camera 51 may be a stereoscopic endoscopic camera configured to capture two side-by-side (i.e., left and right) images of the surgical site to produce a video stream of the surgical scene. The laparoscopic camera 51 is coupled to an image processing device 56, which may be disposed within the control tower 20. The image processing device 56 may be any computing device configured to receive the video feed from the laparoscopic camera 51 and output the processed video stream.

[0036] The surgeon console 30 includes a first screen 32, which displays a video feed of the surgical site provided by camera 51 of the surgical instrument 50 disposed on the robotic arm 40, and a second screen 34, which displays a user interface for controlling the surgical robotic system 10. The first screen 32 and second screen 34 may be touchscreens allowing for displaying various graphical user inputs.

[0037] The surgeon console 30 also includes a plurality of user interface devices, such as foot pedals 36 and a pair of hand controllers 38a and 38b which are used by a user to remotely control robotic arms 40. The surgeon console further includes an armrest 33 used to support clinician’s arms while operating the hand controllers 38a and 38b.

[0038] The control tower 20 includes a screen 23, which may be a touchscreen, and outputs on the graphical user interfaces (GUIs). The control tower 20 also acts as an interface between the surgeon console 30 and one or more robotic arms 40. In particular, the control tower 20 is configured to control the robotic arms 40, such as to move the robotic arms 40 and the corresponding surgical instrument 50, based on a set of programmable instructions and/or input commands from the surgeon console 30, in such a way that robotic arms 40 and the surgical instrument 50 execute a desired movement sequence in response to input from the foot pedals 36 and the hand controllers 38a and 38b. The foot pedals 36 may be used to enable and lock the hand controllers 38a and 38b, repositioning camera movement and electrosurgical activation/deactivation. In particular, the foot pedals 36 may be used to perform a clutching action on the hand controllers 38a and 38b. Clutching is initiated by pressing one of the foot pedals 36, which disconnects (i.e., prevents movement inputs) the hand controllers 38a and/or 38b from the robotic arm 40 and corresponding instrument 50 or camera 51 attached thereto. This allows the user to reposition the hand controllers 38a and 38b without moving the robotic arm(s) 40 and the instrument 50 and/or camera 51. This is useful when reaching control boundaries of the surgical space.

[0039] Each of the control tower 20, the surgeon console 30, and the robotic arm 40 includes a respective computer 21, 31, 41. The computers 21 , 31 , 41 are interconnected to each other using any suitable communication network based on wired or wireless communication protocols. The term “network,” whether plural or singular, as used herein, denotes a data network, including, but not limited to, the Internet, Intranet, a wide area network, or a local area network, and without limitation as to the full scope of the definition of communication networks as encompassed by the present disclosure. Suitable protocols include, but are not limited to, transmission control protocol/intemet protocol (TCP/IP), datagram protocol/intemet protocol (UDP/IP), and/or datagram congestion control protocol (DC). Wireless communication may be achieved via one or more wireless configurations, e.g., radio frequency, optical, Wi-Fi, Bluetooth (an open wireless protocol for exchanging data over short distances, using short length radio waves, from fixed and mobile devices, creating personal area networks (PANs), ZigBee® (a specification for a suite of high level communication protocols using small, low-power digital radios based on the IEEE 122.15.4-1203 standard for wireless personal area networks (WPANs)).

[0040] The computers 21, 31, 41 may include any suitable processor (not shown) operably connected to a memory (not shown), which may include one or more of volatile, non-volatile, magnetic, optical, or electrical media, such as read-only memory (ROM), random access memory (RAM), electrically-erasable programmable ROM (EEPROM), non-volatile RAM (NVRAM), or flash memory. The processor may be any suitable processor (e.g., control circuit) adapted to perform the operations, calculations, and/or set of instructions described in the present disclosure including, but not limited to, a hardware processor, a field programmable gate array (FPGA), a digital signal processor (DSP), a central processing unit (CPU), a microprocessor, and combinations thereof. Those skilled in the art will appreciate that the processor may be substituted for by using any logic processor (e.g., control circuit) adapted to execute algorithms, calculations, and/or set of instructions described herein.

[0041] With reference to FIG. 2, each of the robotic arms 40 may include a plurality of links 42a, 42b, 42c, which are interconnected at joints 44a, 44b, 44c, respectively. Other configurations of links and joints may be utilized as known by those skilled in the art. The joint 44a is configured to secure the robotic arm 40 to the mobile cart 60 and defines a first longitudinal axis. With reference to FIG. 3, the mobile cart 60 includes a lift 67 and a setup arm 61, which provides a base for mounting of the robotic arm 40. The lift 67 allows for vertical movement of the setup arm 61. The mobile cart 60 also includes a screen 69 for displaying information pertaining to the robotic arm 40. In embodiments, the robotic arm 40 may include any type and/or number of joints. [0042] The setup arm 61 includes a first link 62a, a second link 62b, and a third link 62c, which provide for lateral maneuverability of the robotic arm 40. The links 62a, 62b, 62c are interconnected at joints 63a and 63b, each of which may include an actuator (not shown) for rotating the links 62b and 62b relative to each other and the link 62c. In particular, the links 62a, 62b, 62c are movable in their corresponding lateral planes that are parallel to each other, thereby allowing for extension of the robotic arm 40 relative to the patient (e.g., surgical table). In embodiments, the robotic arm 40 may be coupled to the surgical table (not shown). The setup arm 61 includes controls 65 for adjusting movement of the links 62a, 62b, 62c as well as the lift 67. In embodiments, the setup arm 61 may include any type and/or number of joints. [0043] The third link 62c may include a rotatable base 64 having two degrees of freedom. In particular, the rotatable base 64 includes a first actuator 64a and a second actuator 64b. The first actuator 64a is rotatable about a first stationary arm axis which is perpendicular to a plane defined by the third link 62c and the second actuator 64b is rotatable about a second stationary arm axis which is transverse to the first stationary arm axis. The first and second actuators 64a and 64b allow for full three-dimensional orientation of the robotic arm 40.

[0044] The actuator 48b of the j oint 44b is coupled to the j oint 44c via the belt 45a, and the joint 44c is in turn coupled to the joint 46b via the belt 45b. Joint 44c may include a transfer case coupling the belts 45a and 45b, such that the actuator 48b is configured to rotate each of the links 42b, 42c and a holder 46 relative to each other. More specifically, links 42b, 42c, and the holder 46 are passively coupled to the actuator 48b which enforces rotation about a pivot point “P” which lies at an intersection of the first axis defined by the link 42a and the second axis defined by the holder 46. In other words, the pivot point “P” is a remote center of motion (RCM) for the robotic arm 40. Thus, the actuator 48b controls the angle 0 between the first and second axes allowing for orientation of the surgical instrument 50. Due to the interlinking of the links 42a, 42b, 42c, and the holder 46 via the belts 45a and 45b, the angles between the links 42a, 42b, 42c, and the holder 46 are also adjusted in order to achieve the desired angle 0. In embodiments, some or all of the joints 44a, 44b, 44c may include an actuator to obviate the need for mechanical linkages.

[0045] The joints 44a and 44b include an actuator 48a and 48b configured to drive the joints 44a, 44b, 44c relative to each other through a series of belts 45a and 45b or other mechanical linkages such as a drive rod, a cable, or a lever and the like. In particular, the actuator 48a is configured to rotate the robotic arm 40 about a longitudinal axis defined by the link 42a. [0046] With reference to FIG. 2, the holder 46 defines a second longitudinal axis and configured to receive an instrument drive unit (IDU) 52 (FIG. 1). The IDU 52 is configured to couple to an actuation mechanism of the surgical instrument 50 and/or the camera 51 and is configured to move (e.g., rotate) and actuate the instrument 50 and/or the camera 51. IDU 52 transfers actuation forces from its actuators to the surgical instrument 50 to actuate components an end effector 49 of the surgical instrument 50. The holder 46 includes a sliding mechanism 46a, which is configured to move the IDU 52 along the second longitudinal axis defined by the holder 46. The holder 46 also includes a joint 46b, which rotates the holder 46 relative to the link 42c. During endoscopic procedures, the instrument 50 may be inserted through an endoscopic access port 55 (FIG. 3) held by the holder 46. The holder 46 also includes a port latch 46c for securing the access port 55 to the holder 46 (FIG. 2).

[0047] The robotic arm 40 also includes a plurality of manual override buttons 53 (FIG. 1) disposed on the IDU 52 and the setup arm 61, which may be used in a manual mode. The user may press one or more of the buttons 53 to move the component associated with the button 53. [0048] With reference to FIG. 4, each of the computers 21, 31, 41 of the surgical robotic system 10 may include a plurality of controllers, which may be embodied in hardware and/or software . The computer 21 of the control tower 20 includes a controller 21a and safety observer 21b. The controller 21a receives data from the computer 31 of the surgeon console 30 about the current position and/or orientation of the hand controllers 38a and 38b and the state of the foot pedals 36 and other buttons. The controller 21a processes these input positions to determine desired drive commands for each joint of the robotic arm 40 and/or the IDU 52 and communicates these to the computer 41 of the robotic arm 40. The controller 21a also receives the actual joint angles measured by encoders of the actuators 48a and 48b and uses this information to determine force feedback commands that are transmitted back to the computer 31 of the surgeon console 30 to provide haptic feedback through the hand controllers 38a and 38b. The safety observer 21b performs validity checks on the data going into and out of the controller 21a and notifies a system fault handler if errors in the data transmission are detected to place the computer 21 and/or the surgical robotic system 10 into a safe state.

[0049] The controller 21a is coupled to a storage 22a, which may be non-transitory computer-readable medium configured to store any suitable computer data, such as software instructions executable by the controller 21a. The controller 21a also includes transitory memory 22b for loading instructions and other computer readable data during execution of the instructions . In embodiments, other controllers of the system 10 include similar configurations . [0050] The computer 41 includes a plurality of controllers, namely, a main cart controller 41a, a setup arm controller 41b, a robotic arm controller 41c, and an instrument drive unit (IDU) controller 41d. The main cart controller 41a receives and processes joint commands from the controller 21a of the computer 21 and communicates them to the setup arm controller 41b, the robotic arm controller 41c, and the IDU controller 4 Id. The main cart controller 41a also manages instrument exchanges and the overall state of the mobile cart 60, the robotic arm 40, and the IDU 52. The main cart controller 41a also communicates actual joint angles back to the controller 21a.

[0051] Each of joints 63a and 63b and the rotatable base 64 of the setup arm 61 are passive joints (i.e., no actuators are present therein) allowing for manual adjustment thereof by a user. The joints 63a and 63b and the rotatable base 64 include brakes that are disengaged by the user to configure the setup arm 61. The setup arm controller 4 lb monitors slippage of each of joints 63a and 63b and the rotatable base 64 of the setup arm 61, when brakes are engaged or can be freely moved by the operator when brakes are disengaged, but do not impact controls of other joints. The robotic arm controller 41c controls each joint 44a and 44b of the robotic arm 40 and calculates desired motor torques required for gravity compensation, friction compensation, and closed loop position control of the robotic arm 40. The robotic arm controller 41c calculates a movement command based on the calculated torque. The calculated motor commands are then communicated to one or more of the actuators 48a and 48b in the robotic arm 40. The actual joint positions are then transmitted by the actuators 48a and 48b back to the robotic arm controller 41c.

[0052] The IDU controller 41d receives desired joint angles for the surgical instrument 50, such as wrist and jaw angles, and computes desired currents for the motors in the IDU 52. The IDU controller 41d calculates actual angles based on the motor positions and transmits the actual angles back to the main cart controller 41a.

[0053] The robotic arm 40 is controlled in response to a pose of the hand controller controlling the robotic arm 40, e.g., the hand controller 38a, which is transformed into a desired pose of the robotic arm 40 through a hand eye transform function executed by the controller 21a. The hand eye function, as well as other functions described herein, is/are embodied in software executable by the controller 21a or any other suitable controller described herein. The pose of one of the hand controllers 38a may be embodied as a coordinate position and rollpitch-yaw (RPY) orientation relative to a coordinate reference frame, which is fixed to the surgeon console 30. The desired pose of the instrument 50 is relative to a fixed frame on the robotic arm 40. The pose of the hand controller 38a is then scaled by a scaling function executed by the controller 21a. In embodiments, the coordinate position may be scaled down and the orientation may be scaled up by the scaling function. In addition, the controller 21a may also execute a clutching function, which disengages the hand controller 38a from the robotic arm 40. In particular, the controller 21a stops transmitting movement commands from the hand controller 38a to the robotic arm 40 if certain movement limits or other thresholds are exceeded and in essence acts like a virtual clutch mechanism, e.g., limits mechanical input from effecting mechanical output.

[0054] The desired pose of the robotic arm 40 is based on the pose of the hand controller 38a and is then passed by an inverse kinematics function executed by the controller 2 la. The inverse kinematics function calculates angles for the joints 44a, 44b, 44c of the robotic arm 40 that achieve the scaled and adjusted pose input by the hand controller 38a. The calculated angles are then passed to the robotic arm controller 41c, which includes a joint axis controller having a proportional-derivative (PD) controller, the friction estimator module, the gravity compensator module, and a two-sided saturation block, which is configured to limit the commanded torque of the motors of the joints 44a, 44b, 44c.

[0055] With reference to FIG. 5, the surgical robotic system 10 is setup around a surgical table 90. The system 10 includes mobile carts 60a-d, which may be numbered “1” through “4.” During setup, each of the carts 60a-d are positioned around the surgical table 90. Position and orientation of the carts 60a-d depends on a plurality of factors, such as placement of a plurality of access ports 55a-d, which in turn, depends on the surgery being performed. Once the port placement is determined, the access ports 55a-d are inserted into the patient, and carts 60a-d are positioned to insert instruments 50 and the laparoscopic camera 51 into corresponding ports 55a-d.

[0056] During use, each of the robotic arms 40a-d is attached to one of the access ports 55a-d that is inserted into the patient by attaching the latch 46c (FIG. 2) to the access port 55 (FIG. 3). The IDU 52 is attached to the holder 46, followed by the SIM 43 being attached to a distal portion of the IDU 52. Thereafter, the instrument 50 is attached to the SIM 43. The instrument 50 is then inserted through the access port 55 by moving the IDU 52 along the holder 46. The SIM 43 includes a plurality of drive shafts configured to transmit rotation of individual motors of the IDU 52 to the instrument 50 thereby actuating the instrument 50. In addition, the SIM 43 provides a sterile barrier between the instrument 50 and the other components of robotic arm 40, including the IDU 52. The SIM 43 is also configured to secure a sterile drape (not shown) to the IDU 52. [0057] With reference to FIGS. 6-8, the first screen 32 of the surgeon console 30 is configured to display graphical user interfaces, for example GUI 600 (FIG. 6), GUI 700 (FIG. 7), and/or GUI 800 (FIG. 8), which provide a video feed of the camera 51 with assistive overlays. The video feed is within a field of view of the camera 51 and may show the surgical site, the instruments 50, etc. The video processing device 56 is configured to output the graphical user interfaces, which may be displayed on any of the screens of the system 10, e.g., the second screen 34 of the surgeon console 30, the screen 23 of the control tower 20, etc.

[0058] FIGS. 9-11 illustrate example flowcharts of methods for navigating an instrument to a target point within a surgical site based on the navigation of one or more other instruments. The methods and user interfaces disclosed may assist a clinician in achieving the indirect navigation of an instrument (e.g., anon-wristed instrument) to a target point, via the clinician’s direct navigation of one or more other instruments (e.g., wristed or non-wristed instruments) to the target point or to one or more other points within the surgical site. The methods may be embodied as software instructions stored in non-transitory medium (e.g., memory) of the video processing device 56 or in a memory of another component of the surgical robotic system 10 and may be executable by one or more processors (e.g., FPGA, CPU, GPU, etc.) of the video processing device 56 or any other component, processing device, and/or computing device of surgical robotic system 10.

[0059] With particular reference to FIGS. 6 and 9, in a first aspect, a method for navigating an instrument (e.g., a non-wristed surgical instrument) to a target within a surgical site is illustrated and described as method 900. The display aspects of a user interface associated with the steps of method 900 are illustrated, for example, in the GUI 600 of FIG. 6. Method 900 begins at step 901 where a first instrument 601 is navigated to a first point P 1 within the surgical site. The first instrument 601 may be a wristed instrument which is teleoparably navigated by a clinician via surgeon console 30. In step 903, the location of the first point Pl within the surgical site is detected, for example, via image analysis of the video feed and/or the navigational coordinates used to control movement of the first instrument 601. In some aspects, the second instrument 603 trails or lags behind movements of the first instrument 601, such that control of movement of the first instrument 601 by the user results in movement of the second instrument 603.

[0060] Using the detected location of the first point Pl, in step 905, a location of a second point P2 within the surgical site is calculated. The location of the second point P2 is the target point for navigation of the second instrument 603. For example, the location of the second point P2 may be offset along one or more axes from the location of the first point Pl by a distance D. The distance D and direction of the offset may be preconfigured or may be selected by the clinician user.

[0061] In step 907, a trajectory for navigating the second instrument 603 to the second point P2 is calculated and in step 909, a representation of the trajectory for navigating the second instrument 603 to the second point P2 is displayed as an overlay 613 on the GUI 600. The overlay 613 representing the trajectory for navigating the second instrument 603 to the second point P2 may be displayed as a synthetic animation of an image of the second instrument 603 moving along the trajectory (e.g., in the direction of arrow “A”) toward the second point P2. In one aspect, the overlay 613 representing the trajectory for navigating the second instrument 603 to the second point P2 is displayed as highlight or coloring, similar to indocyanine green (ICG) imaging, over areas of the live image feed.

[0062] With the GUI 600 displaying the overlay 613 representing the trajectory for navigating the second instrument 603 to the second point P2, in step 911, a determination may be made (e.g., automatically via computational or image analysis and/or manually by the clinician user) as to whether the calculated trajectory is acceptable, for example, whether the second instrument 603 would collide with other objects in the surgical site while navigating along the traj ectory or whether the second instrument 603 will end up at the correct target. Step 911 may include displaying of a prompt on the GUI 600 or on another device for acceptance of the trajectory by a user clinician. If the calculated trajectory is determined to be acceptable (YES in step 911), then method 900 proceeds to step 913 where the second instrument 603 is navigated along the trajectory to the second point P2. In one aspect, step 913 is only carried out upon receipt of confirmation or approval by a clinician user, for example, via the actuation of a confirmation input 346. However, if it is determined that the calculated trajectory is not acceptable (NO in step 911), then method 900 may revert to step 901 where the first instrument 601 may be moved to a different first point P 1 and/ or the clinician may select a new first point Pl or second point P2 (step 912) directly on the GUI 600 (e.g., via a touch input on screen 32 or via another input device).

[0063] The navigation of step 913 is carried out automatically by the components of the surgical robotic system 10. In particular, the navigation of the second instrument 603 along the calculated trajectory, by the robotic arm 40 to which the second instrument 603 is coupled, is automatically carried out without the need for a clinician to manually control the movements of the robotic arm 40 via the hand controllers 38a and 38b. In an aspect, controller 2 la receives data from the computer 31 of the surgeon console 30 about the first point Pl, the second point P2, and/or the calculated trajectory. The controller 2 la processes this data to determine desired drive commands for each joint of the robotic arm 40 and/or the IDU 52 and communicates these to the computer 41 of the robotic arm 40 to control navigation of the second instrument 603 along the calculated trajectory to the second point P2.

[0064] With particular reference to FIGS. 7 and 10, in a second aspect, a method for navigating an instrument (e.g., a non-wristed surgical instrument) to a target within a surgical site is illustrated and described as method 1000. The display aspects of a user interface associated with the steps of method 1000 are illustrated, for example, on the GUI 700 of FIG. 7. Method 1000 begins at step 1001 where a first instrument 701 is navigated to a target point PT within the surgical site. The first instrument 701 may be a wristed instrument which is teleoparably navigated by a clinician via surgeon console 30, that is, a wristed instrument which is more easily navigable to a target than a non-wristed instrument. In step 1003, the location of the target point PT within the surgical site is detected, for example, via image analysis of the video feed and/or the navigational coordinates used to control movements of the first instrument 701. In some aspects, the second instrument 703 trails or lags behind movements of the first instrument 701, such that control of movement of the first instrument 701 by the user results in movement of the second instrument 703.

[0065] In step 1005, a retraction trajectory is calculated for retracting the first instrument 701 from the target point PT, and in step 1007, a traj ectory for navigating the second instrument 703 to the target point PT is calculated. In step 1009, a representation of the retraction trajectory for retracting the first instrument 701 from the target point PT is displayed as an overlay 711 on the GUI 700 and in step 1011, a representation of the trajectory for navigating the second instrument 703 to the target point PT is displayed as an overlay 713 on the GUI 700. In an aspect, overlay 711 may be displayed as an animation of an image of the first instrument 701 moving along the retraction trajectory (e.g., in the direction of arrow “R”) from the target point PT. Uikewise, overlay 713 may be displayed as an animation of an image of the second instrument 703 moving along the trajectory (e.g., in the direction of arrow “A”) toward the target point PT. In one aspect, either or both of overlay 711 or overlay 713 may be displayed as a highlight or coloring, similar to indocyanine green (ICG) imaging, over the live image feed.

[0066] With the GUI 700 displaying the overlay 711 representing the retraction trajectory of retracting the first instrument 701 from the target point PT and the overlay 713 representing the trajectory for navigating the second instrument 703 to the target point PT, in step 1013, a determination may be made (e.g., automatically via computational or image analysis and/or manually by the clinician user) as to whether the calculated trajectories are acceptable, for example, whether either the first instrument 701 or the second instrument 703 would collide with other objects in the surgical site while navigating along the respective trajectory. Step 1013 may include displaying of a prompt on the GUI 700 or on another device for acceptance of the trajectories by a user clinician. If the calculated trajectories are determined to be acceptable (YES in step 1013), then method 1000 proceeds to step 1015 where the first instrument 701 is retracted along the retraction trajectory from the target point PT and the second instrument 703 is navigated along the trajectory to the target point PT. In an aspect, the advancement of the second instrument 703 toward the target point PT may begin before initiation of the retraction of the first instrument 701 and the retraction of the first instrument 701 may begin when the second instrument 703 is positioned within a predetermined distance of the first instrument 701.

[0067] In one aspect, step 1015 is only carried out upon receipt of confirmation or approval by a clinician user, for example, via the actuation of a confirmation input 347. However, if it is determined that either of the calculated trajectories is not acceptable (NO in step 1013), then method 1000 may revert to step 1001 where the first instrument 701 may be moved to a different target point PT and/or the clinician may select a new target point PT (step 1014) directly on the GUI 700 (e.g., via a touch input on screen 32 or via another input device).

[0068] The navigation of step 1015 is carried out automatically by the components of the surgical robotic system 10. In particular, the retraction of the first instrument 701 along the calculated retraction trajectory, by the robotic arm 40 to which the first instrument 701 is coupled, and/or the navigation of the second instrument 703 along the calculated trajectory, by the robotic arm 40 to which the second instrument 703 is coupled, may be automatically carried out without the need for a clinician to manually control the movements of the robotic arm 40 via the hand controllers 38a and 38b. In an aspect, controller 21a receives data from the computer 31 of the surgeon console 30 about the target point PT, and/or the calculated trajectories. The controller 21a processes this data to determine desired drive commands for each joint of the robotic arm 40 and/or the IDU 52 and communicates these to the computer 41 of the robotic arm 40 to which the first instrument 701 is coupled to control retraction of the first instrument 701 along the retraction trajectory from the target point PT. Additionally, or alternatively, the controller 2 la may process the data to determine desired drive commands for each joint of the robotic arm 40 and/or the IDU 52 and communicates these to the computer 41 of the robotic arm 40 to which the second instrument 703 is coupled to control advancement of the second instrument 703 along the trajectory to the target point PT. [0069] With particular reference to FIGS. 8 and 11, in a third aspect, a method for navigating an instrument (e.g., a non-wristed surgical instrument) to a target within a surgical site is illustrated and described as method 1100. The display aspects of a user interface associated with the steps of method 1100 are illustrated, for example, on the GUI 800 of FIG. 8. Method 1100 begins at step 1101 where a first instrument 801 is navigated to a first point Pl within the surgical site. In step 1103, a second instrument 802 is navigated to a second point P2 within the surgical site. The first instrument 801 and/or the second instrument 802 may be wristed instruments which are teleoparably navigated by a clinician via surgeon console 30, that is, wristed instruments which are more easily navigable through a surgical site than non-wristed instruments. In some aspects, the third instrument 803 trails or lags behind movements of the first instrument 801 and/or the second instrument 802, such that control of movement of the first instrument 801 and/or second instrument 802 by the user results in movement of the third instrument 803.

[0070] In step 1105, the location of the first point Pl and the location of the second point P2 within the surgical site are detected, for example, via computational or image analysis of the video feed and/or the navigational coordinates used to control movement of the first instrument 801 and the second instrument 802.

[0071] Using the detected location of either or both of the first point P 1 or the second point P2, in step 1107, a location of a third point P3 within the surgical site is calculated. The location of the third point P3 is the target point to which the third instrument 803 will be navigated. For example, the location of the third point P3 may be offset along one or more axes from the location of the first point Pl by a distance DI and/or may be offset along one or more axes from the location of the second point P2 by a distance D2. The distance DI and the distance D2, and respective direction of the offset, may be preconfigured or may be selected by the clinician user. In aspects, the third point P3 is between the first point Pl and the second point P2.

[0072] In step 1109, a trajectory for navigating the third instrument 803 to the third point P3 is calculated and in step 1111, a representation of the trajectory for navigating the third instrument 803 to the third point P3 is displayed as an overlay 813 on the GUI 800. The overlay 813 representing the trajectory for navigating the third instrument 803 to the third point P3 may be displayed as an animation of an image of the third instrument 803 moving along the trajectory (e.g., in the direction of arrow “A”) toward the third point P3. In one aspect, the overlay 813 representing the trajectory for navigating the third instrument 803 to the third point P3 is displayed as a highlight or coloring, similar to indocyanine green (ICG) imaging, over the live image feed.

[0073] With the GUI 800 displaying the overlay 813 representing the trajectory for navigating the third instrument 803 to the third point P3, in step 1113, a determination may be made (e.g., automatically via computational or image analysis and/or manually by the clinician user) as to whether the calculated trajectory is acceptable, for example, whether the third instrument 803 would collide with other objects in the surgical site while navigating along the trajectory. Step 1113 may include displaying of a prompt on the GUI 800 or on another device for acceptance of the trajectory by a user clinician. If the calculated trajectory is determined to be acceptable (YES in step 1113), then method 1100 proceeds to step 1115 where the third instrument 803 is navigated along the trajectory to the third point P3. In one aspect, step 1115 is only carried out upon receipt of confirmation or approval by a clinician user, for example, via the actuation of a confirmation input 348. However, if it is determined that the calculated trajectory is not acceptable (NO in step 1113), then method 1100 may revert to one or both of step 1101 where the first instrument 801 may be moved to a different first point P 1 or step 1103 where the second instrument 802 may be moved to a different second point P2. Additionally, or alternatively, the clinician may select a new first point Pl, second point P2, and/or third point P3 (step 1114) directly on the GUI 800 (e.g., via a touch input on screen 32 or via another input device).

[0074] The navigation of step 1115 is carried out automatically by the components of the surgical robotic system 10. In particular, the navigation of the third instrument 803 along the calculated trajectory, by the robotic arm 40 to which the third instrument 603 is coupled, is automatically carried out without the need for a clinician to manually control the movements of the robotic arm 40 via the hand controllers 38a and 38b. In an aspect, controller 2 la receives data from the computer 31 of the surgeon console 30 about the first point Pl, the second point P2, the third point P3, and/or the calculated trajectory. The controller 21a processes this data to determine desired drive commands for each joint of the robotic arm 40 and/or the IDU 52 and communicates these to the computer 41 of the robotic arm 40 to control navigation of the third instrument 803 along the calculated trajectory to the third point P3.

[0075] It will be understood that various modifications may be made to the embodiments disclosed herein. Therefore, the above description should not be construed as limiting, but merely as exemplifications of various embodiments. Those skilled in the art will envision other modifications within the scope and spirit of the claims appended thereto.

Claims

WHAT IS CLAIMED IS:

1. A surgical robotic system comprising: a first surgical robotic arm including a wristed instrument; a second surgical robotic arm including a non-wristed instrument; and a surgeon console including a hand controller configured to receive user input to control navigation of the wristed instrument within a surgical site, the surgeon console including: a display configured to display a user interface; and a processor and memory storing instructions which, when executed by the processor, cause the processor to: detect a location of a first point within the surgical site, the location of the first point corresponding to a navigated position of the wristed instrument; calculate a location of a second point within the surgical site based on the first point; calculate a trajectory for navigating the non-wristed instrument to the second point; display an overlay of the trajectory for navigating the non-wristed instrument to the second point on the user interface; display a prompt for acceptance of the trajectory for navigating the non- wristed instrument to the second point on the user interface; and instruct the second robotic arm to move the non-wristed instrument to the second point along the trajectory for navigating the non-wristed instrument to the second point.

2. The surgical robotic system of claim 1, wherein the processor is configured to calculate the location of the second point based on an offset from the location of the first point.

3. The surgical robotic system of claim 2, wherein the offset is user-selectable.

4. The surgical robotic system of claim 1, wherein the location of the second point is the same as the location of the first point.

5. The surgical robotic system of claim 1, wherein the instructions, when executed by the processor, further cause the processor to: calculate a trajectory for retracting the wristed instrument from the first point; and display on the user interface an overlay of the trajectory for retracting the wristed instrument from the first point.

6. The surgical robotic system of claim 1, wherein the instructions, when executed by the processor, further cause the processor to: detect a location of a third point within the surgical site, the third point corresponding to a navigated position of a second wristed instrument; and calculate the location of the second point based on a first offset from the location of the first point and a second offset from the location of the third point.

7. The surgical robotic system of claim 6, wherein the location of the third point is between the first point and the second point.

8. The surgical robotic system of claim 1, wherein the processor is configured to display the overlay of the trajectory for navigating the non-wristed instrument to the second point on the user interface by displaying an animation of a representation of the non-wristed instrument moving along the trajectory.

9. A method for navigating to a target within a surgical site, the method comprising: detecting a location of a first point within the surgical site, the location of the first point corresponding to a navigated position of a first instrument; calculating a location of the target within the surgical site based on the first point; calculating a trajectory for navigating a second instrument to the target; displaying a prompt for acceptance of the trajectory for navigating the second instrument to the target; and instructing a robotic arm to move the second instrument to the target along the trajectory for navigating the second instrument to the target based on an acceptance of the prompt.

10. The method of claim 9, further comprising displaying an overlay of the trajectory for navigating the second instrument to the target on a user interface of a display.

11. The method of claim 10, wherein displaying the overlay of the trajectory for navigating the second instrument to the target includes displaying an animation of a representation of the second instrument moving along the trajectory.

12. The method of claim 9, wherein calculating a trajectory for navigating a second instrument to the target includes calculating the location of the target based on an offset from the location of the first point.

13. The method of claim 12, wherein the offset is user-selectable.

14. The method of claim 9, wherein the location of the target is the same as the location of the first point.

15. The method of claim 9, further comprising: calculating a trajectory for retracting the first instrument from the first point; and displaying an overlay of the trajectory for retracting the first instrument from the first point.

16. The method of claim 9, further comprising: detecting a location of a third point within the surgical site, the third point corresponding to a navigated position of a third instrument; and calculating the location of the target within the surgical site based on the first point includes calculating the location of the target based on a first offset from the location of the first point and a second offset from the location of the third point.

17. The method of claim 16, wherein the location of the target is between the first point and the third point.

18. A surgical robotic system comprising: a first instrument; a second instrument; a surgeon console including a hand controller configured to receive user input to control navigation of the first instrument within a surgical site, the surgeon console including: a display configured to display a user interface; and a processor and memory storing instructions which, when executed by the processor, cause the processor to: detect a location of a first point within the surgical site, the location of the first point corresponding to a navigated position of the first instrument; calculate a location of a target within the surgical site based on at least one of the location of the first point or an offset from the location of the first point; calculate a trajectory for navigating the second instrument to the target; display an overlay of the trajectory for navigating the second instrument to the target on the user interface by displaying an animation of a representation of the second instrument moving along the trajectory; display a prompt for acceptance of the trajectory for navigating the second instrument to the target; and move the second instrument to the target along the trajectory for navigating the second instrument to the target if the prompt is accepted.

19. The surgical robotic system of claim 18, wherein the instructions, when executed by the processor, further cause the processor to: calculate a trajectory for retracting the first instrument from the first point; and display an overlay of the trajectory for retracting the first instrument from the first point.

20. The surgical robotic system of claim 19, wherein the instructions, when executed by the processor, further cause the processor to determine whether the traj ectory for retracting the first instrument from the first point is acceptable, and the processor is configured to cause the second instrument to navigate to the target based on the trajectory for navigating the second instrument to the target if the prompt is accepted and it is determined that the trajectory for retracting the first instrument from the first point is acceptable.