EP4642378A1

EP4642378A1 - Systems and methods for guided tool change resiliency

Info

Publication number: EP4642378A1
Application number: EP23848662.5A
Authority: EP
Inventors: Brandon D. Itkowitz
Original assignee: Intuitive Surgical Operations Inc
Current assignee: Intuitive Surgical Operations Inc
Priority date: 2022-12-29
Filing date: 2023-12-28
Publication date: 2025-11-05
Also published as: WO2024145414A1; CN120435263A; JP2026502809A; KR20250132469A

Abstract

A medical system may comprise a manipulator assembly and a control system. The control system includes a processing unit. The processing unit determines kinematic information associated with a first tool inserted into a worksite. The first tool is coupled to the manipulator assembly. The processing unit also received image data generated by an endoscopic imaging instrument having a field of view. The image data is generated by the endoscopic imaging instrument after the first tool is removed from the manipulator assembly. The processing unit also determines one or more guided tool change parameters for guiding a second tool into the worksite. The second tool is received into connection with the manipulator assembly after the first tool is removed. The guided tool change parameters are based on the kinematic information associated with the first tool coupled to the manipulator assembly and based on a depth map determined from the image data.

Description

SYSTEMS AND METHODS FOR GUIDED TOOL CHANGE RESILIENCY

CROSS-REFERENCED APPLICATIONS

[0001] This application claims priority to and benefit of U.S. Provisional Application No. 63/477,583 filed December 29, 2022 and entitled “Systems and Methods for Guided Tool Change Resiliency,” which is incorporated by reference herein in its entirety.

FIELD

[0002] Examples described herein relate to systems and methods for using endoscopic image data, including depth mapping or vision-based tool tracking, in robot-assisted tool change procedures.

BACKGROUND

[0003] Minimally invasive medical techniques are intended to reduce the amount of tissue that is damaged during medical procedures, thereby reducing patient recovery time, discomfort, and harmful side effects. Such minimally invasive techniques may be performed through natural orifices in a patient anatomy or through one or more surgical incisions. Through these natural orifices or incisions, an operator may insert minimally invasive medical tools to reach a target tissue location. Minimally invasive medical tools include instruments such as therapeutic, diagnostic, biopsy, and surgical instruments. Minimally invasive medical tools may also include imaging instruments such as endoscopic instruments that provide a user with a field of view within the patient anatomy. Some minimally invasive medical tools and imaging instruments may be robot-assisted or otherwise computer-assisted. Medical procedures may employ a number of different medical tools. When a different tool is desired during the medical procedure, the medical tools may be withdrawn from the surgical site so that it can be removed from its associated manipulator and replaced with a different tool. The new tool is then inserted into the surgical site. Improved systems and methods are needed to improve the reliability and availability of controlsystem assisted tool change procedures. SUMMARY

[0004] The following presents a simplified summary of various examples described herein and is not intended to identify key or critical elements or to delineate the scope of the claims.

[0005] Consistent with some examples, a medical system may comprise a manipulator assembly and a control system. The control system includes a processing unit. The processing unit determines kinematic information associated with a first tool inserted into a worksite. The first tool is coupled to the manipulator assembly. The processing unit also received image data generated by an endoscopic imaging instrument having a field of view. The image data is generated by the endoscopic imaging instrument after the first tool is removed from the manipulator assembly. The processing unit also determines one or more guided tool change parameters for guiding a second tool into the worksite. The second tool is received into connection with the manipulator assembly after the first tool is removed. The guided tool change parameters are based on the kinematic information associated with the first tool coupled to the manipulator assembly and based on a depth map determined from the image data.

[0006] It is to be understood that both the foregoing general description and the following detailed description are illustrative and explanatory in nature and are intended to provide an understanding of the present disclosure without limiting the scope of the present disclosure. In that regard, additional aspects, features, and advantages of the present disclosure will be apparent to one skilled in the art from the following detailed description.

BRIEF DESCRIPTIONS OF THE DRAWINGS

[0007] FIG. 1A provides a schematic illustration of a first medical tool and an imaging tool within a worksite, according to some examples.

[0008] FIG. IB provides a schematic illustration of a second medical tool and an imaging tool within a worksite during or after a tool exchange, according to some examples.

[0009] FIG. 2 is a schematic illustration of a medical system, according to some examples.

[0010] FIG. 3 is a flowchart illustrating a method for performing a guided tool change procedure, according to some examples.

[0011] FIG. 4A is a flow chart illustrating a method for generating guided tool change parameters for guiding a new tool, according to some examples.

[0012] FIG. 4B provides a schematic illustration of a secondary tool target position and insertion path selected to accommodate a manipulator range of motion, according to some examples.

[0013] FIGS. 5-6 are flowcharts illustrating methods for generating guided tool change parameters for guiding a new tool, according to some examples.

[0014] FIGS. 7A-7D illustrate a graphical user interface that display visual guidance for a guided tool change procedure.

[0015] Various examples described herein and their advantages are described in the detailed description that follows. It should be appreciated that like reference numerals are used to identify like elements illustrated in one or more of the figures for purposes of illustrating but not limiting the various examples described herein.

DETAILED DESCRIPTION

[0016] Guided tool change procedures may kinematically determine the position of a tool attached to a robot-assisted manipulator and may help deliver a replacement tool to the position of the first tool. Various conditions may cause a robot-assisted control system to invalidate a guided tool change or to permit a guided tool change when kinematically undetectable changes have occurred. Some systems and methods for guided tool change are described in U.S. Pat. No. 6,645,196 which is incorporated by reference herein, in its entirety. Sometimes the control system for the robot-assisted manipulator determines that the replacement tool cannot be located in the position of the first tool and invalidates or otherwise cancels a control-system guided tool change procedure, leaving medical staff to manually introduce the replacement tool and introducing workflow inefficiencies and inaccurate tool positioning. Various reasons may contribute to invalidation of a control-system guided tool change procedure. For example, tool tip inaccuracies may invalidate a control -system guided tool change procedure. Inaccuracies in kinematic chain evaluation of the position and/or orientation of the tip of a medical tool may be due to, for example, tool end effectors with long jaws or otherwise compliant joints with uncertain kinematic poses. Additionally or alternatively, a control-system guided tool change procedure may be invalidated due to limitations on the range of motion of the manipulator or if the manipulator encounters an obstruction during an attempted reorientation. Additionally or alternatively, a control -system guided tool change procedure may be invalidated if the manipulator is manually adjusted or even if manual intervention is perceived by the control system, such as when a clutch mechanism is engaged to initiate manual motion of the manipulator. Sometimes tissue movement, instrument deflections, and/or manipulator deflections may go kinematically undetected, leading the control system determine parameters for a control-system guided tool change using incomplete information. To provide a more robust control-system guided tool change procedure that minimizes invalidations, improves workflow efficiency, and allows for more robust availability, kinematic-based tool change inputs may be supplemented with image-based information to generate tool change parameters. More robust techniques for guided tool change may utilize endoscopic image data, including vision-based tool tracking and/or depth-mapping, to supplement the kinematic position determination to provide a revised insertion path, a revised insertion depth, and/or a revised tip position.

[0017] FIGS. 1A and IB illustrate a guided tool change operation for a medical instrument or tool 100 (e.g., a first tool) including a shaft 102, a clevis 103, and an end effector 104 having a distal end 106. The distal end 106 may be a tool tip of the end effector 104. The tool 100 may be inserted through an anatomic wall 107 into a worksite which may be an anatomic area 108 in a cavity of a patient's body via a port 109 of entry. An imaging instrument or tool 118 (e.g., an endoscope) having a field of view 119 may be inserted through the anatomic wall 107 into the anatomic area 108 via a port 111. The end effector 104 and anatomic tissue 105 may be within the field of view 119 of the imaging tool 118. The medical tool 100 and the imaging tool 118 may be manipulated from outside the anatomic 108 to have the particular configurations as shown in FIG. 1A. A robot-assisted manipulator (e.g. manipulator assembly 302) having one or more actuators may be used to manipulate the tools 100, 118.

[0018] Before removing the tool 100 from the anatomic area 108 and introducing another tool 110 through the port 109 as shown in FIG. IB, information may be gathered from one or more sensor systems (e.g. sensor system 308) including imaging systems (e.g., imaging tool 118) to record the operating configuration of the medical tool 100, including position, orientation, and/or pose. For example, the location of the distal end 106 may be kinematically determined using sensors such as potentiometers, encoders, or other types of position or velocity sensors used to determine the kinematic configuration of the manipulator assembly and any tools coupled thereto. The sensed location may be used as a target point 122 to calculate parameters for controlling a manipulator arm to manipulate the second tool 110 and guide its movement to an operating configuration, including a position, orientation and/or pose, in which a distal end 116 of the second tool 110 is located at the target point 122. In some examples, the same manipulator arm used to move the first tool 100 is used to manipulate the second tool 110 during and after a tool change operation. In other examples, parameters for controlling the manipulator arm during a guided tool change procedure may be determined from other types of sensors including imaging tools that may generate image data before or after the tool 100 is removed. For example, image data generated prior to removal of the tool 100 may provide information about the position and orientation of the distal end 106, including deflections or obstructions of the tool or manipulator undetectable by kinematic sensors. Image data generated after the removal of the tool 100 may provide information about the location of anatomic structures or other tools in the anatomic area that may have moved after the removal of the tool 100. In some examples, the image data may be stereoscopic image data that may be used to generate a depth map that provides information relating to the distance of the surfaces of objects in the field of view 119 from a distal end of the imaging tool 118. Depth mapping based on stereo image correlation may allow for the reconstruction of the topographic geometry of the anatomic environment and can allow the control system to determine the distance of a point in the anatomic area (e.g., a target insertion point) to the nearest surfaces visible in the field of view. A depth map may represent the perspective distance between an object in the field of view 119 and a plane of the imaging tool 118. In some examples, depth map pixels may have associated quality metrics based on the output of the stereo matching process (e.g. vertical error of closest match from epipolar plane). A quality threshold may be used to assess the integrity of any or all depth map pixels that are sampled for projections or raycasting. The generation of the depth map of the worksite is not limited to the use of data from endoscopic imaging instruments. The depth map may be generated using other modalities of sensor data (e.g., instead of or in combination with endoscopic imaging data). For example, the depth map may be generated using data from depth-sensing sensors (e.g., structured light sensors, time of flight (ToF) sensors, etc.), ultrasonic sensors, computed-tomography (CT), and the like. In some implementations, the depth map may be generated by a composite or multi-modal sensor (e.g., an integrated endoscopic imaging and depth-sensing sensor).

[0019] When the replacement tool 110 is introduced into the anatomic area 108, the recorded information of the operating configuration of the original tool 100 is used to provide guidance for the replacement tool 110 so that it can be readied for operation, including being located in the same position, orientation, and/or pose as the original tool 100. The replacement tool 110 (e.g., the second tool) may include a shaft 112 and an end effector 114 with a distal end 116. Using the recorded operating configuration information for the tool 100, the replacement tool 110 may be accurately positioned in the anatomic area in approximately the same location relative to the anatomic tissue 105, the field of view 119 of an imaging tool 118, and any other tools or structures that may be in the anatomic area 108.

[0020] In some examples, the guided tool change procedure may record the operating position of the distal end 106 of the first tool 100, which is then used as the target point 122 to determine the desired position of the distal end 116 of the second tool 110. In this way, the depth of insertion of the second tool 110 may be limited by the position of the distal end 116 to prevent extending the second tool 110 too far into the anatomic area 108 and causing undesired contact of the distal end 116 with anatomic area 108. In some examples, the second tool 110 may be introduced via a straight line insertion path to position the distal end 116 in the operating position. The insertion path is illustrated as an imaginary line 120 in FIG. IB which extends through a remote center or center for rotation 124 for a manipulator to which the tool 110 is coupled and intersects the target point 122 associated with the distal end 106 of the first tool 100 in the operating position. The remote center 124 may be the point about which the port 109 pivots relative to anatomic wall 107. The remote center 124 position may be fixed with respect to the wall 107 of the patient (e.g., in X, Y, Z Cartesian space). In some examples, the remote center 124 is a hardware constrained remote center of motion with the location of the remote center 124 set based on the configuration of the manipulator assembly. Additionally or alternatively, the remote center 124 may be a software constrained remote center of motion. The insertion path 120 may be used as an insertion axis guide for guiding the second tool 110 and may be referred as an in-out axis or IO axis. The IO axis may represent a degree of freedom of movement of the tool. For instance, the tool may be mounted to a carriage that is driven to translate along a linear guide formation of a robotic manipulator arm which is movable in additional degrees of freedom including angular displacements to position the tool. In other examples, the insertion path may be curvilinear or have any shape which allows the distal end 116 to reach the target point 122.

[0021] In some examples, the distal end 116 of the second tool 110 need not be positioned at precisely the target point 122. A target space 125 may be defined in the vicinity of the target point 122 to provide an acceptable region for positioning the distal end 116, corresponding e.g., to any position within the surgeon's field of view 119. In some examples, the target space may bound an offset or bias region of approximately 0.5cm from the target point. FIG. IB shows a cylindrical target space 125 defined by specifying an acceptable distance from the target point 122 in which to place the distal end 116. In some examples, the target space 125 may be generally biased toward the remote center 124 to reduce a likelihood of the second tool 110 contacting tissue during the insertion. In other examples, the target space may be spherical, a rectangular prism, a conical shape, a frustrum shape or any shape that promotes efficiency in tool change or to prevent extending the tool too far into the anatomic area. An insertion depth limit 126 may provide a depth limit along the insertion path 120 beyond which insertion of the tool 110 may be limited. The depth limit 126 may be imposed as visual guidance on a display, haptic and/or force feedback to an operator control device, haptic and/or force feedback provided via the second tool 110 and/or the manipulator assembly, a software imposed insertion stop, an audio cue, or any other type of guidance that limits the distance beyond which the end effector 116 of the second tool 110 may contact tissue. In some examples, the depth limit 126 may be biased away from obstructions based on depth map uncertainty and on manipulator remote center uncertainty in the endoscope tip frame of reference. In one example, haptic and/or force feedback relating to the depth limit 126 may be provided via the second tool 110 and/or the manipulator assembly during a tool insertion operation, which may be part of a guided tool change procedure, to insert the second tool 116. The tool insertion operation may be performed by a table-side. For instance, as the end effector 116 of second tool 110 approaches the depth limit 126 during tool insertion, the manipulator assembly (e.g., a prismatic or revolute joint on the manipulator assembly) may provide increasing force feedback to prevent the manual tool insertion to insert the end effector 116 of the second tool 110 beyond the depth limit 126.

[0022] In some examples, the components discussed above may be part of a computer-assisted or robotic-assisted system as described in further detail below. Such computer-assisted or robotic- assisted system may be suitable for use in, for example, surgical, robotic-assisted surgical, diagnostic, therapeutic, or biopsy procedures While some examples are provided herein with respect to such procedures, any reference to medical or surgical instruments and medical or surgical methods is non-limiting. The systems, instruments, and methods described herein may be used for animals, human cadavers, animal cadavers, portions of human or animal anatomy, non- surgical diagnosis, as well as for industrial systems and general robotic, general robotic-assisted, or robotic medical systems. [0023] As shown in FIG. 2, a medical system 300 generally includes a manipulator assembly 302 including manipulator arms 303, 304. The tool 100 may initially be coupled to and manipulated by the manipulator arm 303, and after a guided tool change procedure tool 110 may be coupled to and manipulated by the manipulator arm 303. The imaging tool 118 may be coupled to and manipulated by the manipulator arm 304. The tools 100, 110, 118 may be used in performing various procedures on a patient P positioned on a table T. In some examples the manipulator assembly 302 may include more or fewer than the two arms shown. The manipulator assembly 302 may be robotic-assisted, non-robotic-assisted, or a hybrid robotic-assisted and non-robotic- assisted assembly with select degrees of freedom of motion that may be motorized and/or robotic- assisted and select degrees of freedom of motion that may be non-motorized and/or non-robotic- assisted. The medical system 300 may further include input system 306, which generally includes one or more operator control devices for controlling manipulator assembly 302. Manipulator assembly 302 supports tools 100, 110, 118 and may optionally include a plurality of actuators or motors that drive inputs on the medical tools in response to commands from a control system 312. The actuators may optionally include drive systems that when coupled to medical tool 100, 110, 118 may advance medical tool into a naturally or surgically created anatomic orifice.

[0024] Medical system 300 also includes a display system 310 for displaying an image or representation of the surgical site and medical tools generated by the imaging tool 118 and/or a sensor system 308. Display system 310 and input system 306 may be oriented so operator O can control the tools (e.g., medical tools, non-medical tools, imaging instruments, etc.) and the manipulator assembly with the perception of telepresence. The display system 310 may include one or more display screens, including a display screen visible to a table-side operator tasked with interacting with the manipulator assembly and tools during an instrument exchange. Additional information regarding the medical system 300 and the medical tools may be found in International Application Publication No. WO 2018/195216, filed on April 18, 2018, entitled “Graphical User Interface for Monitoring an Image-Guided Procedure,” which is incorporated by reference herein in its entirety.

[0025] In some examples, imaging tool 118 may include components of an imaging system including an endoscopic imaging instrument assembly that records a concurrent or real-time image of a surgical site and provides the image to the operator or operator O through one or more displays of medical system 300, such as one or more displays of display system 310. The concurrent image may be, for example, a two or three-dimensional image captured by an imaging instrument positioned within the surgical site. The imaging system may be implemented as hardware, firmware, software or a combination thereof which interact with or are otherwise executed by one or more computer processors, which may include the processors of the control system 312.

[0026] The sensor system 308 may include a position/location sensors (e.g., an actuator encoder or an electromagnetic (EM) sensor system) and/or a shape sensors (e.g., an optical fiber shape sensor) for determining the position, orientation, speed, velocity, pose, and/or shape of the medical tools, the manipulator arms 303, 304, and/or components of the manipulator assembly 302. The sensor system 308 may also include pressure, force, or contact sensors or the like.

[0027] The control system 312 includes at least one memory 316 and at least one computer processor or processing unit 314 for effecting control between tool 100, 110, 118; input system 306; sensor system 308; and display system 310. Control system 312 also includes programmed instructions (e.g., a non-transitory machine-readable medium storing the instructions) to implement some or all of the methods described in accordance with aspects disclosed herein, including instructions for providing information to display system 310.

[0028] FIG. 3 is a flowchart illustrating a method 400 for performing a guided tool change procedure, according to some examples. The methods disclosed herein may be illustrated as sets of operations or processes that may be performed in the same or in a different order than the order shown. One or more of the illustrated processes may be omitted in some examples of the method. Additionally, one or more processes that are not expressly illustrated the flowcharts may be included before, after, in between, or as part of the illustrated processes. In some examples, one or more of the processes may be implemented, at least in part, by a control system executing code stored on non-transitory, tangible, machine-readable media that when run by one or more processors (e.g., the processors of a control system) may cause the one or more processors to perform one or more of the processes.

[0029] At a process 402, an indication of an intended tool change may be received, for example by a control system of a robot-assisted medical system. The indication generated, for example, by an operator pressing a button on a master assembly, an operator interacting with a graphical user interface, or the control system identifying a condition that corresponds to a tool change such as a change in operational mode or an initiation of a new sequence in the medical procedure. In some examples, the indication may be an indication to change the tool 100. [0030] At a process 404, kinematic information associated with a first tool’s position, orientation, and/or pose may be determined and/or recorded. In some examples, the kinematic information may be used to determine the target point 122 based on the kinematic chain that extends to the distal end 106 of the tool 100. In some examples, the process 404 may be performed before the process 402. The process 404 may determine and/or record any information about the state of the first tool and/or the manipulator assembly to which it is coupled as may be needed by later computations. Kinematic information may include the position, orientation, and/or pose of the structures and joints in the kinematic chain that includes the manipulator assembly and the tool to be exchanged. The kinematic information may be determined using sensors such as potentiometers, encoders, or other types of position or velocity sensors used to determine the kinematic configuration of the manipulator assembly and any tools coupled thereto. Kinematic information may also include the dimensions and shape of the links in the kinematic chain, including the components of the manipulator assembly and of the tool. Kinematic information may be determined and/or recorded with reference to a frame of reference such as the manipulator assembly, the patient, or the distal end of the imaging device used to capture an image of the field of view of the patient anatomy. In some examples, the kinematic information may be stored in the memory of a medical system (e.g. the memory 316).

[0031] At an optional process 406, image data (e.g., first image data) of the tool within the field of view of an imaging instrument (e.g., an endoscopic imaging instrument) may be received from the imaging instrument and/or recorded. For example, image data of the tool 100 within the field of view 119 of the imaging tool 118 may be received and/or recorded. In some examples, the image data may stereoscopic image data received from a stereoscopic endoscope. The stereoscopic image data may be used for vision-based tool tracking to triangulate the position of structures in the field of view, including the distal end 106, the target point 122, the clevis 103, the shaft 102, and/or the anatomic tissue 105. The image data may allow the control system to determine the position of the structures with respect to the endoscope tip, which may also be mapped to a common stationary frame of reference between the tool 100 and the tool 118.

[0032] At a process 408, guided tool change parameters for guiding a second tool may be determined. For example, guided tool change parameters may be determined for guiding tool 110 into the worksite (e.g., the anatomic area 108) through the port 109 after the tool 100 has been removed. The guided tool change parameters may be determined from the kinematic information, the image data, or other inputs as described in greater detail in the methods of FIGS. 4-6. Guided tool change parameters may include one or more of: a target position and orientation for the second tool tip, an insertion path for the second tool between the port and the target position, a location of the remote center for the manipulator arm to which the second tool is coupled, an insertion depth limit for the second tool along the insertion path, or a configuration of the manipulator assembly to generate the insertion path.

[0033] At a process 410 the determined guided tool change parameters may be used to perform the guided tool change procedure to introduce the second tool (e.g., tool 110) into the anatomic area. For example, the manipulator assembly 302 and/or manipulator arm 303 may be repositioned to adjust the guide trajectory for the new tool 110. The new tool 110 may be recognized and engaged at the carriage of the manipulator assembly 302. The new tool 110 will typically be different from the first tool 100, although they may be the same tool in some cases. In some examples data may be retrieved from the second tool 110 via a readable memory chip. Alternatively, kinematic information, including the dimensions and other data for the second tool 110 may be entered via an operator interface. The sensor system 308 may be used to detect engagement between the second tool 110 and the manipulator arm 303 to ensure proper engagement before the introduction process is performed. The manipulator assembly 302 (or one or more portions thereof) may be repositioned prior to engagement with the second tool 110. This adjustment may result in an adjustment to the I/O axis and/or the remote center. The tool 110 may be introduced into the surgical site 108 through the port of entry 109 along the insertion path 120. The control system 312 may direct the manipulator assembly 302 to float the degree of freedom of movement along the insertion path 120 to allow the second tool 110 to move into the worksite (e.g., anatomic area 108), either by a surgeon's assistant or by the controller itself causing the tool to move along the insertion path 120. The new tool 110 may be introduced into the anatomic area 108 until the tip reaches the target space 125 or target point 122. The introduction of the second tool 110 may be performed by an operator, either by direct manual operation, or using a joystick or other interface, or remotely by the operator via the input system 306 but may alternatively be at least partially or completely carried out by the control system 312. A graphic illustration of the insertion path 120, the target space 125, the target point 122, and/or an image of the field of view 119 may be displayed on the display system 310 to guide the operator in inserting the second tool 110. When the distal end of the tool 110 reaches the target space 125 or target point 122, operator control at the input system 306 may encounter a haptic resistance generated by the control system 312 to halt the movement of the tool 110.

[0034] The general introduction of a tool into a patient anatomy may require motion in one or more degrees of freedom. Some degrees of freedom during tool introduction may be constrained. In some guided tool change procedures, a single degree of freedom, such as the degree of freedom corresponding to motion along the insertion axis, is available, while all other degrees of freedom are constrained, particularly the other two proximal degrees of freedom of movement, used to position the tool mount before reinsertion, and the distal degrees of freedom of movement associated with the instrument's clevis and end effector. These degrees of freedom may be released for movement by the operator after operative connectivity between the master controls, the manipulator assembly, and the tool is reestablished, preferably after tool exchange is completed. The unconstrained degree of freedom may constitute a linear axis of movement coinciding exactly with one joint of a tool carriage on the manipulator arm. In general, to describe a guided tool change, some degrees of freedom of movement that are free to move are identified, with the remaining degrees of freedom of movement either absolutely or substantially constrained, via servo control or the like, so as not to move or to resist motion, respectively, in that direction. Entry of the tip of the second tool into the target space 125, as sensed by the sensor system 308, may indicate the end of the guided tool change. Transfer of control back to the operator is then provided, either automatically or by indicating that the operator can acquire control by taking some action, e g., by actuating an end effector or by pressing an input button. It will be appreciated that other criteria may also indicate the end of the guided tool change. These include the location of other points on the tool in other target spaces, the location of joints in some predefined or precomputed regions, other sensor inputs, for example, visual or image based detection of the tool, or explicit user input.

[0035] FIG. 4A illustrates a method 500 for generating one or more guided tool change parameters for guiding a new tool. The method 500 may be used as part of the process 408 of determining guided tool change parameters for guiding a second tool. At a process 502, guided tool change parameter inputs may be generated based on kinematic information associated with the first tool’s configuration, including position, pose, and/or orientation. The guided tool change parameter inputs may include any of the kinematic information recorded at process 404 such as the position, orientation, and/or pose of the structures and joints in the kinematic chain that includes the manipulator assembly and the tool to be exchanged. The guided tool change parameter inputs may also be derived from the kinematic information. Such inputs may include the target point 122 at the distal end 106 of the tool 100, the target space 125, the insertion depth limit 126, and/or the kinematically determined distances between operational tools, imaging tools, and anatomic structures.

[0036] At a process 504, guided tool change parameters for guiding a second tool may be generated from the guided tool change parameter inputs. Guided tool change parameters may include, for example, a target position and orientation for the second tool tip, an insertion path for the second tool between the port and the target position, a location of the remote center for the second tool, an insertion depth limit for the second tool along the insertion path, and a configuration of the manipulator assembly to generate the insertion path. For example the insertion path 120 for the new tool 110 may be determined from the guided tool change parameter inputs. The determination may include calculating the position of the tool carriage on the manipulator arm 303 which, when engaged with the new tool 110, will allow the new tool 110 to begin to be introduced into the anatomic area 108 along the specified path 120, so that the new tool path will intersect the target point 122 or the target space 125 previously occupied by the first tool's end effector. The determined insertion path may be used to operate the manipulator assembly 302 to reposition the tool carriage on the arm 303. The parameters may also include, for example, coordinates of specific points and joint angles of specific joints between linkages contained in the manipulator assembly 302 and the second tool 110. In a specific example, the parameters used to reposition the tool mount include outer pitch angle and outer yaw angle relative to a reference frame for guiding the tool 110 along the insertion path 120. The outer pitch and outer yaw for the second tool 110 may generally be different from the outer pitch and outer yaw for the first tool 100 in the operating position prior to its removal.

[0037] In some examples, the kinematically determined guided tool change parameters may be initial parameters such as an initial target position 122 or an initial insertion path 120 that may be modified, adjusted, corrected or otherwise changed in view of additional parameter inputs derived from the imaging system, as described in further examples below. In some examples, a guided tool change procedure may be invalidated if the generated guided tool change parameters violate other conditions of the guided tool change procedure. For example, if a guided tool change parameter includes an insertion path that exceeds a range of motion for the manipulator assembly (e.g. if motion exceeds a cone angle of 30 degrees) or if the manipulator arm encounters an obstruction when moving into alignment with the insertion path, the guided tool change may be invalidated.

[0038] FIG. 4B provides an example of secondary target point selected to avoid invalidating a guided tool change. An initially selected target point 122A and insertion path 120A, which may correspond to the target point 122 and insertion path 120 of FIG. 2B determined for the new tool may not be achievable due to a range of motion limitation for the manipulator am. To avoid invalidation of the guided tool change procedure, the target point 122A may be moved, for example, to a target point 122B at a location within the volume 150 of the first tool (e.g., the clevis or a location along the shaft of the first tool 100) along a new insertion path 120B that does not exceed the manipulator assembly range of motion. In some examples, the target point 122B may be identified as a distal most location along the initial tool assembly that satisfies the kinematic constraints of the subsequent insertion trajectory while still being within the endoscope’s field of view.

[0039] At an optional process 506, visual guidance may be displayed to assist an operator, such as a table-side operator, with performing the guided tool change. For example, visual guidance including graphic illustrations of the target point 122, the target space 125, the insertion path 120, arrows depicting a direction of motion, and/or the field of view 119 may be displayed on the display system 310 to assist with guidance of the new tool 110. Examples of visual guidance may be provided via graphical user interfaces such as the ones illustrated in FIGS. 7A-7D.

[0040] At an optional process 508, haptic guidance may be provided to assist an operator, such as a table-side operator, with performing the guided tool change. For example, haptic guidance in the form of a resistive force, a vibration, or other tactile sensation may be provided to the manipulator arm 303 so that the arm may snap to the insertion path as the insertion axis of the arm becomes aligned with the target position. For instance, as the end effector of the second tool approaches or reaches the target point and/or a depth limit during tool insertion by the table-side operator, the manipulator assembly (e.g., a prismatic or revolute joint on the manipulator assembly) may provide haptic and/or force feedback. The operator may, optionally, override the haptic force.

[0041] In some examples, a guided tool change procedure may be invalidated if the manipulator arm is clutched or moved by an operator away from the constrained pose during the instrument exchange. To avoid invalidation of the guided tool change procedure and to allow the operator to recover the use of the guided tool change procedure, the visual and/or haptic guidance may be provided. For example, when the control system becomes alerted to the clutch or motion, haptic or visual guidance may be provided to an operator to relocate the manipulator arm. The visual and haptic guides may allow the operator to optionally realign the arm with the guided tool change trajectory prior to resuming the instrument exchange procedure and prior to advancing the instrument into the body. The guidance may include a displayed location of the target position and the insertion path. The guidance may also be updated to include a virtual extension of the insertion path based on the moved manipulator configuration. An operator may compare the target position, the insertion path, and the virtual extension. Haptic guidance may be provided as the user moves the manipulator assembly to align the insertion axis of the manipulator arm with the insertion path. Accurately calibrated image and manipulator frames of reference may be needed to provide visual guidance as the manipulator assembly is moved.

[0042] FIG. 5 illustrates a method 600 for generating one or more guided tool change parameters for guiding a new tool. The method 600 may be used as part of the process 408 of determining guided tool change parameters for guiding a second tool. At a process 602, guided tool change parameter inputs may be generated based on the first image data recorded at process 406. In some examples, the first image data may be stereoscopic endoscope image data of the field of view 119 gathered while the tool 100 is within the anatomic area 108. The image data may be used to supplement or modify the kinematic information recorded at process 404. For some instruments or instrument configurations the kinematic information may not accurately indicate the true position and orientation of the distal tip of the instrument end effector. For example, long- jawed end effectors may be deflected by interaction with tissues or other structures in the anatomic area, causing the tip of the end effector to be several millimeters displaced from the expected kinematic position. Additionally or alternatively, backlash or drive train compliance issues with the control cables that articulate the end effector may cause the tip of the end effector to be displaced from the expected and commanded kinematic position. Additionally or alternatively, kinematic uncertainty may be associated with the joints and links of the manipulator assembly. The position and orientation of the end effector distal tip visible in the image data may be used to correct the kinematic information to provide a more accurate determination of the target point 122, the target space 125, and the insertion path 120. The image data frame of reference may be registered to a common frame of reference with the manipulator assembly and the tool 100 to correct the position and orientation in the common frame of reference. The image data from the field of view 119 may provide guided tool change parameter inputs, for example, in the form of tool and tissue position and orientation information that may be used to correct or modify the kinematic-derived parameter inputs.

[0043] At a process 604, guided tool change parameters for guiding a second tool may be generated from the guided tool change parameter inputs. Guided tool change parameters may include, for example, a target position and orientation for the second tool tip, an insertion path for the second tool between the port and the target position, an insertion depth limit for the second tool along the insertion path, and a configuration of the manipulator assembly to generate the insertion path. For example, guided tool change parameter inputs from the first image data may be combined with or used to modify the guided tool change parameter inputs from the kinematic information (process 502) to generate more accurate guided tool change parameters. For example, to more accurately determine the target position and orientation for the second tool tip, a more accurate target point 122 for the first tool may be determined. More specifically, an image-based position of the instrument distal end 106 may be determined in an image frame of reference. The position of the distal end 106 may be transformed from the image frame of reference to a manipulator assembly frame of reference. A comparison, in the manipulator assembly frame of reference, of the transformed image-based position of the distal end 106 with the kinematically determined target point 122 (corresponding to the kinematically determined position of the distal end 106) may identify inaccuracies in the kinematically determined position. A modified target point 122 may be determined based on the identified inaccuracy to generate a modified target position and orientation for the second tool tip. More specifically, in some examples, the distal tip position uncertainty associated with the kinematic based solution may be compared to the uncertainty associated with the image-based distal tip position. The system may conditionally use the estimate with the lowest acceptable uncertainty (e.g. a threshold or predetermined lowest uncertainty). If neither estimate has sufficiently low uncertainty, then guided tool change process may be cancelled. Other parameters including the insertion path and the configuration of the manipulator assembly may be adjusted based on the revised target point.

[0044] At an optional process 606, visual guidance may be displayed to assist an operator, such as a table-side operator, with performing the guided tool change. For example, visual guidance including graphic illustrations of the image-adjusted parameters including the adjusted target point 122, the adjusted target space 125, and/or the adjusted insertion path 120 may be displayed with an image of the field of view 119 on the display system 310 to assist with guidance of the new tool 110. In some examples, indicators may be displayed on the display system 310 to indicate an obscured target point. For example, if the determined target point location is not visible in the image data (e.g., due to a tissue or tool obstructing the view or because it is outside the field of view), then during the tool change procedure, a graphical or textual indicator may be displayed to indicate the location of the obscured or off-screen target point.

[0045] At an optional process 608, haptic guidance may be provided to assist an operator, such as a table-side operator, with performing the guided tool change. For example, haptic guidance in the form of a resistive force, a vibration, or other tactile sensation may be provided as described at process 508.

[0046] FIG. 6 illustrates a method 700 for generating one or more guided tool change parameters for guiding a new tool. The method 700 may be used as part of the process 408 of determining guided tool change parameters for guiding a second tool. At a process 702, image data (e.g., second image data) of the field of view of the endoscopic instrument may be recorded after the first tool is removed. For example, the image data of the anatomic area 108, the anatomic tissue 105 and any other tools or structures within the field of view 119 of the imaging tool 118 may be recorded after the tool 100 is removed and before another tool is inserted through the port 109. The second image data may capture displacement or movement of the tissue 105 or other tools and structures in the field of view 119 after the tool 100 is removed, providing information about insertion path obstructions that may impede the introduction of a subsequent tool. The second image data may be used to supplement or modify the kinematic information recorded at process 404 and/or the first image data recorded at process 406. The process 702 may be optional to the method 700.

[0047] At a process 704, an image depth map may be generated from the second image data. In some examples, the image data may be stereoscopic image data received from a stereoscopic endoscope and may be used to generate a depth map that provides information relating to the distance of the surfaces of objects in the field of view 119 from a distal end of the imaging tool 118. A depth map may represent the perspective distance between an object in the field of view 119 and a plane of the imaging tool 118. In some examples, mapping points and/or vectors between depth map image space and imaging tool tip coordinates may depend on a calibrated camera model (i.e. intrinsic and extrinsic parameters).

[0048] This invention also depends on being able to map the remote center position and insertion axis of the instrument manipulator into endoscope tip coordinates using the kinematic chains and common reference frames between the manipulators.

[0049] At a process 706 guided tool change parameter inputs may be generated based on the image depth map. The image depth map may be used to supplement or modify the kinematic information recorded at process 404 and/or the first image data recorded at process 406. Sometimes the anatomic area 108 may change after the removal of the first tool 100. For example, tissue in contact with the first tool 100 may move into the space vacated by the tool. Additionally or alternatively, deformable tissues may move or slide relative to other tissues and some tissue may move in response to respiration, cardiac motion, or blood flow. Guided tool change parameter inputs determined from the depth map may include, for example, a distance between the kinematically identified target point 122 and the surface of anatomic tissue 105 or a distance between the surface of anatomic tissue 105 and the anatomic wall 107 adjacent the imaging tool or adjacent the port 109. In some examples, guided tool change parameter inputs may include the location of an intersection of the kinematically derived insertion path 120 and the anatomic tissue 105. The proximity of the target point 122 and/or the insertion path 120 to depth map corresponding to the anatomic tissue 105, may determine whether the target position and insertion path remain valid, become invalidated, or may be adjusted.

[0050] At a process 708, the depth map may be used to generate indicators for an obscured target position if the target position remains valid and reachable by the second tool. For example, if the depth map indicates that the kinematically determined target point 122 is obscured by anatomic tissue 105, graphical markers, text, or other indicators of the location of the target position may be displayed.

[0051] At a process 710, guided tool change parameters may be generated for guiding the second tool. Guided tool change parameters may include, for example, a target position and orientation for the second tool tip, an insertion path for the second tool between the port and the target position, an insertion depth limit for the second tool along the insertion path, and a configuration of the manipulator assembly to generate the insertion path. For example, guided tool change parameter inputs based on the depth map may be combined with or used to modify the guided tool change parameter inputs from the kinematic information (process 502) and/or the image information (process 602) to generate more accurate guided tool change parameters.

[0052] In some examples, an insertion depth limit, as trimmed or limited by the intersection of the insertion path with the depth map, may be continuously computed during insertion of the second tool in order to account for motion of the anatomy such as respiratory or pulsatile motion. The insertion depth limit may be determined as the minimum depth along the insertion path observed over a period of time just prior to (e.g. within a threshold distance or projected arrival time to the target position) the final advancement of the instrument to the target position. The insertion depth map limit can be continuously refined until the new instrument tip has been advanced to a predetermined location, for example, a location at which the instrument tip occludes or the endoscopic view of the target.

[0053] In some examples, the tool 110 may first be modeled as a cylinder extending along the insertion path 120 and tested for intersection with the depth map corresponding to the anatomic tissue 105 in the anatomic area 108. The target point 122 may remain valid if the modeled cylinder does not intersect the depth map corresponding to the anatomic tissue 105. The remote center 124 may also be maintained at the kinematically determined location. In some examples, the diameter of the modeled cylinder may be larger than the diameter of the tool shaft to account for uncertainty of the manipulator remote center with respect to the endoscope tip frame of reference.

[0054] In some examples, if the kinematically-determined target point 122 has been invalidated due to, for example, tool tip kinematic location inaccuracies, limitations on the range of motion of the manipulator or manual adjustment of the manipulator assembly, a modified target position may be determined using the depth map. For example, a ray-casting procedure may cast ray segments along the kinematically-determined insertion path 120 to determine if the cast rays intersect the depth map before reaching the kinematically-determined target point 122. If the rays intersect the depth map, the target position may be considered to be occluded and the target position may be adjusted to a modified target position at or near the intersection point of the cast rays and the depth map.

[0055] In some examples, a target position may be determined or modified from the kinematically-determined target position by determining a trajectory between a remote center for a manipulator arm and a central point in the field of view captured by the second image data. One or more test rays may be cast along the trajectory to adjust the insertion depth and target position based on the location of interference with the depth map. Once an optimized target position is determined by the test rays, inverse arm kinematics may be used to determine a manipulator arm pose that has an insertion axis that intersects the optimized target position and to determine a corresponding insertion depth to reach the optimized target position. In some examples, the ray cast should emanate from an unoccluded point in space such as the distal end of the port 109, a distal tip of a cannula extending within the port, or the distal tip of the instrument. Such an emanation point may avoid premature intersections with depth surfaces contributed by the components (e.g., port, cannula, instrument) themselves. Additionally, the ray cast computation may be constrained to only consider one-sided depth map surface transitions. For example, if a ray originates inside of the depth map, then the initial transition from inside-to-outside could be ignored.

[0056] In some examples, a guided tool change procedure may be invalidated if the manipulator arm is clutched or moved by an operator. To avoid invalidation of the guided tool change procedure, the depth map may be referenced to determine whether the kinematically-determined target point 122 or the insertion path 120 is obstructed by anatomic tissue 105 or other structures in the anatomic area 108. If no obstructions are identified based on the depth map, the tool change procedure may proceed without invalidation. In some examples, the guided tool change procedure may only proceed if the adjustment of a manipulator pose is sufficiently small (e.g., below a threshold pose change from the initial pose) such that the insertion path is within a tolerance of the original insertion path.

[0057] In some examples, the depth map is used to configure the end effector of the second tool after the second tool is inserted into the anatomic area. For example, the depth map may be evaluated to determine the location of surrounding tissue and determine a jaw opening and/or wrist or clevis orientation of the second tool that may avoid impacting surrounding tissue.

[0058] At an optional process 712, visual guidance may be displayed to assist an operator with performing the guided tool change. For example, visual guidance including graphic illustrations of the image-adjusted parameters including a new or modified target point, a modified target space, and/or a modified insertion path 120 may be displayed with an image of the field of view 119 on the display system 310 to assist with guidance of the new tool 110.

[0059] FIG. 7A-7D illustrate a graphical user interface 800 that displays visual guidance including image-adjusted parameters. The graphical user interface 800 may be displayed, for example, on a display system (e.g. display system 310) of a medical system. FIG. 7A illustrates the graphical user interface 800 including an image of an imaging tool field of view 802 (e.g. field of view 119) including anatomic tissue 804 and a first tool 806. A distal end portion of the first tool 806 may be located at a position 808. As shown in FIG. 7B, after the first tool 806 is removed, a marker 810 may be displayed at a target position which, as described herein, may be determined based on kinematic information associated with the first tool 806. For instance, the target position may correspond to or may otherwise be determined based on the position 808 of the removed first tool 806. In some examples, the target position may be based on the most distal jaw tip position of the first tool just prior to removal. A modeled or synthetic cylinder 812 may provide guidance in the form of a graphical insertion path for a second tool to advance toward the marker 810. In some examples, as shown in FIG. 7C, the target position may be adjusted based on structures in the field of view. For example, the tissue 804 may move, including shifting, distending, bulging, or otherwise becoming displaced, to occlude the marked position 808. As described in method 700, depth map information may be used to identify the obstructing tissue and generate guided tool change parameters including a revised target position and a revised marker 814 may be displayed at the revised target position. The revised target position may be located near the tissue 804 without being obstructed by the tissue. A second tool 816 may be guided along the modeled cylinder 802 until a distal end portion reaches the revised target position. By delivering the second tool 816 to the revised target position, direct or penetrative contact with the tissue 804 may be avoided. In some examples, as shown in FIG. 7D, revision or alteration of the target location may be displayed in the graphical user interface to enhance the operator’s awareness of the shift in target position. For instance, a marker 818 corresponding to the original target location and the marker 814 corresponding to the revised target location may be displayed concurrently within the graphical user interface 800. A feature of the marker 818, such as color, texture, opacity or shape, may provide an indication that the original position is occluded by tissue and thus unreachable by the second tool 816. In addition, or as an alternative, the volume within the cylinder 812 that is bounded by the marker 818 and 814 may be displayed as being visually distinguished (e.g., in color, shading, transparency level, texture, etc.) from the rest of the cylinder 812 to provide additional visual cues of the change in the target position. In some examples, the orientation of the modeled cylinder 802 may remain constant and the revised target position 814 may be located along the longitudinal axis of the modeled cylinder 802 at a location proximal of the target position 808 (e.g., as marked by marker 810, 818). In other examples, the orientation of the modeled cylinder may be changed in response to the moved tissue, and the revised target position may be located along the revised longitudinal axis of the modeled cylinder. In some examples, the manipulator assembly (e.g. manipulator assembly 302) to which the second tool is attached may enforce the revised target position 814 and restrict movement of the second tool beyond the revised target position.

[0060] At an optional process 714, haptic guidance may be provided to assist an operator with performing the guided tool change. For example, haptic guidance in the form of a resistive force, a vibration, or other tactile sensation may be provided to the operator control device at the master assembly 306.

[0061] The singular forms “a”, “an”, and “the” are intended to include the plural forms as well, unless the context indicates otherwise. And the terms “comprises,” “comprising,” “includes,” “has,” and the like specify the presence of stated features, steps, operations, elements, and/or components but do not preclude the presence or addition of one or more other features, steps, operations, elements, components, and/or groups. Components described as coupled may be electrically or mechanically directly coupled, or they may be indirectly coupled via one or more intermediate components. The auxiliary verb “may” likewise implies that a feature, step, operation, element, or component is optional.

[0062] In the description, specific details have been set forth describing some embodiments. Numerous specific details are set forth in order to provide a thorough understanding of the embodiments. It will be apparent, however, to one skilled in the art that some embodiments may be practiced without some or all of these specific details. The specific embodiments disclosed herein are meant to be illustrative but not limiting. One skilled in the art may realize other elements that, although not specifically described here, are within the scope and the spirit of this disclosure. [0063] Elements described in detail with reference to one embodiment, implementation, or application optionally may be included, whenever practical, in other embodiments, implementations, or applications in which they are not specifically shown or described. For example, if an element is described in detail with reference to one embodiment and is not described with reference to a second embodiment, the element may nevertheless be claimed as included in the second embodiment. Thus, to avoid unnecessary repetition in the following description, one or more elements shown and described in association with one embodiment, implementation, or application may be incorporated into other embodiments, implementations, or aspects unless specifically described otherwise, unless the one or more elements would make an embodiment or implementation non-functional, or unless two or more of the elements provide conflicting functions. Not all the illustrated processes may be performed in all embodiments of the disclosed methods. Additionally, one or more processes that are not expressly illustrated in may be included before, after, in between, or as part of the illustrated processes. In some embodiments, one or more of the processes may be performed by a control system or may be implemented, at least in part, in the form of executable code stored on non-transitory, tangible, machine-readable media that when run by one or more processors may cause the one or more processors to perform one or more of the processes.

[0064] Any alterations and further modifications to the described devices, instruments, methods, and any further application of the principles of the present disclosure are fully contemplated as would normally occur to one skilled in the art to which the disclosure relates. In addition, dimensions provided herein are for specific examples and it is contemplated that different sizes, dimensions, and/or ratios may be utilized to implement the concepts of the present disclosure. To avoid needless descriptive repetition, one or more components or actions described in accordance with one illustrative embodiment can be used or omitted as applicable from other illustrative embodiments. For the sake of brevity, the numerous iterations of these combinations will not be described separately. For simplicity, in some instances the same reference numbers are used throughout the drawings to refer to the same or like parts.

[0065] The systems and methods described herein may be suited for procedures involving any of a variety of anatomic systems, including the lung, colon, the intestines, the stomach, the liver, the kidneys and kidney calices, the brain, the heart, the circulatory system including vasculature, and/or the like. While some embodiments are provided herein with respect to medical procedures, any reference to medical or surgical instruments and medical or surgical methods is non-limiting. For example, the instruments, systems, and methods described herein may be used for non-medical purposes including industrial uses, general robotic uses, and sensing or manipulating non-tissue work pieces. Other example applications involve cosmetic improvements, imaging of human or animal anatomy, gathering data from human or animal anatomy, and training medical or nonmedical personnel. Additional example applications include use for procedures on tissue removed from human or animal anatomies (without return to a human or animal anatomy) and performing procedures on human or animal cadavers. Further, these techniques can also be used for surgical and nonsurgical medical treatment or diagnosis procedures.

[0066] One or more elements in embodiments of this disclosure may be implemented in software to execute on a processor of a computer system such as control processing system. When implemented in software, the elements of the embodiments of this disclosure may be code segments to perform various tasks. The program or code segments can be stored in a processor readable storage medium or device that may have been downloaded by way of a computer data signal embodied in a carrier wave over a transmission medium or a communication link. The processor readable storage device may include any medium that can store information including an optical medium, semiconductor medium, and/or magnetic medium. Processor readable storage device examples include an electronic circuit; a semiconductor device, a semiconductor memory device, a read only memory (ROM), a flash memory, an erasable programmable read only memory (EPROM); a floppy diskette, a CD-ROM, an optical disk, a hard disk, or other storage device. The code segments may be downloaded via computer networks such as the Internet, Intranet, etc. Any of a wide variety of centralized or distributed data processing architectures may be employed. Programmed instructions may be implemented as a number of separate programs or subroutines, or they may be integrated into a number of other aspects of the systems described herein. In some examples, the control system may support wireless communication protocols such as Bluetooth, Infrared Data Association (IrDA), HomeRF, IEEE 802.11, Digital Enhanced Cordless Telecommunications (DECT), ultra-wideband (UWB), ZigBee, and Wireless Telemetry.

[0067] Note that the processes and displays presented might not inherently be related to any particular computer or other apparatus. Various general-purpose systems may be used with programs in accordance with the teachings herein, or it may prove convenient to construct a more specialized apparatus to perform the operations described. The required structure for a variety of these systems will appear as elements in the claims. In addition, the embodiments of the invention are not described with reference to any particular programming language. It will be appreciated that a variety of programming languages may be used to implement the teachings of the invention as described herein.

[0068] This disclosure describes various instruments, portions of instruments, and anatomic structures in terms of their state in three-dimensional space. As used herein, the term position refers to the location of an object or a portion of an object in a three-dimensional space (e.g., three degrees of translational freedom along Cartesian x-, y-, and z-coordinates). As used herein, the term orientation refers to the rotational placement of an object or a portion of an object (e.g., in one or more degrees of rotational freedom such as roll, pitch, and/or yaw). As used herein, the term pose refers to the position of an object or a portion of an object in at least one degree of translational freedom and to the orientation of that object or portion of the object in at least one degree of rotational freedom (e.g., up to six total degrees of freedom). As used herein, the term shape refers to a set of poses, positions, or orientations measured along an object.

[0069] While certain illustrative embodiments of the invention have been described and shown in the accompanying drawings, it is to be understood that such embodiments are merely illustrative of and not restrictive on the broad invention, and that the embodiments of the invention not be limited to the specific constructions and arrangements shown and described, since various other modifications may occur to those ordinarily skilled in the art.

Claims

CLAIMS What is claimed is:

1. A medical system comprising: a manipulator assembly; and a control system, wherein the control system includes a processing unit including one or more processors, and wherein the processing unit is configured to: determine kinematic information associated with a first tool inserted into a worksite, wherein the first tool is coupled to the manipulator assembly; receive image data generated by an endoscopic imaging instrument having a field of view, the image data being generated by the endoscopic imaging instrument after the first tool is removed from the manipulator assembly; and determine one or more guided tool change parameters for guiding a second tool into the worksite, wherein the second tool is received into connection with the manipulator assembly after the first tool is removed and wherein the one or more guided tool change parameters are based on the kinematic information associated with the first tool coupled to the manipulator assembly and based on a depth map determined from the image data.

2. The medical system of claim 1, wherein the one or more guided tool change parameters include a target position for a tool tip of the second tool.

3. The medical system of claim 2, wherein the one or more guided tool change parameters include an insertion path for the second tool through a port of entry to the target position.

4. The medical system of claim 3, wherein the one or more guided tool change parameters include an insertion depth limit for the second tool along the insertion path.

5. The medical system of claim 3, wherein the one or more guided tool change parameters includes a configuration of the manipulator assembly that directs the second tool along the insertion path.

6. The medical system of claim 1, wherein the one or more guided tool change parameters includes a location of a remote center of motion for a manipulator arm of the manipulator assembly.

7. The medical system of claim 1, wherein determining one or more guided tool change parameters includes determining an initial target position for a tool tip of the second tool based on the kinematic information associated with the first tool and modifying the initial target position based on the depth map.

8. The medical system of claim 7, wherein modifying the initial target position includes determining a distance between the initial target position and an anatomic surface based on the depth map.

9. The medical system of claim 7, wherein modifying the initial target position includes determining if an initial insertion path from the manipulator assembly to the initial target position is obstructed based on the depth map.

10. The medical system of claim 9, wherein determining if the initial insertion path is obstructed includes modeling the initial insertion path as a cylinder and testing the cylinder for intersection with the depth map.

11. The medical system of claim 9, wherein a modified target position is at or near an intersection between the initial insertion path and the depth map.

12. The medical system of claim 7, wherein modifying the initial target position includes: determining a trajectory between a center of rotation for the manipulator assembly and a central point in a current endoscopic field of view, determining a modified target position at an insertion depth where the trajectory intersects the depth map, and determining a pose of the manipulator assembly that has an insertion axis that intersects the modified target position and based on the insertion depth.

13. The medical system of claim 7, further comprising: recording image data of the first tool in the field of view, wherein the initial target position for the tool tip of the second tool is further based on the image data of the first tool.

14. The medical system of claim 7, further comprising determining a configuration of an end effector of the second tool based on the depth map.

15. The medical system of claim 1, wherein the kinematic information includes a corrected remote center or in-out axis of the manipulator assembly.

16. The medical system of claim 1 further comprising: a display system configured to display a field of view of the endoscopic imaging instrument.

17. The medical system of claim 16, wherein the processing unit is further configured to: display visual guidance on the display system for conducting a guided tool change operation.

18. The medical system of claim 17, wherein the visual guidance includes graphical indicators to indicate a target position for second tool.

19. The medical system of claim 18, wherein the target position is obscured in the displayed field of view.

20. The medical system of claim 18, wherein the target position is adjusted based on structures in the field of view.

21. The medical system of claim 17, wherein the visual guidance includes a graphical depiction of an insertion path displayed on an image of the field of view.

22. The medical system of claim 17, wherein displaying the visual guidance includes concurrently displaying a first marker corresponding to an initial target position and a second marker corresponding to a modified target position.

23. The medical system of claim 1, wherein the processing unit is further configured to provide haptic guidance to an operator control device, wherein the manipulator assembly is responsive to movement of the operator control device.

24. A method comprising: determining kinematic information associated with a first tool inserted into a worksite, wherein the first tool is coupled to a manipulator assembly; receiving image data generated by an endoscopic imaging instrument having a field of view, the image data being generated by the endoscopic imaging instrument after the first tool is removed from the manipulator assembly; and determining one or more guided tool change parameters for guiding a second tool into the worksite, wherein the second tool is received into connection with the manipulator assembly after the first tool is removed and wherein the one or more guided tool change parameters are based on the kinematic information associated with the first tool coupled to the manipulator assembly and based on a depth map determined from the image data.

25. The method of claim 24, wherein the one or more guided tool change parameters include a target position for a tool tip of the second tool.

26. The method of claim 25, wherein the one or more guided tool change parameters include an insertion path for the second tool through a port of entry to the target position.

27. The method of claim 26, wherein the one or more guided tool change parameters include an insertion depth limit for the second tool along the insertion path.

28. The method of claim 26, wherein the one or more guided tool change parameters includes a configuration of the manipulator assembly that directs the second tool along the insertion path.

29. The method of claim 24, wherein the one or more guided tool change parameters includes a location of a remote center of motion for a manipulator arm of the manipulator assembly.

30. The method of claim 24, wherein determining one or more guided tool change parameters includes determining an initial target position for a tool tip of the second tool based on the kinematic information associated with the first tool and modifying the initial target position based on the depth map.

31. The method of claim 30, wherein modifying the initial target position includes determining a distance between the initial target position and an anatomic surface based on the depth map.

32. The method of claim 30, wherein modifying the initial target position includes determining if an initial insertion path from the manipulator assembly to the initial target position is obstructed based on the depth map.

33. The method of claim 32, wherein determining if the initial insertion path is obstructed includes modeling the initial insertion path as a cylinder and testing the cylinder for intersection with the depth map.

34. The method of claim 32, wherein a modified target position is at or near an intersection between the initial insertion path and the depth map.

35. The method of claim 30, wherein modifying the initial target position includes: determining a trajectory between a center of rotation for the manipulator assembly and a central point in a current endoscopic field of view, determining a modified target position at an insertion depth where the trajectory intersects the depth map, and determining a pose of the manipulator assembly that has an insertion axis that intersects the modified target position and based on the insertion depth.

36. The method of claim 30, further comprising: recording image data of the first tool in the field of view, wherein the initial target position for the tool tip of the second tool is further based on the image data of the first tool.

37. The method of claim 30, further comprising determining a configuration of an end effector of the second tool based on the depth map.

38. The method of claim 24, further comprising: displaying, on a display system, a field of view of the endoscopic imaging instrument and visual guidance for conducting a guided tool change operation.

39. The method of claim 38, wherein the visual guidance includes graphical indicators to indicate a target position for second tool.

40. The method of claim 39, wherein the target position is obscured in the displayed field of view.

41. The method of claim 39, wherein the target position is adjusted based on structures in the field of view.

42. The method of claim 38, wherein the visual guidance includes a graphical depiction of an insertion path displayed on an image of the field of view.

43. The method of claim 38, wherein displaying the visual guidance includes concurrently displaying a first marker corresponding to an initial target position and a second marker corresponding to a modified target position.

44. The method of claim 24, further comprising: providing haptic guidance to an operator control device, wherein the manipulator assembly is responsive to movement of the operator control device.

45. A non-transitory machine-readable media storing instructions that, when run by one or more processors, cause the one or more processors to: determine kinematic information associated with a first tool inserted into a worksite, wherein the first tool is coupled to a manipulator assembly; receive image data generated by an endoscopic imaging instrument having a field of view, the image data being generated by the endoscopic imaging instrument after the first tool is removed from the manipulator assembly; and determine one or more guided tool change parameters for guiding a second tool into the worksite, wherein the second tool is received into connection with the manipulator assembly after the first tool is removed and wherein the one or more guided tool change parameters are based on the kinematic information associated with the first tool coupled to the manipulator assembly and based on a depth map determined from the image data.

46. The non-transitory machine-readable media of claim 45, wherein the one or more guided tool change parameters include a target position for a tool tip of the second tool.

47. The non-transitory machine-readable media of claim 46, wherein the one or more guided tool change parameters include an insertion path for the second tool through a port of entry to the target position.

48. The non-transitory machine-readable media of claim 47, wherein the one or more guided tool change parameters include an insertion depth limit for the second tool along the insertion path.

49. The non-transitory machine-readable media of claim 47, wherein the one or more guided tool change parameters includes a configuration of the manipulator assembly that directs the second tool along the insertion path.

50. The non-transitory machine-readable media of claim 45, wherein the one or more guided tool change parameters includes a location of a remote center of motion for a manipulator arm of the manipulator assembly.

51. The non-transitory machine-readable media of claim 45, wherein determining one or more guided tool change parameters includes determining an initial target position for a tool tip of the second tool based on the kinematic information associated with the first tool and modifying the initial target position based on the depth map.

52. The non-transitory machine-readable media of claim 51, wherein modifying the initial target position includes determining a distance between the initial target position and an anatomic surface based on the depth map.

53. The non-transitory machine-readable media of claim 51, wherein modifying the initial target position includes determining if an initial insertion path from the manipulator assembly to the initial target position is obstructed based on the depth map.

54. The non-transitory machine-readable media of claim 53, wherein determining if the initial insertion path is obstructed includes modeling the initial insertion path as a cylinder and testing the cylinder for intersection with the depth map.

55. The non-transitory machine-readable media of claim 53, wherein a modified target position is at or near an intersection between the initial insertion path and the depth map.

56. The non-transitory machine-readable media of claim 51, wherein modifying the initial target position includes: determining a trajectory between a center of rotation for the manipulator assembly and a central point in a current endoscopic field of view, determining a modified target position at an insertion depth where the trajectory intersects the depth map, and determining a pose of the manipulator assembly that has an insertion axis that intersects the modified target position and based on the insertion depth.

57. The non-transitory machine-readable media of claim 51, storing instructions that, when run by one or more processors, further cause the one or more processors to: record image data of the first tool in the field of view, wherein the initial target position for the tool tip of the second tool is further based on the image data of the first tool.

58. The non-transitory machine-readable media of claim 51, storing instructions that, when run by one or more processors, further cause the one or more processors to: determine a configuration of an end effector of the second tool based on the depth map.

59. The non-transitory machine-readable media of claim 45, storing instructions that, when run by one or more processors, further cause the one or more processors to: display, on a display system, a field of view of the endoscopic imaging instrument and visual guidance for conducting a guided tool change operation.

60. The non-transitory machine-readable media of claim 59, wherein the visual guidance includes graphical indicators to indicate a target position for second tool.

61. The non-transitory machine-readable media of claim 60, wherein the target position is obscured in the displayed field of view.

62. The non-transitory machine-readable media of claim 60, wherein the target position is adjusted based on structures in the field of view.

63. The non-transitory machine-readable media of claim 59, wherein the visual guidance includes a graphical depiction of an insertion path displayed on an image of the field of view.

64. The non-transitory machine-readable media of claim 59, wherein displaying the visual guidance includes concurrently displaying a first marker corresponding to an initial target position and a second marker corresponding to a modified target position.

65. The non-transitory machine-readable media of claim 45, storing instructions that, when run by one or more processors, further cause the one or more processors to: provide haptic guidance to an operator control device, wherein the manipulator assembly is responsive to movement of the operator control device.