WO2009097461A1 - Apparatus and methods for automatically controlling an endoscope - Google Patents

Abstract

Systems for controlling a controllable elongate articulatable instrument, or endoscope, are provided which may include any number of features. One feature of the system is that it can determine the shape, position, and/or orientation of the instrument with the use of position and orientation sensors. Another feature of the system is a control system that receives a user command, data related to a keep-out region, and shape, position, and/or orientation information, and selects a commanded configuration to produce a controlled movement of the instrument. Methods associated with use of the controllable elongate articulatable instrument are also covered.

Description

APPARATUS AND METHODS FOR AUTOMATICALLY CONTROLLING AN ENDOSCOPE

CROSS REFERENCE TO RELATED APPLICATIONS [0001] This application claims the benefit under 35 U.S.C. 119 of U.S. Provisional Patent Application No. 61/024,463, filed January 29, 2008, titled "APPARATUS AND METHOD FOR AUTOMATICALLY CONTROLLING AN ARTICULATABLE ELONGATE DEVICE"; U.S. Provisional Patent Application No. 61/055,884, filed May 23, 2008, titled "APPARATUS AND METHOD FOR AUTOMATICALLY CONTROLLING AN ARTICULATABLE ELONGATE DEVICE"; U.S. Provisional Patent Application No. 61/074, 117, filed June 19, 2008, titled "APPARATUS AND METHOD FOR AUTOMATICALLY CONTROLLING AN ARTICULATABLE ELONGATE DEVICE"; and U.S. Provisional Patent Application No. 61/135,498, filed July 21, 2008, titled "TARGET IDENTIFICATION AND REGISTRATION IN SPACE". These applications are herein incorporated by reference in their entirety.

INCORPORATION BY REFERENCE

[0002] All publications and patent applications mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication or patent application was specifically and individually indicated to be incorporated by reference.

FIELD OF THE INVENTION

[0003] In the last two decades, the field of surgery has gone through major changes in the direction reduced invasiveness. Since the introduction of laparoscopy, an increasing number of surgical procedures in performed in this minimally invasive way. In the last few years, surgery took another step in the same direction by introducing single port surgery (SPA) and Natural Orifice Transluminal Endoscopic Surgery (NOTES) that ofer less or no scaring with potentially less pain.

BACKGROUND OF THE INVENTION [0004] There has been a steady progression in surgical procedures to reduce the difficulty for the surgeon and the recovery time required for the patient. Open surgical procedures have given way to laparoscopic and single port access (SPA) surgery. Laparoscopic procedures are evolving towards minimally invasive surgical procedures. [0005] While these advances are reducing the exterior incisions needed to access the internal organs, other procedures seek to remove external access and instead rely on the naturally occurring openings in the body to provide surgical access. Such procedures enter the body through a natural orifice and then create the surgical access within the body at the desired location. On such procedure is referred to as a natural orifice transluminal endoscopic surgery (NOTES).

[0006] While intra-abdominal and trans-luminal procedures have been suggested for several years, many problems remain unsolved or with sub-optimum solutions. Controllable instruments or endoscopes have been inadequate for the full spectrum of minimally-invasive surgical procedures due to insufficient flexibility, poor vision and illumination, and lack of stability to perform a surgical operation. Traditionally, controllable instruments for surgical procedures have included user steerable control of the tip of the instrument, but ignored any type of control for the trailing portions or segments. Thus, shortcomings exist in control systems and methods of controlling instruments to provide sufficient flexibility, manipulability, and control to perform surgery while also ensuring that trailing segments of the flexible instrument or endoscope do not harm surrounding body tissues and allow the desired tip control. Difficulties remain in providing controllable instruments and associated control systems that provide a physician with the confidence of operating an instrument within an open cavity in the body. [0007] In view of the ongoing challenges confronting the advancement of trans-luminal procedures, improvements are still needed. In particular, improvements are needed in the performance of position, orientation and movement of the entirety of a controllable instrument to free the physician from concern for the placement and movement of the controllable instrument, and in particular concern for the safe control of the trailing portions of the instrument.

SUMMARY OF THE INVENTION

[0008] In one embodiment of the present invention there is a position and orientation based control system for controlling a segmented instrument. The control system includes a controllable, segmented instrument having a distal segment; a position and orientation determination system adapted and configured to provide an output relating to the shape, position, and orientation of the distal segment; an actuation system coupled to the instrument that operates to bend the instrument; and a user interaction device. In addition, the control system is adapted and configured to receive a user command from the user interaction device, to receive data related to a keep-out region, to receive the output from the position and orientation determination system, and select a commanded configuration that produces a control output to the actuation system based on the commanded configuration to produce a controlled movement of the segmented instrument.

[0009] In one aspect of a position and orientation based control system the user command may also include a user configuration optimization preference. In another aspect, the control output is based at least in part on the configuration optimization preference. In another aspect, the user command from the user interaction device includes a commanded position. In another aspect, the user command from the user interaction device includes a commanded orientation. In another aspect, the user command from the user interaction device includes a commanded velocity. The commanded velocity may include a commanded velocity of a position of a portion of the instrument. In another aspect, the commanded velocity includes a commanded velocity of an orientation of a portion of the instrument. In still other aspects, the commanded velocity relates to a linear velocity, to an angular velocity or to a linear and an angular velocity of a portion of the instrument. The portion of the segmented instrument may be a proximal portion, a distal portion or a portion of the segmented instrument that is between a proximal portion and a distal portion of the segmented instrument.

[00010] In still other aspects, the control output is based at least in part on: the user command from the user interaction device, or a signal or data related to a keep out region. In one alternative, the control output is based on a comparison of a kinematic model of the controllable segmented instrument and the outputs from the first and second position and orientation sensors. In still another aspect, the control output is based on a comparison of a kinematic model of the controllable segmented instrument and the output from the position and orientation determination system. In another aspect, the position and orientation control system further adapted and configured to determine a number of possible configurations based on the received signals. [00011] In another alternative, the position and orientation control system selects the commanded configuration that does not violate the keep-out region. In still another aspect, the control output is based on one of the number of possible configurations. In another alternative, the control system applies a cost function term to select a configuration that emphasizes a preferred motion leading to the commanded configuration. In one aspect, the cost function term relates to minimizing joint velocities used to achieve a configuration of the controllable instrument that satisfies the received inputs. In another aspect, the cost function term relates to minimizing joint angles used to achieve a configuration of the controllable instrument that satisfies the received inputs. In one aspect, the cost function term relates to minimizing the articulation of a joint in the segmented instrument. In another aspect, the cost function term relates to increasing movement of the distal end of the segmented instrument over the movement of the proximal end of the segmented instrument. In another alternative, the cost function term relates to increasing movement of the proximal end of the segmented instrument over the movement of the distal end of the segmented instrument.

[00012] In another alternative, the data related to the keep-out region is provided using a position and orientation sensor on the instrument. In one aspect, the signal related to the keep out region is provided by a user. In one alternative, the signal or data related to the keep out region is provided by a device inserted through the instrument or from an external imaging modality in communication with the position and orientation control system, hi one aspect, the device is a mechanical device having a position and orientation sensor. In other aspects, the device utilizes a light signal or data or an ultrasound signal or data to indicate a portion of the keep out region. The light signal or data may be provided through the use of a laser. [00013] In still other aspects of the position and orientation based control system there is provided an apparatus to generate or sense an electromagnetic field. In another aspect, the actuation system includes a motor that may be connected to a tendon attached to the distal segment.

[00014] In another aspect, there is also provided a position and orientation based control system for controlling a segmented instrument. The system may include any of the above described aspects, features or alternatives. The system may also include a controllable, segmented instrument having a distal segment, a first position and orientation sensor on the distal segment that provides an output corresponding to a position and an orientation of a distal portion of the distal segment; and a second position and orientation sensor on the instrument that provides an output corresponding to a position and an orientation of a proximal portion of the distal segment. There is also an actuation system coupled to the instrument that operates to bend the instrument and a user interaction device. The control system is adapted and configured to receive a user command from the user interaction device, to receive data related to a keep-out region, to receive the outputs from the first and second position and orientation sensors, and to produce a control output to the actuation system to produce a controlled movement of the segmented instrument. [00015] In another embodiment of the invention, there is provided a method of method of performing surgery on a patient, comprising. The method may include the step of accessing an interior body portion of the patient with a controllable instrument. The method may also include the step of defining within the interior body portion at least one keep out region. The method may also include the step of navigating the controllable instrument to a surgical site within the body portion while preventing the controllable instrument within the body portion from intruding into the at least one keep out region. Additionally, the method may include the step of performing a surgical procedure at the surgical site within the body portion while preventing the controllable instrument within the body portion from intruding into the at least one keep out region. These steps may be performed in a different order than listed or may be repeated more than once depending upon the circumstances of a particular surgery. For example, the step of defining a keep out region may be repeated under circumstances indicating that the surgical field or the patient may have changed. If the surgical environment changes (i.e., loss or change in insufflation pressure), the patient moves or is moved or if the surgical platform or table is tilted or manipulated then the step of confirming the location of the keep out region(s) may be repeated. Additionally, the system using one or more sensors (such as position and orientation sensors located on the patient or other sensors such as one monitoring insufflation pressure) may provide a signal, alarm or warning indication to a user that patient or surgical environment has changed and that keep out regions or other system parameters should be checked, validated or reestablished. [00016] In another embodiment, the method includes insufflating at least a portion of the interior body portion before the defining step. In one aspect, the accessing step may also include introducing the controllable instrument through a natural orifice in the patient. In still another aspect, the accessing step may also include introducing the controllable instrument through a surgically created opening in the patient. In still another aspect, the accessing step may also include introducing the controllable instrument through a natural opening in the patient and thereafter through a surgically created opening.

[00017] In another embodiment of the method of performing surgery on a patient, accessing an interior body portion of the patient step includes accessing a portion of the peritoneal or retroperitoneal cavity. In addition, the method may include performing a surgical procedure on an organ accessed via the portion of the peritoneal or the retroperitoneal cavity. In one aspect, the interior body portion of the patient is a portion of the pelvic region accessed after the accessing step. In another aspect, the surgical procedure may also include performing a surgical procedure on an organ or tissue in the pelvic area. The organ or tissue in the pelvic area may relate to the prostate. The step of performing the surgical procedure step comprises a prostatectomy. [00018] In another aspect of the method of performing surgery on a patient, accessing an interior body portion of the patient step may include accessing a portion of the thoracic cavity. In other aspects, the method includes performing surgery on an organ, a tissue or a bone in the thoracic cavity. In still another aspect, the method includes performing surgery on a lung, a heart, a blood vessel, an esophagus, a trachea, a pleura, or a diaphragm. In still another aspect, the method includes performing surgery on an organ or tissue accessed via the thoracic cavity. [00019] In still another aspect of the method, navigating the controllable instrument step may also include steering the distal end of the controllable instrument based on a user input. In still another alternative embodiment, the step of navigating the controllable instrument step further comprising: controlling the distal end of the controllable instrument in accordance to the user input while controlling the movement of the proximal end of the controllable instrument so as to prevent the proximal end of the controllable instrument from intruding into the at least one keep out region. In yet another aspect, the navigating the controllable instrument step is performed so as to satisfy a user defined position, orientation or velocity preference. [00020] In still other aspects, the method of performing surgery on a patient where the accessing step, the navigating step or the performing step is performed using a command configuration that satisfies a cost function. The cost function may be any of those described herein. Examples of cost functions that maybe used include: minimized joint angles or segment angles of a portion of the instrument; maximum flexibility of a portion of the instrument, maximum maneuverability for a portion of the instrument controlled by a user interaction device, maximum maneuverability for axial movement into the body portion, and/or maximizing a distance from the keep out region. More than one cost function may be used to determine a command configuration and the same or different cost functions may be applied to the different segments or portions of a controllable instrument. [00021] In still another aspect of the method of performing surgery the performing a surgical procedure step may also include steering the distal end of the controllable instrument based on a user input. Additionally, the other aspects of the performing a surgical procedure step may include controlling the distal end of the controllable instrument in accordance to the user input while controlling the movement of the proximal end of the controllable instrument so as to prevent the proximal end of the controllable instrument from intruding into the at least one keep out region.

[00022] In still another aspect of the method of performing surgery on a patient, the step of performing a surgical procedure step is performed so as to satisfy a user defined position, orientation or velocity preference. In another aspect, the step of performing a surgical procedure step is performed so as to satisfy a cost function. [00023] In still other aspects of the method of performing surgery on a patient the performing a surgical procedure step may also include the step of: performing tissue manipulation. Tissue manipulation includes any known technique for manipulating tissue in the body including but not limited to excising tissue, cutting tissue, suturing tissue, stapling tissue, fastening tissue, cauterizing tissue, retracting tissue, expanding tissue, clipping tissue, approximating tissue, ablating tissue and repositioning tissue. Any of the actions described herein may be performed using an device surgical implement provided through one or more working channels of the segmented instrument or supported by the segmented instrument.

[00024] In still other aspects of the method of performing surgery on a patient, the performing a surgical procedure step includes any of a wide variety of surgical procedures. For example, the surgery may include performing a step related to the placement of diaphragmatic pacing leads, performing a peritoneoscopy, performing a bariatric surgery, performing surgery on the gastrointestinal tract, performing surgery on an organ of the gastrointestinal system, performing surgery on an organ of urinary system, performing surgery on one or more of a kidney, an ureter, a urinary bladder, or an urethra. Additionally or alternatively, the step of performing a surgical procedure may also include, for example, performing gynecological surgery, performing surgery within the peritoneal cavity, performing surgery within the thoracic cavity. [00025] In other aspects of the methods for performing surgery, the accessing, the defining and the navigating steps are performed so that the surgical site within the body portion includes an organ, a tissue, a vessel or a bone within the body portion. In still other aspects of the methods for performing surgery, the accessing, the defining and the navigating steps are performed so that the surgical site within the body portion includes an organ, a tissue, a vessel or a bone in or accessed via the thoracic cavity. In other embodiments, the accessing, the defining and the navigating steps are performed so that the surgical site within the body portion includes an organ, a tissue, a vessel or a bone in or accessed via the peritoneal or retroperitoneal cavity. In still another alternative, the accessing, the defining and the navigating steps are performed so that the surgical site within the body portion includes: a liver, a gall bladder, a stomach, an intestine, a heart, a lung, a kidney, a bladder, an artery, a vein, or a prostate. [00026] In still other aspects of the method, the at least one keep out region includes an organ, a tissue, a vessel or a bone in the peritoneal or retroperitoneal cavity. In another aspect, the at least one keep out region includes an organ, a tissue, a vessel or a bone in the thoracic cavity. In still another aspect, the at least one keep out region includes an organ, a tissue, a vessel or a bone in the interior body portion. It is to be appreciated that the systems and methods described herein may used to provide access to body portions to enable performance of a wide variety of surgical procedures including, but not limited to: peritonoscopy, intra-abdominal biopsy, gastric resection, colectomy, splenectomy, appendectomy, Nissen funduplication, gall bladder resection, diaphragmatic pacing, hernia repair, pancreatic pseudo cyst drainage, nephrectomy, tubal ligation, oophorectomy, gynecological procedure, gastric bypass surgery, and gastric banding. [00027] Another aspect of the invention includes a controllable, segmented instrument, comprising a distal segment having at least two links connected by a hinge and at least one tendon connected to the distal segment, whereby movement of the tendon connected to the distal segment causes bending of the distal segment about the at least one hinge. The instrument can also comprise a first position and orientation sensor on the distal segment that provides an output corresponding to a position and an orientation of a distal portion of the distal segment, and a second position and orientation sensor on the controllable segmented instrument that provides an output corresponding to a position and an orientation of a proximal portion of the distal segment. [00028] In one embodiment, the controllable, segmented instrument can further comprise a control system adapted and configured to receive the outputs from the first and second position and orientation sensors and determine the shape, position, and orientation of a portion of the segmented instrument. The portion of the instrument that the control system can determine the shape, position, and orientation of can be, for example, the distal segment, or alternatively, the portion of the instrument between the first and second position and orientation sensors. [00029] In another embodiment of the controllable, segmented instrument, the control system can send a control signal to an actuation system to move the at least one tendon to cause bending of the distal segment. [00030] In some embodiments, the control signal is based in part on a user command from a user interaction device.

[00031] In yet other embodiments, the first position and orientation sensor provides an output comprising a three coordinate position of a distal tip of the distal segment and at least one of a roll, a pitch or a yaw orientation of the distal tip of the distal segment, and the second position and orientation sensor provides an output comprising a three coordinate position of a proximal end of the distal segment and at least one of a roll, a pitch or a yaw orientation of the proximal end of the distal segment.

[00032] In another embodiment, the controllable segmented instrument can further comprise a second segment having at least two links connected by a hinge and at least one tendon connected to the second segment, whereby movement of the tendon connected to the second segment causes bending of the second segment about the at least one hinge, the second segment being proximal to the distal segment. In one embodiment, the second position and orientation sensor is on the second segment. In yet another embodiment, the controllable segmented instrument further comprises a third position and orientation sensor on the controllable segmented instrument that provides an output corresponding to a position and an orientation of a proximal portion of the second segment.

[00033] In one embodiment of the controllable segmented instrument the second segment can comprise a coil tube in the second segment containing the at least one tendon connected to the distal segment. [00034] In some embodiments, the first position and orientation sensor is attached to an outside surface of an outer wall of the distal segment. In an alternative embodiment, the first position and orientation sensor is attached to an interior surface of a wall of the distal segment. In yet another embodiment, the first position and orientation sensor is embedded in the wall of a link in the distal segment. In some embodiments, the first position and orientation sensor is attached to the distal segment with an adhesive. In other embodiments, the first position and orientation sensor is held on the distal segment using a plastic sheath.

[00035] In one embodiment of the controllable segmented instrument the tendon is a flexible cable. The tendon may be made from polyethylene, for example. [00036] In yet another embodiment of the controllable segmented instrument the first and the second position and orientation sensors are located in the same relative segment position. The relative segment position may be in the distal portion of the segment, for example. In another embodiment, the relative segment position is in the proximal portion of the segment. [00037] Another embodiment of the controllable segmented instrument further comprises third and fourth position and orientation sensors located in the same relative segment position.

BRIEF DESCRIPTION OF THE DRAWINGS

[00038] The novel features of the invention are set forth with particularity in the claims that follow. A better understanding of the features and advantages of the present invention will be obtained by reference to the following detailed description that sets forth illustrative embodiments, in which the principles of the invention are utilized, and the accompanying drawings of which.

[00039] In the drawings:

[00040] Fig. 1 is a schematic diagram of system for controlling a controllable elongate articulatable instrument.

[00041] Fig. 2 is a side view of distal segment of a controllable elongate articulatable instrument.

[00042] Figs. 3A-3B show side and cross sectional views of a link of a controllable elongate articulatable instrument.

[00043] Fig. 4A is a flow chart describing one method of calculating a commanded configuration of a controllable elongate articulatable instrument.

[00044] Fig. 4B illustrates an inverse Jacobian matrix of a controllable elongate articulatable instrument having n segments. [00045] Fig. 5 illustrates another embodiment of a system for controlling a controllable elongate articulatable instrument that further includes a device for measuring insertion depth and/or rotation.

[00046] Fig. 6 illustrates one embodiment of the system of Fig. 5 that further includes shape sensors for measuring the angles between adjacent links on the controllable elongate articulatable instrument.

[00047] Fig. 7 is a side view of distal segment of a controllable elongate articulatable instrument having shape sensors.

[00048] Fig. 8 is a diagram illustrating the flow of commands and feedback in a system incorporating joint angle feedback.

[00049] Fig. 9 is a schematic diagram of system for controlling a controllable elongate articulatable instrument with inputs from position and orientation sensors. [00050] Fig. 10 is a side view of distal segment of a controllable elongate articulatable instrument having position and orientation sensors.

[00051] Fig. 1 IA is a schematic diagram of system for controlling a controllable elongate articulatable instrument.

[00052] Fig. 1 IB is a schematic diagram of a system for determining the position and orientation of a controllable elongate articulatable instrument.

[00053] Figs. 12A-C are schematic diagrams of an embodiment of a control system for controlling a controllable elongate articulatable instrument with position and orientation sensors. [00054] Figs. 13A-B illustrate a keep-out region and a controllable elongate articulatable instrument operating within a keep-out region.

[00055] Fig. 14 illustrates a controllable elongate articulatable instrument being used to define a keep-out region.

[00056] Figs. 15 A-D illustrate defining a keep-out region with a device on a controllable elongate articulatable instrument.

[00057] Fig. 16 illustrates the use of external position and orientation sensors to define a keep- out region.

[00058] Fig. 17 is a flowchart illustrating a method of performing surgery on a patient using the controllable instrument or endoscope and the control system.

[00059] Figs. 18A-C illustrate various types of position and orientation determination systems that can be used to determine the shape, position, and/or orientation of a controllable elongate articulatable instrument.

[00060] Fig. 19 is a flowchart illustrating a method of performing surgery on a patient using the controllable instrument or endoscope and the control system. DETAILED DESCRIPTION OF THE INVENTION

[00061] Fig. 1 represents a system 100 for controlling a controllable elongate articulatable instrument, or endoscope 102. System 100 includes a user interaction device 104, which allows a user to input a user command, such as a commanded position, orientation, and/or velocity (angular and/or linear velocity) to the endoscope. System 100 further includes a control system 106 adapted to receive the user command from the user interaction device and direct the movement of the endoscope. In Fig. 1, the control system is in communication with an actuation system 108, which uses actuation motors to mechanically actuate and move the endoscope 102. More details on how the endoscope can be mechanically actuated and moved are further described below and in U.S. Ser. No. 12/036,976. System 100 further includes a display 112, such as a computer monitor, a television screen, or a graphical user interface (GUI), which may allow a user to input commands or system parameters to the system and review information relating to the endoscope, the surgical procedure, patient parameters, etc. [00062] The endoscope of system 100 further comprises a plurality of individually articulatable segments, such as distal segment 114a and proximal segments 114b-d. The embodiment illustrated in Fig. 1 comprises a total of four segments (i.e., distal segment 114a and three proximal segments 114b-d). It should be understood, however, that the endoscope can comprise any number of segments, depending on the desired length of the endoscope. The endoscope may be covered with a thin plastic webbing and an overall sheath, to ensure that the interior of the endoscope is sealed against ingress of physiological fluids and to protect the interior of the endoscope during decontamination and sterilization. [00063] Fig. 2 illustrates a side view of distal segment 214, which corresponds to distal segment 114a of system 100 in Fig. 1. The basic structure of all the segments (distal segment 114a and proximal segments 114b-d) in the endoscope is the same, with the exception of the most distal segment containing the tip, an optical and electronics payload, as well as the termination of the endoscopic surgical tools channels, which are not contained in the more proximal segments. The segment 214 is illustrated with the sheath removed, for ease of visualization. As shown in Fig. 2, distal segment 214 comprises a plurality of articulating vertebrae or links 218. Each link is attached to an adjacent link at a pair of hinges 220 to form a joint. Segment 214 (and thus endoscope 102 of system 100) can further include working channel(s) 230, which extend along the length of the endoscope. The working channels can provide access for air, surgical tools, water, suction, electrosurgical devices, and other small diameter devices which can assist in surgical procedures. [00064] Figs. 3A-3B show side and cross sectional views of a link 318, respectively. It can be seen that the proximal end of link 318 has a pair of hinges 320. The distal end of the link also has a pair of hinges, but the distal pair of hinges is positioned perpendicular to the proximal pair of hinges. This method of staggering pairs of hinges in perpendicular planes allows each link to move at the joints with one degree of freedom relative to an adjacent link and three links result in angular motion having two degrees of freedom. Thus, the first and second links in a segment can move along a first plane (i.e., left and right), while the second and third links in a segment can move along a second plane (i.e., up and down). As shown in Figs. 3A-3B, the proximal pair of hinges 320 can be flush with the outer wall of the link, and the distal pair of hinges can be slightly recessed from the outer wall of the link. This allows a distal pair of hinges of a first link to engage with a proximal pair of hinges of a second link and allow the surfaces to slide against each other. The adjacent link can then be articulatably coupled to one another at the hinges to form a joint, such as by inserting a pin, rivet, etc, through a hole in the hinges, for example. Alternatively, any method known in the art for hinging or forming joints between adjacent links or links can be used.

[00065] Referring back to Fig. 2, each segment of the endoscope 202 further includes a plurality of cables or tendons configured to actuate and move the endoscope segments. The tendons are preferably made from ultra high molecular weight polyethylene (UHMWP). In the embodiment illustrated in Fig. 2, each segment includes four tendons 222 (although only three tendons are shown for ease of illustration), each tendon running within the walls of the segment along an axis of each hinge. Coil tubes 224 each carry a tendon and terminate at the proximal portion of each segment. The coil tubes provide a protective and structure to house the tendons and provide column strength. After the tendons exit the coil tubes at the proximal terminus of the coil tubes, they continue towards the the distal portion of the segment, where they can be wound around pulley 226 and loop back towards the proximal portion of the segment. Pulleys 226 can be discrete mechanical pulleys, or can alternatively be integrally machined or formed in the walls of each segment. The tendons can then be attached to the segment on the proximal end using mechanical stops 228. More description of the tendons and pulleys can be found in U.S. Ser. No. 11/871,104, titled "SYSTEM FOR MANAGING BOWDEN CABLES IN ARTICULATING INSTRUMENTS", filed on October 11 , 2007, and U.S. Ser. No. 12/036,976, titled "Systems and Methods for Articulating an Elongate Body," filed on February 25, 2008. In an alternative embodiment, each segment does not include pulleys, but rather, the tendons can be fixed directly at the distal portion or end of each segment. [00066] In operation, pulling each of the respective tendons 222 will increase the tension in the pulled tendon and cause the segment to articulate in the direction of the pulled tendon as the links articulate at their respective joints. Additionally, when one tendon is tensioned, the opposite tendon in the segment needs to be slacked to accommodate movement of the segment, especially the tendon positioned 180 degrees or opposite from the tensioned tendon. For example, tensioning or pulling the tendon located near the top of the page in Fig. 2 will cause the distal end of the segment to bend in that direction (i.e., towards the top of the page in Fig. 2). This articulation requires that the opposite tendon near the bottom of the page in Fig. 2 be slacked to accommodate movement of the segment. Similarly, tensioning the tendon near the bottom of the page in Fig. 2 will cause the distal end of the segment to bend towards the bottom of the page in Fig. 2. This articulation requires that the opposite tendon near the top of the page in Fig. 2 be slacked to accommodate movement of the segment.

[00067] Applying these principles to the endoscope illustrated in Fig. 1 , it can be understood that if each segment contains four tendons, and the coil tubes terminate at the proximal end of each segment, then a four segment endoscope will require sixteen tendons and sixteen coil tubes. A first set of four tendons and coil tubes will attach to distal segment 114a, as described above in Reference to Fig. 2. It should be noted, however, that since these coil tubes terminate at the proximal end of the distal segment, the distal segment is the only segment in the endoscope that does not have coil tubes positioned within it. This set of coil tubes and tendons will also run through the three proximal segments 114b-d. Similarly, a second set of four tendons and coil tubes will attach to proximal segment 114b, as described above. This set will also run through the other two proximal segments 114c-d. The tendons and coil tubes attaching to proximal segments 114b-d will be connected in the same manner. Thus, in this example, a four segment endoscope as shown in Fig. 1 would have sixteen tendons and twelve coil tubes running through proximal segment 114d, twelve tendons and eight coil tubes running through proximal segment 114c, eight tendons and four coil tubes running through proximal segment 114b, and four tendons running through distal segment 114a. It should be understood that other quantities of coil pipes and tendons could be used in other embodiments, such as two, three, or more than four tendons/coil pipes per segment, for example.

[00068] The amount of tension applied to a particular tendon or tendons determines the angle of articulation of the endoscope segment (i.e., the distal end of the segment relative to the proximal end of the segment). Thus, the control system of the present invention employs a tendon force-to-articulate model to estimate the tendon tension necessary to achieve a particular segment articulation angle. This force-to-articulate model is used to "get close" to the correct tension for a particular segment angle to enable the control system to more quickly respond to an angle change command (also known as "feed-forward" control). [00069] Referring once again to Fig. 1 , a user, such as a physician or surgeon, can input a user command to the user interaction device 104, such as a commanded position (i.e., x, y, z), and/or orientation (i.e., yaw, pitch, or roll), or a velocity command (i.e., a rate-of-change command). The user command is the position, orientation, and/or velocity that the user desires a reference frame 101 of the endoscope to assume. The reference frame is typically positioned at the distal tip of the endoscope, however in other embodiments the reference frame can be positioned at any position along the endoscope. The user interaction device can be a joystick with up to six degrees of freedom (e.g., x, y, z, yaw, pitch, and roll in a Cartesian coordinate system). Alternatively, the user interaction device can be multiple joysticks, a mouse and/or keyboard, or any other type of electronic or mechanical interface configured to receive a user command from the user to control the movement of the endoscope. Additionally, the user interaction device may provide feedback to the user when being used to control the endoscope, such as haptic feedback or audio/visual feedback. [00070] System 100 further includes a control system 106 configured to direct the movement of the endoscope. The control system 106 is in communication with the user interaction device 104, from which it receives a user command corresponding to a commanded position, orientation, and/or velocity. When the control system receives the user command, it determines a commanded configuration needed to achieve the user command and provides a command to the actuation system to move to achieve the configuration or velocity. It should be understood that if the user command is a commanded position or orientation, then the commanded configuration from the control system will be a position and orientation in space, leaving the control system to determine the velocity of the individual segments to achieve the commanded position and orientation. However, if the user command is a commanded velocity, then the commanded configuration will be a series of configurations over multiple time-steps that approximate the velocity profile commanded by the user. However, for ease of description, both of these scenarios will be referred to as a "commanded configuration" herein.

[00071] For Fig. 1, the commanded configuration is determined with a feed-forward model, which consists of a combination of predictive models (e.g., cable, configuration dependent friction, force-to-articulate, kinematic, and gravity models) that estimate the combination of cable tensions and/or actuator positions necessary to achieve user command. The feed-forward model is used primarily to speed the response of the system, in essence a best guess at how to achieve the desired result. In another embodiment discussed below, any error in endoscope configuration remaining after the feed-forward model can then be absorbed using sensor feedback control. [00072] Actuation system 108 is in communication with control system 106 and is configured to receive a command from the control system to mechanically actuate the tendons within the endoscope and to provide feedback to the control system on the state of that motion. The actuation system can also include tension sensors to monitor the tension of each tendon and provide improved knowledge of tendon state. The command send to the actuation system can take the form of a motor voltage command.

[00073] Fig. 4A is a flow chart describing one method of calculating the commanded configuration the endoscope of Fig. 1 based on a user command. At step 30 of Fig. 4A, a Jacobian matrix is calculated. The terms of the Jacobian are calculated based on the kinematics of the endoscope, and comprise the local, linearized relationships between each of the endoscope's segment angle velocities and velocities of a particular reference frame or frames. Fig. 4B illustrates an inverse Jacobian matrix of an endoscope having n segments, according to step 30 of Fig. 4A. Thus, using the system of Fig. 1 as an example, n = 4. As illustrated in Fig. 4B, each of the n rows of the inverse Jacobian matrix contains the partial derivative of a segment angle velocity (x or y for a given segment) with respect to each of the reference frame's velocity terms (x, y, z, yaw, pitch, and roll, which are symbolized, respectively, as x , y , z , ά , β , and γ in Fig. 4B). Many methods can be used to calculate the Jacobian, as known in the art, including but not limited to a numerical calculation or an approximation with a closed-form analytical algorithm, for example. [00074] At step 32 of Fig. 4A, a user inputs a user command into the user interaction device. As described above, the user command directs the movement of the reference frame on the endoscope, which is typically a reference frame 101 positioned on the distal tip of the endoscope. Additionally, the user is not required to command all six degrees of freedom of the reference frame(s) but rather can leave them unconstrained and therefore free to be determined by the control system to obtain the best response to the commanded degree of freedom (e.g. in a one- segment scope, command a yaw rotation and accept whatever x translation results). [00075] At steps 34 and 36 of Fig. 4A, the control system estimates the new joint velocities or the new joint angles required to achieve the user command of the endoscope from step 32. This is done by multiplying the inverse Jacobian by the commanded velocities (or the velocities resulting from the difference between current and commanded position/orientation divided by the numerical timestep). The result is an approximation of the segment angle velocities of the endoscope that will result in the commanded position, orientation, and/or velocity at the reference frame(s) of interest. It should be appreciated that when endoscope has more degrees of freedom of movement than are constrained by a command, that command can be achieved with infinite solutions involving different segment velocity or joint angle combinations. Thus, the control system can incorporate a variety of optimizations when selecting which set of new segment velocities or angles to use to navigate to the commanded configuration. Each optimization may be biased towards a different set of segment angles, velocities, or global positions to change the behavior of the endoscope as it moves. These optimizations will be discussed below in more detail.

[00076] At step 38 of Fig. 4A, the control system calculates the reference frame velocities resulting from the segment angle velocities calculated above via forward kinematics. Essentially the velocities are integrated over time resulting in change in segment angle. That segment angle change is added to the initial segment angles to determine the segment angles that would result. From there the new position and orientation of the reference frame(s) of interest is calculated by stepping through forward kinematics.

[00077] At step 40 of Fig. 4A, the control system calculates the error of the reference frame velocities from step 38 by comparing it to the desired velocity or position/orientation (converted to velocity by subtracting the starting position/orientation and dividing the result by the numerical timestep) supplied by the user in step 32. If this error exceeds a predetermined error limit, then the calculation begins again, starting from the configuration arrived at in step 36. The Jacobian matrix is recalculated at the new configuration (as in step 30) and steps 34-40 are repeated. The new "commanded" velocity is the initial commanded velocity or velocity calculated from commanded position/orientation and numerical timestep, minus the velocity achieved in step 38. The allowable reference frame velocity error can be fixed on the order of one millimeter per second and one degree per second, for example, or may be set as a percentage of the commanded reference frame velocity, for example.

[00078] At step 42 of Fig. 4A, after the control system has found a configuration of the endoscope that achieves the commanded position, orientation, and/or velocity within error limits, the control system can compare the new joint angles against the physical joint limits of the hinged links in the endoscope. If the new joint angles can be physically obtained, then at step 44 the control system sends the commanded angles/velocities (the commanded configuration) through the lower level controllers, resulting in actuation of the resulting motor commands. If the new joint velocities cannot physically be obtained by the endoscope, the process is repeated from step 34 with the joints corresponding to exceeded joint angles removed from the Jacobian (this results in removing a row from the inverse Jacobian).

[00079] Until this point, all position, orientation, and velocity commands to the reference frame of interest have been user inputs. However, it is also possible for endoscope insertion to result in a commanded position, orientation, or velocity at the reference frame of interest if desired. For instance, if the user does not want scope insertion to affect the position or orientation of the reference frame of interest, it is possible to calculate the velocity or change in position or orientation resulting from the insertion, and command a new geometry or segment angle velocities that negate those effects.

[00080] The methods above are used to determine the segment angles (and resulting motor commands) needed to achieve a user command. However, for the purposes of both safety (segments located in the esophagus for instance could cause damage if articulated) and, as mentioned above, endoscope motion resulting from insertion into a body orifice or body cavity, a knowledge of depth is necessary. [00081] Fig. 5 illustrates another system for controlling an endoscope that further includes a device for measuring insertion depth and/or rotation. System 500 can include all the same components as described in reference to system 100 of Fig. 1, but further includes an external positioning sensor (EPS) 510, which provides a signal corresponding to depth and/or rotation information for the endoscope as it is advanced or withdrawn past the EPS. More details on an EPS and some specific embodiments are further described in commonly owned and co-pending U.S. Ser. No. 10/384,252, titled "METHOD AND APPARATUS FOR TRACKING INSERTION DEPTH," filed on March 7, 2003, and U.S. Ser. No. 11/648,408, titled "INSTRUMENT HAVING RADIO FREQUENCY INDENTIFICATION SYSTEMS AND METHODS FOR USE," filed on December 28, 2006. An EPS can provide an estimate of the depth of endoscope as it is inserted and withdrawn, provide a depth velocity signal to the control system, and provide information to determine which portions or segments of an endoscope are allowed to be "active" (i.e., which segments of the endoscope have passed a foundation point and are safe to actively control).

[00082] The systems described above, in particular system 100 of Fig. 1 and system 500 of Fig. 5, rely heavily on mathematical models to predict the shape of the endoscope. System 500 of Fig. 5, additionally includes an EPS to help calculate the position of the endoscope in space. However, in actual operation, several factors contribute to the deviation from predictions by the feed-forward model and limit the accuracy of predicted and orientation. For instance, tendons can stretch during articulation and external forces can affect force-to-articulate model. In addition, the endoscope may exit the EPS sensing device non-coaxially (at an angle relative to the central axis of the EPS) which can affect position calculations.

[00083] One way to improve the accuracy of the mathematical models is to monitor the realtime joint angles and use that information as an input into the control system and use that information to provide shape feedback. Fig. 6 illustrates a system 600 that includes all the same components as described in reference to system 500 of Fig. 5, but further includes shape sensors 632 positioned along the length of the endoscope 602 at each hinge or joint between adjacent links. The shape sensors are adapted to provide a sensor data signal 609 to the control system corresponding to a joint angle of articulation.

[00084] Fig. 7 illustrates a distal segment 714 corresponding to distal segment 614a of endoscope 602 in Fig. 6. As shown in Fig. 7, the endoscope segment can include shape sensors 732 positioned at each joint between adjacent links along the length of the segment. The shape sensors are configured to measure a joint angle between each adjacent link. Suitable shape sensors can include optical shape sensors, resistance changing flexible band sensors, potentiometers, magnetic sensors, or any other angle or bend measuring sensor as known in the art. Further details about shape sensors being used in an articulating endoscope to measure the angles between adjacent links can be found in commonly owned and co-pending International Application No. PCT/US08/52365, titled "METHODS AND APPARATUS FOR USING SHAPE SENSOR WITH CONTROLLABLE INSTRUMENTS," filed on January 29, 2008. [00085] Referring back to Fig. 6, the shape sensors 632 measure the joint angles of articulation at each joint of the endoscope. As described above, the control system can use a kinematics model, a feed-forward model, and depth/rotation information from an EPS to determine the shape, position, and orientation of the endoscope in space. In Fig. 6, the system additionally includes joint angle information from shape sensors 632 in a feedback control loop with the control system to continually update the kinematics model, the feed-forward model, and the real time shape, position, and orientation information of the endoscope. [00086] Fig. 8 is a diagram illustrating the flow of commands and feedback in a system incorporating joint angle feedback, such as the system of Figs. 6-7. At step 80, a commanded a configuration ^cmd , such as the commanded configuration from step 44 of Fig. 4 A, is sent from the control system. It is run through the feed- forward model at step 82 to predict what actuator commands, ^cmd , are necessary to achieve the commanded angles, and those actuator commands are sent to the endoscope (i.e., physical plant) at step 84. Tendons are pulled and the θ θ endoscope moves. The resulting angles, ^seg , are measured by the sensors, ^ses , at step 86, and

Q compared to the originally commanded angles from the supervisory controller, ^cmd . The difference between the measured angles and the commanded angles is fed through an incremental motion controller in the control system at step 88 to determine the additional

A θ actuator command, ^cm" , necessary to achieve the commanded configuration, ^cmd .

[00087] The accuracy with which the shape, position, and/or orientation of the endoscope are calculated can be increased drastically if the real-time position and orientation of the segments is known. In a preferred embodiment, the system described herein includes a position and orientation determination system which interacts with a control system for accurately calculating the shape, position, and/or orientation of the endoscope and improving the accuracy of the endoscope's movement. Fig. 1 IA is a schematic diagram of a system that includes an endoscope 1102 and a control system 1106 that receives, as inputs, a user command from a user interaction device 1104, keep-out region constraints 1178 from a user interaction device and/or the endoscope, and position and orientation information from a position and orientation determination system 1180. Details about the keep-out region constraints will be discussed below. The control system uses these inputs to direct the movement of the endoscope. [00088] Fig. 1 IB illustrates a schematic diagram of various embodiments of a position and orientation determination system 1180. In a first embodiment, the position and orientation determination system can provide position and orientation information about the endoscope from position and orientation sensors 1134 that operate within an externally generated field and resolve the position and orientation of the sensors based on each sensor's interaction with the generated field. An example of this type of system is shown in Fig. 18 A. Not shown but included in the system of FIG. 18A are the other elements of system 1100 shown in FIG. 11 A. Fig. 18 A illustrates an embodiment of an endoscope 1802 comprising position and orientation sensors 1834 and a magnetic field transmitter 1836, which emits a variable magnetic field 1838. As shown, the transmitter can be embedded in flat plate that is placed under the endoscope. However, in other embodiments, the transmitter can be any kind of external field emitting transmitter placed near the endoscope. The position and orientation sensors 1834 in Fig. 18A are configured to detect changes in the variable magnetic field 1838 to determine the orientation and position of the sensor in the magnetic field. The sensors 1834 themselves can comprise coils that conduct an induced current when in the presence of the variable magnetic field. The changes in the induced current in the sensors can then be used to determine the position and orientation of the sensor for up to six degrees of freedom with respect to the transmitter, the magnetic field or both.

[00089] Magnetic sensors as described above are known in the art. See, for example, the 3D Guidance medSAFE ™ system produced by Ascension Technology Corp. However, the position and orientation sensors described with respect to Fig. 18 may be any available sensor configured to detect and operate within the particular generated field. The sensors 1834 are not limited to magnetic sensors such as those manufactured by Ascension Technology Corp. Further details on these types of position and orientation sensors and their use can be found in U.S. Provisional No. 61/074,117, titled "APPARATUS AND METHOD FOR AUTOMATICALLY CONTROLLING AN ARTICULAT ABLE ELONGATE DEVICE," filed on June 19, 2008. [00090] Referring back to Fig. 1 IB, in another embodiment, the position and orientation determination system 1180 can provide position and orientation information about the endoscope from transmitters or emitters 1184 that propagate a signal to external detector(s) or receiver(s) which can resolve the position and orientation of the transmitters or emitters from this signal. An example of this type of system is shown in Fig. 18B. Not shown but included in the system of FIG. 18B are the other elements of system 1100 shown in FIG. 1 IA. Fig. 18B illustrates an embodiment of an endoscope 1802 comprising transmitters or emitters 1184 that emit a signal 1890, and receivers or detectors 1892, which can receive or detect the position and orientation of the transmitters or emitters from this signal. [00091] Referring back to Fig. 1 IB, in another embodiment, the position and orientation determination system 1180 can provide position and orientation information about the endoscope from an external measurement system 1186 that can resolve the position and orientation of the endoscope without any kind of sensor, emitter, or transmitter on the endoscope. The external measurement system may include IR, microwave, or similar systems. An example of this type of system is shown in Fig. 18C. Not shown but included in the system of FIG. 18C are the other elements of system 1100 shown in FIG. 1 IA. Fig. 18C illustrates an embodiment of an endoscope 1802 and external measurement system 1186 that emits a signal 1896 that it uses to detect the position and orientation of the endoscope. [00092] Referring back to Fig. 1 IB, in another embodiment, the position and orientation determination system 1180 can provide position and orientation information about the endoscope with various instrumentation 1188. This embodiment refers to devices that may be placed on the endoscope that may provide information to be used along with the other types of position and orientation determination systems described above, such as position and orientation sensors 1134, transmitters or emitters 1184, and external measurement system 1186. The instrumentation 1188 could also be accelerometers or gyroscopes, for example. Many small scale gyroscopes, gyrocompass and accelerometers are currently available that would be suited for use as described herein. Optically based gyroscopes, such as resonant and interferometric fiberoptic gyroscopes, are available in a number of configurations. Vibration based gyroscopes, such as those found in micro electrical mechanical system or MEMS gyroscopes could be used. It is to be appreciated that the instrumentation 1188 may also be incorporated into or used in combination with the various alternative position and orientation determination systems. Moreover, the instrumentation 1188 may also be combined with other information in the control system (i.e., kinematic information) in order to resolve position and orientation of the endoscope. [00093] Fig. 9 illustrates a preferred embodiment of a system 900 for controlling an endoscope 902. Fig. 9 includes all of the components of system 100 in Fig. 1. Therefore, endoscope 902, user interaction device 904, control system 906, actuation system 908, and display 912 of Fig. 9 correspond, respectively, to endoscope 102, user interaction device 104, control system 106, actuation system 108, and display 112 of Fig. 1. Additionally, distal segment 914a, proximal segments 914b-d, and reference frame 901 of Fig. 9 correspond, respectively, to distal segment 114a, proximal segments 114b-d, and reference frame 101 of Fig. 9.

[00094] System 900 further includes position and orientation sensors 934 (e.g., sensors 934a-e in Fig. 9). The position and orientation sensors are configured to output a signal to the control system corresponding to their real time position and orientation in space. Thus, in addition to position information, the sensors 934a-d can give orientation data for up to six degrees of freedom, including along the x, y, z, yaw, pitch, and roll axes of the sensor. Some embodiments of system 900 may use sensors that provide orientation data for less than six degrees of freedom. For example, a five degree of freedom sensor may provide position in x, y, z directions, but orientation in yaw and pitch, but not roll. Sensors providing data along only five degrees of freedom may be smaller or easier to manufacture, for example. The position and orientation sensors can be sensors configured to detect a variable magnetic field or some other external reference to determine position and orientation, for example. In Fig. 9, the endoscope is illustrated with multiple position and orientation sensors. However, in another embodiment, only a single position and orientation sensor may be used on the distal tip of the endoscope. The shape, position, and/or orientation of the endoscope may then be calculated using information from the sensor with a kinematic model, as described above.

[00095] As shown in Fig. 9, position and orientation sensors can be positioned along the endoscope 902. In the embodiment of Fig. 9, the sensors are situated at the distal ends of each segment (i.e., on distal segment 914a and proximal segments 914b-d). Additionally, another sensor is positioned at the proximal end of the endoscope (i.e., the proximal end of proximal segment 914d). However, in other embodiments, any number of position and orientation sensors 934 can be placed on the endoscope. In another embodiment, position sensors may also be placed near the center of each segment. [00096] Fig. 10 shows a close up view of distal segment 1014, which corresponds to distal segment 914a of Fig. 9. Link 1018, hinges 1020, tendons 1022, coil tubes 1024, pulley 1026, and mechanical stops 1028 in Fig. 10 correspond, respectively, to link 218, hinges 220, tendons 222, coil tubes 224, pulley 226, and mechanical stops 228 in Fig. 2. As shown in Fig. 10, position and orientation sensor 1034a is positioned at the distal end of segment 1014. The sensors are not limited to being placed at the distal end of each segment, however, and may be placed anywhere along the segment as desired, such as the proximal end or middle of the segment. The sensors may need to be "zeroed" to the endoscope to be able to translate the position of the sensor to the desired position on the endoscope to be measured. Ideally, the sensors are positioned in the center of the endoscope at the distal end of each segment, however this placement may interfere with the endoscope working channels. The sensors may be positioned in the same place in each segment (i.e., at the distal end of the segment on the inside wall with a specified position/orientation). The sensor can be attached to any fixed position within the segment, such as the inner or outer wall of the segment with an adhesive, incorporated or embedded into the walls of a link of the segment, or even held in place on the outside of the segment when the plastic sheath, as described above, covers the segment. Any other method or technique for attaching a sensor to an endoscope may be used, as known in the art.

[00097] By placing a position and orientation sensor near the distal ends of each segment, as illustrated in Figs. 9-10, the control system can accurately and reliably approximate the shape of the endoscope, as well as its position and orientation in space. For example, referring to Fig. 9, the approximate shape, position, and orientation of distal segment 914a can be determined with the position and orientation sensors 934a and 934b.

[00098] The control system can use position and orientation information from the sensors embedded in fixed locations of each segment and the kinematic model described above to accurately approximate the real-time shape, position, and orientation of each segment of the endoscope. This allows more accuracy in approximating the shape, position, and orientation of the endoscope than in the embodiments of Figs. 1, 4, and 6, which, do not include position and orientation sensors. It should be noted that the system of Fig. 9 does not require an EPS to determine position and orientation. However, to approximate the insertion depth of the system of Fig. 9, the position and orientation information from sensors 934 may be referenced to a foundation point on or near the endoscope. For example, the foundation point may be the point of entry into a patient during a surgical procedure. The foundation point could even be integral to an external fixture that holds the endoscope in place. In a NOTES or single port access procedure, the foundation point could be at the patients mouth, vagina, colon, umbilicus, or at the port of entry into the patient. [00099] Figs. 12A-C are schematic diagrams of one embodiment of a control system for controlling an endoscope with position and orientation sensors. Fig. 12A illustrates a system for controlling an endoscope, such as the endoscope 902 of Fig. 9. Endoscope 1202, user interaction device 1204, control system 1206, actuation system 1208, and display 1212 of Fig. 12 can correspond, respectively, to endoscope 902, user interaction device 904, control system 906, actuation system 908, and display 912 of Fig. 9. To better understand how the control system can control the endoscope, the control system can discussed with respect to two separate control subsystems; a supervisory controller 1240 and a real-time controller 1242. It should be understood that the control system is being described with respect to multiple subsystems, to simplify description of the various functions that the control system is responsible for coordinating and executing. The control system itself may be computer software, hardware, or a combination of computer software and hardware.

[000100] As described above, a user 1201 provides a user command to user interaction device 1204. The main input provided is a commanded position, orientation, and/or velocity to move the endoscope. However, the user or endoscope can also input keep-out regions or individual keep-out data points. The keep-out region can be a set of boundaries that describe a volume, area, or point(s) in space beyond which the endoscope 1202 will not be allowed to intrude or touch. The keep-out regions can be defined by a user, by the endoscope, or by other methods. [000101] In one embodiment, the physical boundaries of the keep-out region can be manually input into the control system by a user, such as through the user interaction device or through the display. For example, a predetermined keep-out volume and keep-out region shape (i.e., box, sphere, semi-sphere, etc) may be input by the user and positioned so that the keep-out region defines the surgical space. These principles are illustrated in Figs. 13A-13B. As shown in Fig. 13 A, a three dimensional keep-out region 1364 can be formed over a reference plane 1366. The reference plane can be an operating table, for example. In another embodiment, the reference plane can be the "organ level" or organ plane of a patient. This keep-out region defines the boundaries beyond which the control system will not allow movement of the endoscope. So in the example where the reference plane is the organ level of the patient, the endoscope will not be allowed to move below the organ level, and will also be constrained to the rest of the volume defined by keep-out region 1364. Fig. 13B illustrates an endoscope 1302 operating within the confines of the keep-out region 1364. It can be seen in Fig. 13B that the various segments of the endoscope 1302 can approach the boundaries of the keep-out region 1364 during articulation, but cannot actually breach the boundaries of the keep-out region.

[000102] Keep-out regions can be especially useful in NOTES or other surgical procedures, where the body cavity of a patient can be preset as the boundaries of the keep-out region, for example. A user would then be able to control the endoscope 1302 within the keep-out region 1364 (i.e., the body cavity of the patient) without fear of bumping the endoscope into the delicate internal tissues of the patient.

[000103] While the keep-out region 1364 is illustrated as a cube in this simplified illustrative embodiment, it is to be appreciated that the user defined keep-out region may be virtually any shape or size depending upon the specific needs of the user, the endoscope and the operating environment. Moreover, the keep-out region may be used to define not only complete volume, but rather an area, a region, or a portion of a volume or specific area to be avoided or indexed. [000104] In other embodiments, a user may use the endoscope itself to form the keep-out region(s). One embodiment is illustrated in Fig. 14, which includes an endoscope 1402 corresponding to the endoscope 902 of Fig. 9. A user controlled position and orientation sensor, such as the sensor positioned on the tip of the endoscope, can be used to form multiple keep-out regions or keep-out waypoints as the user controls the endoscope. For example, if the endoscope is being moved within a body cavity in a NOTES or other surgical procedure, waypoints or keep- out regions can be created to identify the locations of crucial organs or tissue, or similarly, to identify the exact location of a target site.

[000105] This technique is illustrated in Fig. 14. For example, the distal tip of the endoscope can be moved to a first tissue site 1468a. When the user has identified the tissue site, the user can input to the control system, such as by pushing a button on the user interaction device, that the current position of the tip of the endoscope (i.e., the position of sensor 1434a) should be a first keep-out region point. The user can continue to guide the endoscope to additional tissue sites 1468b-e, inputting each tissue site as a keep-out region in the same manner as just described. Once these keep-out regions have been stored in the control system, the control system knows not to command the endoscope to move beyond those keep-out regions. In this manner, user defined keep-out regions can be used to create an internal map of a body cavity or workspace to allow the endoscope to move freely within the cavity without danger of bumping into and damaging the delicate body tissue within a patient.

[000106] The keep-out region can also be defined by the movement of the endoscope itself. As a user navigates through a region, such as a body cavity, the distal end of the endoscope will likely describe a "safe" path through the body cavity. The safe path can be defined as any position that the endoscope has traveled that has not otherwise been labeled a keep-out region. As the endoscope advances further into the cavity, the control system can control the trailing or proximal segments of the endoscope to follow the distal segment of the endoscope to ensure that the trailing segments will also fall within the "safe" path defined by the distal end of the endoscope. This safe path can define a boundary on a keep-out region so that movement of the endoscope never intrudes on or touches the keep-out region. In a related embodiment, the keep- out region can be configured to be a predetermined distance from a safe path defined by movement of the distal end of the endoscope. More details can be found in U.S. Provisional No. 61/113,534, titled "METHOD FOR ROBOTIC ENDOSCOPE PATH DEFINITION AND FOLLOWING DURING INSERTION AND RETRACTION," filed on November 11, 2008. For example, a tubular keep-out boundary with a radius of 3 inches might be defined by the movement of an endoscope as it winds through a body cavity.

[000107] Defining a keep-out region (or alternatively, a safe path or safe region) can also be useful when the endoscope is withdrawn or removed from a patient. For example, when the system knows a safe path or safe region, the endoscope can maintain its position within the safe path during withdrawal by having each segment trail or follow the segments proximal to it. [000108] The endoscope described herein may also define keep-out data or keep-out regions wirelessly with a device positioned within the working channel of the endoscope, such as with a laser, a light emitter, an ultrasonic transducer, or similar device for measuring distances. Figure 15A illustrates an endoscope 1502 which corresponds to endoscope 902 of Fig. 9. A wireless distance measuring device 1570 is positioned within a working channel 1530 of the endoscope, and emits a beam 1572. Depending -on the device used (i.e., laser, light, ultrasonic transducer, etc), the device can accurately measure the distance that the beam travels from the position of the device 1570 until the beam reaches an obstruction 1574. The obstruction can be, for example, an organ within a body cavity, a cavity wall, bone, muscle, adhesion, or other tissues within a body cavity. When the distance measuring device calculates the distance from the device to the obstruction, a user can map the location of the obstruction with respect to position and orientation sensor 1534a, which is positioned on the distal end of the endoscope. If desired, the user can instruct the control system to mark the obstruction as a keep-out region, similar to the method described above in Fig. 14. In another embodiment, the user can mark the obstruction as a safe region if the endoscope is allowed to touch the obstruction. The user may also be able mark the position of the obstruction as a desired position of the endoscope. For example, if the user desires that the endoscope approach or arrive at the obstruction, the user may provide the position of the obstruction as a commanded position to the endoscope. [000109] In a related embodiment, the distance measuring device 1570 can automatically mark all obstructions as keep-out regions. This can be a continuous process as the endoscope moves within a body cavity. Fig. 15B illustrates a method of continually mapping obstructions within a body cavity as keep-out regions. In Fig. 15B, the endoscope 1502 can be moved by a user to point at different locations within a cavity. A first keep-out region can be tagged by a user, at boundary 1576a. Similarly, second and third keep-out regions can be tagged by a user at boundaries 1576b-c. Using this technique, the endoscope can define a keep-out region by tagging boundary points. The boundary points can be tagged manually by the user, such as by pressing a button on the user interaction device, or alternatively, the endoscope can define the keep-out region automatically by remaining in an always-on or "always-tagging" state while the distance measuring device 1570 emits a beam. Alternatively, a keep-out region volume can be assigned to a tagged point. More specifically, the user can assign that a keep-out region, such as a cubical or spherical/semi-spherical volume originate or be centered upon a tagged point in space. Further details relating to defining a keep-out region with a distance measuring device are described in U.S. Provisional No. 61/135,498, titled "Target identification and registration in space," filed on August 21 , 2008.

[000110] Figs. 15C-15D illustrate an embodiment related to the embodiment described above in Figs. 15A-15B. In this embodiment, a position and orientation sensor 1571, such as the position and orientation sensors already described herein, can be mounted to a rigid extension member, such as a guidewire or rigid shaft, and extended out of the working channel 1530 until reaching an obstruction 1574. Thus, the obstruction can be mapped as a keep-out region boundary based on the position of the sensor 1571. Fig. 15D illustrates using a sensor 1571 in the same manner as described in Fig. 15B to define a keep-out boundary, or alternatively, a safe region.

[000111] In Fig. 16, an external sensor or multiple external sensors 1634, such as position and orientation sensors 934 in Fig. 9 as described above, can be used to define the boundaries of a keep-out region from outside the cavity or space within which the endoscope is operating. In Fig. 16, a patient is illustrated having multiple position and orientation sensors placed on the patient's abdomen, with the endoscope 1602 positioned inside the patient. Sensors on the endoscope can reference the external sensors to determine the position, orientation, and/or shape of the endoscope in reference to the external sensors. Thus, the thickness of the patients tissue walls, including skin, fat, muscle, etc, can easily be determined if the endoscope navigates to the interior wall of the body cavity, which can be used to define a keep-out region corresponding to the interior walls of the patient. Alternatively, instead of having a sensor or multiple sensors placed in a static configuration on the patient, as in Fig. 16, a sensor can be traced or moved along the exterior of the patient, such as along the outside of the patient's abdomen, to define the external wall of the patient. In another embodiment, the distance from the outside of the patient's abdomen to the inner wall of the body cavity can be determined with a patient's body mass index (BMI) or a physical measurement. [000112] Referring back to Fig. 12A, a user can specify information that allows the control system to choose a preferred endoscope configuration optimization. Examples include the type of surgical procedure, or the stage of the surgical procedure (i.e., insertion, incision, navigation, surgery, closure, withdrawal), the type of surgical tool used, etc. As described above, when a user issues a commanded position and orientation to the endoscope, the endoscope may be able to achieve the commanded position by forming a number of different configurations or geometries. Allowing the user to specify information corresponding to a preferred endoscope configuration optimization will tell the control system to emphasize one or more desirable features of one or more configurations over others. The user can input this information through the user interaction device (e.g., with a button, switch, lever, keyboard etc), or, in another embodiment, through the display 1212. Further details regarding the various kinds of endoscope configuration optimizations will be discussed in more detail below.

[000113] As shown in Fig. 12A, the user interaction device is in communication with the supervisory controller 1240 of the control system. Thus, the user interaction device communicates the inputs described above to the supervisory controller. Fig. 12B is a schematic diagram illustrating the various functions that supervisory controller 1240 is responsible for coordinating. First, the subsystem coordination engine 1246 receives input from user interaction device 1204, including the commanded position, orientation, and/or velocity, and information relating to the preferred endoscope configuration optimization. The subsystem coordination engine also receives position and orientation information from the real-time controller 1242. The position and orientation information corresponds to signals received from position and orientation sensors on the endoscope, such as shape sensors 934 in Fig. 9. The subsystem coordination engine can also control the graphical display 1212, which includes, for example, displaying a live feed from the endoscope camera and providing a real-time rendering of the shape, position, and orientation of the endoscope on the display. [000114] Next, the commanded position, orientation, and/or velocity, information relating to the preferred endoscope configuration optimization, the position and orientation information, and keep-out region constraints from keep-out region engine 1250 are sent to the endoscope configuration engine 1248 to determine the configuration or geometry the endoscope will assume to achieve the user command. As discussed above, there are often multiple configurations that the endoscope can take to achieve the commanded position. First, the configuration engine uses the sensed position and orientation information in combination with the kinematic model 1252 to determine the shape, position, and orientation of each segment in the endoscope. Next, the configuration engine 1248 calculates the segment angles or segment angle velocities needed to achieve the user command using by running an optimization with input from the keep-out region engine 1250, the kinematic model 1252, and the cost function engine 1254.

[000115] The keep-out region engine 1250 provides a constraint on the configuration engine optimization to ensure that the endoscope remains within the keep-out region boundaries. When the endoscope is unable to achieve the user command without intruding beyond a keep-out boundary, the supervisory controller can provide feedback to the user, such as through haptic feedback in the user interaction device or a notification on the display to inform the user that the user command cannot be achieved. Among the other constraints that can be input to the configuration engine are hardware constraints 1244, which may include joint angle limits, motor velocity limits, etc.

[000116] The remaining input to the configuration engine 1248 is from the cost function engine 1254. When there are multiple configurations that satisfy the user command and the keep-out region constraints, and hardware constraints, the cost function engine 1254 determines the commanded configuration to be sent to the real time controller. The use of the cost function engine allows the control system to preference a desired behavior when determining which endoscope configuration to achieve. Stated another way, the control system uses a cost function term to select a commanded configuration that emphasizes a preferred motion of the endoscope leading to the commanded configuration.

[000117] For example, one possible term in the cost function may emphasize minimizing the joint velocities needed to achieve the user command. This term in the cost function is a measure of total joint velocities in the endoscope such that when the cost function is minimized, smaller joint velocities are preferenced. This cost function allows the endoscope to achieve the user command with the least amount of movement.

[000118] Another term in the cost function may emphasize minimizing the joint angles needed to achieve the user command. This term in the cost function is a measure of total joint angles in the endoscope such that when the cost function is minimized, smaller joint angles are preferenced. This cost function allows the endoscope to achieve the commanded position with the least amount of articulation by the joints. Minimizing the joint angles may result in an endoscope that is less prone to knotting up or tangling during operation, and whose motion is less likely to be impeded by physical joint limits. [000119] Yet another term in the cost function may cause the endoscope to tend towards a predetermined configuration. This cost function term will minimize deviation in shape from the predetermined configuration. For example, if the endoscope is bent in a U-shape and commanded to achieve a new position, the cost function engine can select a configuration that generally maintains the U-shape. It should be understood that this U-shape may become wider or narrower as a result of the movement while still maintaining the U-shape. This type of cost function term can have surgical procedure uses. For example, if the procedure is known, there may be a known optimal predetermined shape. This can be useful to arrive at a surgical site with maximum manipulability, or, once the user has arrived at the site, allow the user to operate with a more manipulable configuration. [000120] Additionally, other terms in the cost function may emphasize motion in a distal portion of the endoscope, motion in a proximal portion of the endoscope, maximize manipulability, or even encourage motion using only one tendon to minimize compressive force within the segment. One term in the cost function may emphasize endoscope movement that maintains the furthest distance from any keep-out region or keep-out boundaries. [000121] The configuration engine 1248 minimizes the cost function subject to the constraints provided. The lowest cost function value corresponds to the "best" configuration of the endoscope. There are various methods for minimizing the cost function, including but not limited to various search and/or gradient search algorithms to find the minimum cost function value. In the scenario where there is clearly a minimum cost function value, the cost function mathematical expression is referred to as being convex, and the minimum cost function can easily be found by a method known as a gradient search, as known in the art. However, sometimes there will be several "minimum" cost function values. There are several mathematical methods, such as simulated annealing, as known in the art, that can be used to find the "lowest" of the multiple minimum cost function values. [000122] Once the configuration engine arrives at a configuration that satisfies all constraints and minimizes the cost function, that configuration is communicated back to the subsystem coordination engine 1246, and then to the real-time controller 1242 as a commanded configuration. This is the configuration to which the endoscope will be actuated to achieve the user command [000123] Fig. 12C is a schematic diagram illustrating the various functions that real-time controller 1242 is responsible for coordinating. One primary function of the real-time controller is to act as a feedback loop with the rest of the control system, by monitoring information from the position and orientation sensors to ensure that incremental motion by the endoscope is accurate. More specifically, when the endoscope is moving to achieve user command, the realtime controller can monitor the movement of the endoscope, based on position and orientation information from the sensors, to ensure that the endoscope is moving as commanded. [000124] In Fig. 12C, the incremental motion controller 1256 receives a configuration command from the supervisory controller. The incremental motion controller 1256 also receives position and orientation information from sensors on the endoscope, such as sensors 934 in Fig. 9. The incremental motion controller can then use the position and orientation information with kinematic model 1258, as described above, to determine the real-time shape of the endoscope, and then compare the real-time shape to the expected position and orientation of the endoscope at that time step, based on the configuration command from the supervisory controller. If the real-time shape of the endoscope does not match the configuration that the endoscope is supposed to be in, the incremental motion controller can interact with the tendon tension controller and/or the actuation system to better achieve the commanded configuration. Thus, the incremental motion controller is responsible for taking sensor position and orientation feedback, as well as motor position and feedback (from the actuation system) and sending commands to the actuation system and/or tendon tension controller to achieve the configuration command from the supervisory controller. [000125] Another main function of the real-time controller is to monitor and adjust the tension in the tendons to ensure optimal endoscope response to commands from the hardware controller and prevent deviations from the feed-forward model. It is also possible to continuously update the feed-forward model based on system response to actuation. In Fig. 12C, tendon tension controller 1260 receives tendon tension information from the tendon sensors in actuation system 1208. In order for the endoscope segment motions to be smooth, rapid, and predictable, the tension in the tendons should be accurately controlled. As described above, when a tendon is tensioned to articulate a particular segment, the opposite tendon should also be slacked to accommodate the segment articulation. The tendon tension controller employs a feedback algorithm, along with the feed-forward model 1262, to ensure that each tendon has the minimum tension necessary to create the commanded configuration, while ensuring sufficient tension such that the tendon can respond rapidly to a change in direction.

[000126] The method and apparatus described herein can be used by doctors to perform natural orifice transluminal endoscopic surgeries (NOTES). In one example of a NOTES procedure, an endoscope is passed through the mouth, through the lower esophageal sphincter, through an incision in the stomach wall, and into an insufflated peritoneal cavity. Once a segment of the endoscope is in a location where it is safe to articulate, it can be controlled by the control system. This safe location can be selected by the user, such as at a specified insertion length beyond the foundation point. For example, in a transgastric approach, a physician may deem that any articulation beyond the esophageal sphincter allows the endoscope to articulate within the stomach, facilitating the transgastric incision while not damage the soft, pliable, and moveable tissues of the stomach. As will be appreciated, the method and apparatus could also be applied in a trans-colonic or trans-vaginal NOTES procedure, or a single port access (SPA) technique as well. [000127] FIG. 19 illustrates a flow chart 1900 describing a method of performing surgery on a patient using the controllable instrument or endoscope and the control system described herein. The method of performing surgery can comprise the system 900 of Fig. 9, including the control system described in Figs. 12A. Alternatively, embodiments of the method can be performed using other alterative control systems and endoscopes described herein. At step 1901 there is the step of accessing an interior body portion of a patient with a controllable instrument. This step includes accessing the interior of the body via any natural orifice such as the mouth, anus or vagina or via a surgically created opening in the body such as performed during laparoscopic procedures such as, for example, single port access (SPA) procedures. Depending upon the access path used to enter the body, there may also be an additional step of forming an opening in a sidewall of a body lumen. For example, an opening may be formed in a wall the alimentary canal in order to access the peritoneal cavity or the thoracic cavity.

[000128] At step 1902 there is the step of defining within the interior body portion at least one keep-out region. In this step, the physician or user uses the techniques described herein to identify and characterize the various areas within the interior body portion in order to aid in the safe manipulation of the controllable instrument within the body. Of course, more than one keep-out region may be created and, in some procedures, numerous keep-out regions will be created depending upon the specific circumstances and anatomical considerations of a patient. [000129] Next at step 1903, after defining the keep-out regions, the physician navigates the controllable instrument to a surgical site within the body portion while preventing the controllable instrument within the body portion from intruding into the keep-out region. In this step, the control system takes the steering input from the user to position the distal end of the scope while also controlling the proximal end of the endoscope to remain clear or not intrude on the previously identified keep-out regions. In this way, the control system - through the use of defined keep-out regions - is not burdened with controlling or being concerned with the location and possible risk of harmful contact between the proximal instrument and the interior body portion. Once the user defines the keep-out regions (or redefines the regions as the surgery proceeds or circumstances warrant) then the user is relieved of controlling the proximal end. The user may input control signals that manipulate or maneuver the distal end of the segmented instrument with the confidence that the control system is maintaining the position of the proximal end within the previously defined constraints such as keep-out regions, user defined preferences or cost functions.

[000130] Next at step 1904, there is the step of performing a surgical procedure at the surgical site within the body portion while preventing the controllable instrument from intruding into the keep-out region. Here again, the user may input control signals that manipulate or maneuver the distal end of the segmented instrument with the confidence that the control system is maintaining the position of the proximal end within the previously defined constraints such as keep-out regions, user defined preferences or cost functions.

[000131] Fig. 17 illustrates a flowchart 1700 describing another method of performing surgery on a patient using the endoscope and control system described herein. The method of performing surgery can comprise the system 900 of Fig. 9, including the control system described in Figs. 12 A. Alternatively, embodiments of the method can be performed using other alterative control systems and endoscopes described herein. At step 1701 of Fig. 17, the endoscope is inserted into an orifice of a patient. The orifice can be a natural orifice, such as a mouth, a colon, or a vagina, or the orifice can be an incision in the skin formed by a physician. When the orifice is a natural orifice, at step 1702 an incision can be made within the orifice to enter a body cavity of the patient. For example, if the endoscope is inserted into a mouth of the patient and the surgery is to be performed in the peritoneal cavity, an incision will need to be made to gain access to the desired body cavity. In this example, an incision may be made from within the stomach to gain access to the peritoneal cavity. [000132] When the lumen is not a natural orifice, but rather is an incision in the skin of the patient, step 1702 is not necessary.

[000133] At step 1703, the body cavity can be insufflated, as known in the art. [000134] At step 1704, the endoscope is inserted through the incision and into the body cavity. [000135] At step 1705, a keep-out region or multiple keep-out regions can be defined within the body cavity or outside of the body cavity. These keep-out regions can be defined in any manner as described above, especially with respect to Figs. 13A-B, 14, and 15A-D.

[000136] At step 1706, the endoscope is navigated to a surgical site without intruding beyond or touching the keep-out region. The manner in which a the control system directs the movement of the endoscope to stay within a keep-out region is also described above, particularly with respect to Figs. 9, 12A-12C, 13A-B, 14, and 15A-D. The endoscope can also be automatically navigated to the surgical site.

[000137] At step 1707, the user can perform a surgical procedure at the surgical site. Surgical operations contemplated for use with this system can be, for example, peritonoscopy, intraabdominal biopsy, gastric resection, colectomy, splenectomy, Nissen funduplication, gall bladder resection, diaphragmatic pacing, hernia repair, pancreatic pseudo cyst drainage, nephrectomy, tubal ligation, oophorectomy, genycological procedure, gastric bypass surgery, gastric banding, prostatectomy, etc.

[000138] As for additional details pertinent to the present invention, materials and manufacturing techniques may be employed as within the level of those with skill in the relevant art. The same may hold true with respect to method-based aspects of the invention in terms of additional acts commonly or logically employed. Also, it is contemplated that any optional feature of the inventive variations described may be set forth and claimed independently, or in combination with any one or more of the features described herein. Likewise, reference to a singular item, includes the possibility that there are plural of the same items present. More specifically, as used herein and in the appended claims, the singular forms "a," "and," "said," and "the" include plural referents unless the context clearly dictates otherwise. It is further noted that the claims may be drafted to exclude any optional element. As such, this statement is intended to serve as antecedent basis for use of such exclusive terminology as "solely," "only" and the like in connection with the recitation of claim elements, or use of a "negative" limitation. Unless defined otherwise herein, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. The breadth of the present invention is not to be limited by the subject specification, but rather only by the plain meaning of the claim terms employed.

Claims

CLAIMSWhat is claimed is:

1. A position and orientation based control system for controlling a segmented instrument, comprising: a controllable, segmented instrument having a distal segment; a position and orientation determination system adapted and configured to provide an output relating to the shape, position, and orientation of the distal segment; an actuation system coupled to the instrument that operates to bend the instrument; a user interaction device; a control system adapted and configured to receive a user command from the user interaction device, to receive data related to a keep-out region, to receive the output from the position and orientation determination system, and selects a commanded configuration that produces a control output to the actuation system based on the commanded configuration to produce a controlled movement of the segmented instrument.

2. The position and orientation based control system according to claim 1 wherein the user command further comprises a user configuration optimization preference.

3. The position and orientation based control system according to claim 2 wherein the control output is based at least in part on the configuration optimization preference.

4. The position and orientation based control system according to claim 1 wherein the user command from the user interaction device includes a commanded position.

5. The position and orientation based control system according to claim 1 wherein the user command from the user interaction device includes a commanded orientation.

6. The position and orientation based control system according to claim 1 wherein the user command from the user interaction device includes a commanded velocity.

7. The position and orientation based control system according to claim 6 wherein the commanded velocity includes a commanded velocity of a position of a portion of the instrument.

8. The position and orientation based control system according to claim 6 wherein the commanded velocity includes a commanded velocity of an orientation of a portion of the instrument.

9. The position and orientation based control system according to claim 6 wherein the commanded velocity relates to a linear velocity of a portion of the instrument.

10. The position and orientation based control system according to claim 6 wherein the commanded velocity relates to an angular velocity of a portion of the instrument.

11. The position and orientation based control system according to claim 6 wherein the commanded velocity relates to a linear and an angular velocity of a portion of the instrument.

12. The position and orientation based control system according to one of claims 7-11 wherein the portion of the segmented instrument is the proximal portion of the segmented instrument.

13. The position and orientation based control system according to one of claims 7-11 wherein the portion of the segmented instrument is the distal portion of the segmented instrument.

14. The position and orientation based control system according to one of claims 7-11 wherein the portion of the segmented instrument is between the proximal portion and the distal portion of the segmented instrument

15. The position and orientation based control system according to any one of claims 4, 5 and 6 wherein the user command relates to a portion of the segmented instrument.

16. The position and orientation based control system according to claim 15 wherein the portion of the segmented instrument is the proximal portion of the segmented instrument.

17. The position and orientation based control system according to claim 15 wherein the portion of the segmented instrument is the distal portion of the segmented instrument.

18. The position and orientation based control system according to claim 15 wherein the portion of the segmented instrument is between the proximal portion and the distal portion of the segmented instrument.

19. The position and orientation based control system according to any one of claims

4, 5 and 6 wherein the user command relates to the distal portion of the segmented instrument.

20. The position and orientation based control system according to claim 1 wherein the control output is based at least in part on the user command from the user interaction device.

21. The position and orientation based control system according to claim 1 wherein the control output is based at least in part on the signal related to a keep out region.

22. The position and orientation based control system according to claim 1 wherein the control output is based on a comparison of a kinematic model of the controllable segmented instrument and the outputs from the first and second position and orientation sensors.

23. The position and orientation based control system according to claim 1 wherein the control output is based on a comparison of a kinematic model of the controllable segmented instrument and the output from the position and orientation determination system.

24. The position and orientation based control system according to claim 1, the position and orientation control system further adapted and configured to determine a number of possible configurations based on the received signals.

25. The position and orientation based control system according to claim 24, wherein the position and orientation control system selects the commanded configuration that does not violate the keep-out region.

26. The position and orientation based control system according to claim 25 wherein the control output is based on one of the number of possible configurations.

27. The position and orientation based control system according to claim 1 wherein the control system applies a cost function term to select a configuration that emphasizes a preferred motion leading to the commanded configuration.

28. The position and orientation based control system according to claim 27 wherein the cost function term relates to minimizing joint velocities used to achieve a configuration of the controllable instrument that satisfies the received inputs.

29. The position and orientation based control system according to claim 27 wherein the cost function term relates to minimizing joint angles used to achieve a configuration of the controllable instrument that satisfies the received inputs.

30. The position and orientation based control system according to claim 27 wherein the cost function term relates to minimizing the articulation of a joint in the segmented instrument.

31. The position and orientation based control system according to claim 27 wherein the cost function term relates to increasing movement of the distal end of the segmented instrument over the movement of the proximal end of the segmented instrument.

32. The position and orientation based control system according to claim 27 wherein the cost function term relates to increasing movement of the proximal end of the segmented instrument over the movement of the distal end of the segmented instrument.

33. The position and orientation based control system according to claim 1 wherein the data related to the keep-out region is provided using a position and orientation sensor on the instrument.

34. The position and orientation based control system according to claim 1 wherein the signal related to the keep out region is provided by a user.

35. The position and orientation based control system according to claim 1 wherein the signal related to the keep out region is provided by a device inserted through the instrument.

36. The position and orientation based control system according to claim 35 wherein the device is a mechanical device having a position and orientation sensor.

37. The position and orientation based control system according to claim 35 wherein the device utilizes a light signal to indicate a portion of the keep out region.

38. The position and orientation based control system according to claim 37 wherein the light signal is a laser.

39. The position and orientation based control system according to claim 35 wherein the device utilizes an ultrasound signal to indicate a portion of the keep-out region.

40. The position and orientation based control system according to claim 1 the position and orientation determination system further comprising: an apparatus to generate or sense an electromagnetic field.

41. The position and orientation based control system according to claim 1 wherein the actuation system includes a motor.

42. The position and orientation based control system according to claim 41 wherein the motor is connected to a tendon attached to the distal segment.

43. A position and orientation based control system for controlling a segmented instrument, comprising: a controllable, segmented instrument having a distal segment, a first position and orientation sensor on the distal segment that provides an output corresponding to a position and an orientation of a distal portion of the distal segment; and a second position and orientation sensor on the instrument that provides an output corresponding to a position and an orientation of a proximal portion of the distal segment; an actuation system coupled to the instrument that operates to bend the instrument; a user interaction device; a control system adapted and configured to receive a user command from the user interaction device, to receive data related to a keep-out region, to receive the outputs from the first and second position and orientation sensors, and to produce a control output to the actuation system to produce a controlled movement of the segmented instrument.

44. A method of performing surgery on a patient, comprising: accessing an interior body portion of the patient with a controllable instrument; defining within the interior body portion at least one keep out region; navigating the controllable instrument to a surgical site within the body portion while preventing the controllable instrument within the body portion from intruding into the at least one keep out region; and performing a surgical procedure at the surgical site within the body portion while preventing the controllable instrument within the body portion from intruding into the at least one keep out region.

45. The method of performing surgery on a patient according to claim 44 further comprising: insufflating at least a portion of the interior body portion before the defining step.

46. The method of performing surgery on a patient according to claim 44, the accessing step further comprising introducing the controllable instrument through a natural orifice in the patient.

47. The method of performing surgery on a patient according to claim 44, the accessing step further comprising introducing the controllable instrument through a surgically created opening in the patient.

48. The method of performing surgery on a patient according to claim 44, the accessing step further comprising introducing the controllable instrument through a natural opening in the patient and thereafter through a surgically created opening.

49. The method of performing surgery on a patient according to claim 44 wherein the accessing an interior body portion of the patient step comprises: accessing a portion of the peritoneal or retroperitoneal cavity.

50. The method of performing surgery on a patient according to claim 49 the performing a surgical procedure step further comprising: performing a surgical procedure on an organ accessed via the portion of the peritoneal or the retroperitoneal cavity.

51. The method of performing surgery on a patient according to claim 49 wherein the interior body portion of the patient is a portion of the pelvic region accessed after the accessing step.

52. The method of performing surgery on a patient according to claim 51 the performing the surgical procedure step further comprising: performing a surgical procedure on an organ or tissue in the pelvic area.

53. The method of performing surgery on a patient according to claim 52 wherein the organ or tissue in the pelvic area relates to the prostate.

54. The method of performing surgery on a patient according to claim 52 wherein the performing the surgical procedure step comprises a prostatectomy.

55. The method of performing surgery on a patient according to claim 44 wherein the accessing an interior body portion of the patient step comprises: accessing a portion of the thoracic cavity.

56. The method of performing surgery on a patient according to claim 55 wherein the step of performing a surgical procedure comprises performing surgery on an organ, a tissue or a bone in the thoracic cavity.

57. The method of performing surgery on a patient according to claim 55 the step of performing a surgical procedure further comprising: performing surgery on a lung, a heart, a blood vessel, an esophagus, a trachea, a pleura, or a diaphragm.

58. The method of performing surgery on a patient according to claim 55 the step of performing a surgical procedure further comprising: performing surgery on an organ or tissue accessed via the thoracic cavity.

59. The method of performing surgery on a patient according to claim 44, the navigating the controllable instrument step further comprising: steering the distal end of the controllable instrument based on a user input.

60. The method of performing surgery on a patient according to claim 44, the navigating the controllable instrument step further comprising: controlling the distal end of the controllable instrument in accordance to the user input while controlling the movement of the proximal end of the controllable instrument so as to prevent the proximal end of the controllable instrument from intruding into the at least one keep out region.

61. The method of performing surgery on a patient according to claim 44 wherein the navigating the controllable instrument step is performed so as to satisfy a user defined position, orientation or velocity preference.

62. The method of performing surgery on a patient according to claim 44 wherein the accessing step, the navigating step or the performing step is performed using a command configuration that satisfies a cost function.

63. The method of performing surgery on a patient according to claim 62 wherein the cost function includes minimized joint angles or segment angles of a portion of the instrument.

64. The method of performing surgery on a patient according to claim 62 wherein the cost function includes maximum flexibility of a portion of the instrument.

65. The method of performing surgery on a patient according to claim 62 wherein the cost function includes maximum maneuverability for a portion of the instrument controlled by a user interaction device.

66. The method of performing surgery on a patient according to claim 62 wherein the cost function includes maximum maneuverability for axial movement into the body portion.

67. The method of performing surgery on a patient according to claim 62 wherein the cost function includes maximizing a distance from the keep out region.

68. The method of performing surgery on a patient according to claim 44, the performing a surgical procedure step further comprising: steering the distal end of the controllable instrument based on a user input.

69. The method of performing surgery on a patient according to claim 44, the performing a surgical procedure step further comprising: controlling the distal end of the controllable instrument in accordance to the user input while controlling the movement of the proximal end of the controllable instrument so as to prevent the proximal end of the controllable instrument from intruding into the at least one keep out region.

70. The method of performing surgery on a patient according to claim 44 wherein the performing a surgical procedure step is performed so as to satisfy a user defined position, orientation or velocity preference.

71. The method of performing surgery on a patient according to claim 44 wherein the performing a surgical procedure step is performed so as to satisfy a cost function.

72. The method of performing surgery on a patient according to claim 44 wherein the performing a surgical procedure step comprises performing a bariatric surgery.

73. The method of performing surgery on a patient according to claim 44 wherein the performing a surgical procedure step further comprising the step of: performing tissue manipulation.

74. The method of performing surgery on a patient according to claim 73 the step of performing tissue manipulation including at least one of: excising tissue, cutting tissue, suturing tissue, stapling tissue, fastening tissue, cauterizing tissue, retracting tissue, expanding tissue, repositioning tissue, clipping tissue, approximating tissue, and ablating tissue.

75. The method of performing surgery on a patient according to claim 44 wherein the performing a surgical procedure step comprises performing a step related to the placement of diaphragmatic pacing leads.

76. The method of performing surgery on a patient according to claim 44 wherein the performing a surgical procedure step comprises performing a peritoneoscopy.

77. The method of performing surgery on a patient according to claim 44 wherein the performing a surgical procedure step comprises performing surgery on the gastrointestinal tract.

78. The method of performing surgery on a patient according to claim 44 wherein the performing a surgical procedure step comprises performing surgery on an organ of the gastrointestinal system.

79. The method of performing surgery on a patient according to claim 44 wherein the performing a surgical procedure step comprises performing surgery on an organ of urinary system.

80. The method of performing surgery on a patient according to claim 44 wherein the performing a surgical procedure step comprises performing surgery on one of a kidney, an ureter, a urinary bladder, or an urethra.

81. The method of performing surgery on a patient according to claim 44 wherein the performing a surgical procedure step comprises performing gynecological surgery.

82. The method of performing surgery on a patient according to claim 44 wherein the performing a surgical procedure step comprises performing surgery within the peritoneal cavity.

83. The method of performing surgery on a patient according to claim 44 wherein the performing a surgical procedure step comprises performing surgery within the thoracic cavity.

84. The method of performing surgery on a patient according to claim 44 wherein the accessing, the defining and the navigating steps are performed so that the surgical site within the body portion includes an organ, a tissue, a vessel or a bone in or accessed via the thoracic cavity.

85. The method of performing surgery on a patient according to claim 44 wherein the accessing, the defining and the navigating steps are performed so that the surgical site within the body portion includes an organ, a tissue, a vessel or a bone in or accessed via the peritoneal or retroperitoneal cavity.

86. The method of performing surgery on a patient according to claim 44 wherein the at least one keep out region includes an organ, a tissue, a vessel or a bone in the peritoneal or retroperitoneal cavity.

87. The method of performing surgery on a patient according to claim 44 wherein the at least one keep out region includes an organ, a tissue, a vessel or a bone in the thoracic cavity.

88. The method of performing surgery on a patient according to claim 44 wherein the at least one keep out region includes an organ, a tissue, a vessel or a bone in the interior body portion.

89. The method of performing surgery on a patient according to claim 44 wherein the accessing, the defining and the navigating steps are performed so that the surgical site within the body portion includes: a liver, a gall bladder, a stomach, an intestine, a heart, a lung, a kidney, a bladder, an artery, a vein, or a prostate.

90. A controllable, segmented instrument, comprising a distal segment having at least two links connected by a hinge and at least one tendon connected to the distal segment whereby movement of the tendon connected to the distal segment causes bending of the distal segment about the at least one hinge; a first position and orientation sensor on the distal segment that provides an output corresponding to a position and an orientation of a distal portion of the distal segment; and a second position and orientation sensor on the controllable segmented instrument that provides an output corresponding to a position and an orientation of a proximal portion of the distal segment.

91. A controllable, segmented instrument according to claim 90, further comprising: a control system adapted and configured to receive the outputs from the first and second position and orientation sensors and determine the shape, position, and orientation of a portion of the segmented instrument.

92. A controllable, segmented instrument according to claim 91 wherein the portion of the segmented instrument comprises the distal segment.

93. A controllable, segmented instrument according to claim 91 wherein the shape, position, and orientation of a portion of the segmented instrument comprises the portion of the segmented instrument between the first and second position and orientation sensors.

94. A controllable, segmented instrument according to claim 91 wherein the control system sends a control signal to an actuation system to move the at least one tendon to cause bending of the distal segment.

95. A controllable, segmented instrument according to claim 94 wherein the control signal is based in part on a user command from a user interaction device.

96. A controllable, segmented instrument according to claim 90 wherein the first position and orientation sensor provides an output comprising a three coordinate position of a distal tip of the distal segment and at least one of a roll, a pitch or a yaw orientation of the distal tip of the distal segment; and wherein the second position and orientation sensor provides an output comprising a three coordinate position of a proximal end of the distal segment and at least one of a roll, a pitch or a yaw orientation of the proximal end of the distal segment.

97. A controllable segmented instrument according to claim 90 further comprising: a second segment having at least two links connected by a hinge and at least one tendon connected to the second segment whereby movement of the tendon connected to the second segment causes bending of the second segment about the at least one hinge, the second segment being proximal to the distal segment.

98. A controllable segmented instrument according to claim 97 wherein the second position and orientation sensor is on the second segment.

99. A controllable segmented instrument according to claim 97 further comprising: a third position and orientation sensor on the controllable segmented instrument that provides an output corresponding to a position and an orientation of a proximal portion of the second segment.

100. A controllable segmented instrument according to claim 97, the second segment comprising: a coil tube in the second segment containing the at least one tendon connected to the distal segment.

101. A controllable segmented instrument according to claim 90, wherein the first position and orientation sensor is attached to an outside surface of an outer wall of the distal segment.

102. A controllable segmented instrument according to claim 90, wherein the first position and orientation sensor is attached to an interior surface of a wall of the distal segment.

103. A controllable segmented instrument according to claim 90 wherein the first position and orientation sensor is embedded in the wall of a link in the distal segment.

104. A controllable segmented instrument according to claim 90, wherein the first position and orientation sensor is attached to the distal segment with an adhesive.

105. A controllable segmented instrument according to claim 90, wherein the first position and orientation sensor is held on the distal segment using a plastic sheath.

106. A controllable segmented instrument according to claim 90, wherein the tendon is a flexible cable.

107. A controllable segmented instrument according to claim 90, wherein the tendon is made from polyethylene.

108. The controllable segmented instrument according to claim 90 wherein the first and the second position and orientation sensors are located in the same relative segment position.

109. The controllable segmented instrument according to claim 108 wherein the relative segment position is in the distal portion of the segment.

110. The controllable segmented instrument according to claim 108 wherein the relative segment position is in the proximal portion of the segment.

111. The controllable segmented instrument according to claim 108 further comprising third and fourth position and orientation sensors located in the same relative segment position.