Classifications

• • G—PHYSICS
  • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
  • G09B—EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS
  • G09B23/00—Models for scientific, medical, or mathematical purposes, e.g. full-sized devices for demonstration purposes
  • G09B23/28—Models for scientific, medical, or mathematical purposes, e.g. full-sized devices for demonstration purposes for medicine
• • G—PHYSICS
  • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
  • G09B—EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS
  • G09B23/00—Models for scientific, medical, or mathematical purposes, e.g. full-sized devices for demonstration purposes
  • G09B23/28—Models for scientific, medical, or mathematical purposes, e.g. full-sized devices for demonstration purposes for medicine
  • G09B23/285—Models for scientific, medical, or mathematical purposes, e.g. full-sized devices for demonstration purposes for medicine for injections, endoscopy, bronchoscopy, sigmoidscopy, insertion of contraceptive devices or enemas
• • G—PHYSICS
  • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
  • G09B—EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS
  • G09B9/00—Simulators for teaching or training purposes
• • A—HUMAN NECESSITIES
  • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
  • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
  • A61B34/00—Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
  • A61B34/30—Surgical robots

Definitions

• Disclosed features concern medical training equipment and methods, and more particularly medical training equipment and methods used for training in minimally invasive surgical procedures and techniques.
• Medical procedures on patients can involve a variety of different tasks by one or more medical personnel. Some medical procedures are minimally-invasive surgical procedures performed using one or more devices, including teleoperated medical devices. In some such systems, a surgeon operates controls via a console, which remotely and precisely control surgical instruments that interact with the patient to perform surgery and other procedures. In some systems, various other components to the system can also be used to perform a procedure. For example, the surgical instruments can be provided on a separate instrument device or cart that is positioned near or over a patient, and a video output device and other equipment and devices can be provided on one or more additional units.
• a simulator unit for example, can be coupled to a surgeon console instead of the actual other system components, to provide a surgeon with a simulation of performing the procedure. With such a system, the surgeon can learn how simulated instruments respond to manipulation of the console controls.
• assistants may move and position teleoperated arms and instruments of an instrument unit in the correct positions, which can have a significant effect on the procedure. It can be beneficial for assistants to also be able to find needed information quickly during a procedure. In addition, it can be beneficial to quantify training and performance of such tasks by surgeons and assistants, thereby enabling such personnel to track progress and improve performance.
• a system includes a simulation processing component including at least one processor and generating a virtual environment using position signals that describe at least one of a position and a configuration of a physical surgical instrument relative to a physical surgical site.
• the simulation processing component updates the virtual environment according to changes in the position signals and according to control signals corresponding to inputs by a user of the system.
• the updating of the virtual environment comprises moving a virtual surgical instrument within the virtual environment, where an interaction of the virtual surgical instrument with a virtual surgical site of the virtual environment is defined at least partly by a physical relationship between the physical surgical instrument and the physical surgical site.
• the simulation processing component outputs a representation of a simulation state signal indicative of a current state of the virtual environment.
• implementations can include a dummy instrument, anatomical model, control console, display device, teleoperable medical device, and/or other variations.
• a method includes coordinating a simulated medical procedure using a simulation processing component and receiving position signals based on one or more positions of elements of a teleoperable medical device moved by at least one trainee during the simulated medical procedure.
• the elements are physically positionable relative to a physical surgical site in order to perform the simulated medical procedure.
• Simulation state signals are determined based on the position signals, where the simulation state signals are indicative of a current state of the simulated medical procedure including integration of the position signals from the teleoperable medical device.
• the simulation state signals are sent to at least one output device operative to output a representation of the simulation state signals.
• Various implementations of the method can include receiving the position signals in a simulated setup procedure for setup tasks performed by a trainee, and/or in a simulated surgical operation following the simulated setup procedure, outputting real-time feedback information to at least one trainee performing the tasks, and other variations.
• a method includes receiving position signals indicating positions of one or more physical surgical instruments relative to a physical simulated surgical site in a simulated medical procedure.
• a virtual environment is updated based on the position signals, where the virtual
• Control signals are received from a control console and indicate manipulation of one or more input controls of the control console by a user.
• the method updates the virtual environment based on the control signals, including moving one or more virtual surgical instruments within the virtual environment.
• Interaction of the virtual instruments with the virtual surgical site are based on the positions of the one or more physical surgical instruments relative to the physical surgical site.
• Simulation state signals are output to at least one output device to cause output of a representation of the simulation state signals, where the simulation state signals are indicative of a current state of the virtual environment.
• Various implementations of the method can include physical surgical instruments being coupled to associated manipulator arms of a teleoperated medical device, or physical surgical instruments being manually operated by one or more users relative to a physical anatomical model, and other variations.
• Fig. 1 is a diagrammatic illustration of an example simulation system including a teleoperated medical system, according to some implementations;
• Fig. 2 is a block diagram illustrating an example of a simulation processing component and communication with other components of the simulation system;
• Fig. 3 is a flow diagram illustrating an example method for providing a simulated setup procedure according to one or more implementations described herein;
• Fig. 4 is a flow diagram illustrating an example method for providing a simulated surgical operation according to one or more implementations described herein;
• Fig. 5 is a diagrammatic illustration of aspects of an example system which can be used for automated evaluation of simulated medical procedures
• Fig. 6A and 6B are examples of training image screens which can be displayed on one or more display screens of a simulation system
• Fig. 7 A shows one example simulation system including examples of several components described herein;
• Fig. 7B shows an example display screen provided on the surgeon console of Fig. 7A;
• Fig. 8 is a perspective view of an example teleoperated medical device and anatomical model
• Fig. 9A shows another example simulation system including examples of several components described herein;
• Fig. 9B shows an example display screen provided on the surgeon console of Fig. 9A;
• Figs. 1 0A-1 0C illustrate examples related to tracking instruments within an anatomical model
• Figs. 1 1 A and 1 1 B are diagrammatic illustrations of one example of the use of an anatomical model in simulated medical procedures that include the use of both teleoperated and manual surgical instruments;
• Fig. 1 2 is a flow diagram illustrating an example method for using an anatomical model with reference to Figs. 1 1 A-1 1 B;
• Figs. 1 3A and 1 3B are diagrammatic illustrations of a second example of the use of an anatomical model in simulated medical procedures that include the use of manual surgical instruments;
• Fig. 14 is a flow diagram that illustrates an example method for using an anatomical model with reference to Figs. 1 3A-1 3B.
• the present application discloses features relating to simulated surgical procedures and training exercises.
• Various disclosed implementations of simulation systems and methods provide and teach realistic setup procedures for positioning, placement of simulation equipment for particular surgical procedures, as well as the realistic use of such equipment for the actual surgical operation.
• Simulations can involve some or all of the components involved in every stage of actual medical procedures and can involve any personnel in such procedures, to provide highly realistic training.
• Various tasks performed during all of these simulated medical procedures can be recorded and evaluated, with appropriate feedback on performances provided, allowing a high degree of analysis in the details of the procedures and enabling trainees for every function of a medical procedure to improve their skills more efficiently.
• Various simulation features described herein can allow users to learn and practice, and can allow quantification of user performance and tracking of user progress.
• a teleoperated medical system such as a da Vinci® Surgical System (e.g., a Model IS3000, marketed as the da Vinci® SiTM HDTM Surgical System), commercialized by Intuitive Surgical, Inc. of Sunnyvale, California.
• a da Vinci® Surgical System e.g., a Model IS3000, marketed as the da Vinci® SiTM HDTM Surgical System
• Intuitive Surgical, Inc. of Sunnyvale, California a teleoperated medical system
• a da Vinci® Surgical System e.g., a Model IS3000, marketed as the da Vinci® SiTM HDTM Surgical System
• Intuitive Surgical, Inc. of Sunnyvale, California Intuitive Surgical, Inc. of Sunnyvale, California.
• features disclosed herein may be embodied and implemented in various ways, including teleoperated and, if applicable, non-teleoperated (e.g., manual)
• da Vinci® Surgical Systems e.g., the Model IS3000; the Model IS2000, commercialized as the da Vinci® STM HDTM Surgical System
• Model IS2000 commercialized as the da Vinci® STM HDTM Surgical System
• a "setup procedure” or “surgical setup procedure” refers to setup tasks that configure system components to perform one or more later surgical operations.
• a “surgical operation” or “ surgical site procedure” refers to the actual surgical operation including surgical tasks at a surgical site.
• a "simulated medical procedure” or “simulated surgical procedure” can refer to the entire simulated procedure including setup procedure and surgical operation, or can include just setup procedure or surgical operation.
• the term “teleoperated medical system” refers to a system of one or more components used to perform surgical procedures using one or more master controller devices and one or more slave teleoperated medical devices.
• a “teleoperated medical device” can be a slave device controlled by a remote master device and can include one or more elements, such as manipulator arms and/or surgical instruments, that can be moved or manipulated in response to signals provided by one or more of the master controller devices, such as a control console or surgeon console operated by a user remotely from the teleoperated medical device.
• a "wet-lab” exercise refers to any exercise on actual (real) tissue, such as tissue samples, a porcine model, or cadaver.
• a “dry-lab” exercise refers to an exercise using non-tissue models or objects, including
• Fig. 1 is a diagrammatic illustration of an example simulation system 1 00 including a teleoperated medical system, according to some implementations.
• Simulation system 100 can be used to simulate actual medical procedures without using actual patients. Any simulated medical procedure or training activity that does not take place on an actual human patient can be performed using simulation system 1 00 or a variation thereof. For example, simulations of dry-lab training tasks (e.g., inanimate exercises) and/or wet-lab training tasks (e.g., exercises on real tissue, porcine model, or cadaver) can be performed.
• dry-lab training tasks e.g., inanimate exercises
• wet-lab training tasks e.g., exercises on real tissue, porcine model, or cadaver
• simulation system 1 00 can include a simulation processing component (or "processing component") 1 02, a surgeon console 1 04, a patient side cart 1 06, and a vision side cart 1 08.
• Other components can additionally or alternatively be included in the simulation system 1 00, as described in various implementations herein.
• Simulation processing component 1 02 can coordinate, control, and/or implement a simulation that involves the various other components of the simulation system 1 00.
• the simulation simulates a medical procedure environment involving the system components as if an actual patient were to be, or being, operated upon.
• the simulation processing component implements and controls the display of a virtual environment that includes a virtual surgical site depicting one or more elements of an actual physical surgical site.
• implementations can include a physical surgical site that includes a physical model and/or objects.
• the processing component 1 02 can coordinate simulation components of the simulation system and/or monitor and record parameters obtained during simulated medical procedures.
• the simulation is an interactive one that involves the simulation processing component 1 02 receiving a number of inputs from the other components of the system based on user manipulation of those components within the simulation.
• the simulation processing component also provides a number of outputs based on those inputs, where the outputs can coordinate the components of the simulation system and provide output to users of the system via any of different types of output devices (display screens, audio speakers, motors, etc.) provided on one or more of the components of the system 1 00.
• the simulation processing component can provide output to users via simulation state signals provided to one or more output devices.
• the simulation processing component can also provide feedback information to users via signals that it outputs.
• the simulation state signals can be indicative of a current state of the simulated medical procedure including integration of (e.g., influence from) inputs from one or more system components.
• the current "state" of the simulated medical procedure is the current point of progress or status in the performance of the medical procedure as influenced by the inputs of the components of the simulation system.
• simulation state signals can indicate the current positions of physical teleoperated surgical instruments of a teleoperated medical device of patent side cart 1 06 with respect to a physical surgical site, where these current instrument positions indicate the current state of progress in positioning the surgical instruments relative to the site in the simulated medical procedure, e.g., in a setup procedure.
• the simulation state signals can also indicate events in the medical procedure, such as collisions of any instruments with other instruments or surfaces, or mis-positioning of component elements.
• the current state of the simulated medical procedure can include a current state of a virtual environment, such as a virtual surgical site, implemented by the simulation processing component.
• the simulation state signals can include data that describes the virtual environment including virtual representations and current positions of surgical instruments.
• Simulation state signals can indicate current positions of virtual surgical instruments in the virtual environment based on control input from a surgeon console 1 02, where the positions of the virtual surgical instruments indicate the current state of a surgical task of a simulated medical procedure.
• the output device(s) can output a representation of the simulation state signals.
• the representation can be output using a variety of types of output, such as graphical (e.g., fully virtual / synthetic images, fully camera images, or combined camera/virtual images), tactile, haptic, aural, etc.
• the output representation can include graphical representations of physical instruments, displayed visual statuses, notifications, visual text and markers, audio cues and other output, haptic responses, and/or other output.
• the output representation can include a displayed environment at the surgical site, such as a virtual environment or images of a physical site.
• an initial state of a virtual environment can be selected by the simulation processing component by providing various controlling output signals to the other components, and users can experience current updates to the virtual environment via the state simulation signals based on user inputs via components such as the surgeon console 1 04 and/or patient side cart 1 06.
• the output device(s) can also output representations of signals providing feedback information.
• the simulation processing component 1 02 can be implemented using one or more processors (e.g., microprocessors, integrated circuits, logic, and/or other processing circuitry), as well as memory, input/output interfaces, and other components, as described below.
• simulation processing component 1 02 can be implemented as a particular external or standalone unit that is separate from the other components in the simulation system. In other
• the processing component 1 02 can be provided within or a part of one of the other components of the simulation system 100, and/or distributed within multiple other components of the system 1 00.
• One or more master consoles 1 04 can be included in system 1 00 to provide a user, such as a surgeon trainee, input controls by which surgical instruments can be teleoperated as well as various other controls.
• Surgeon console 104 can also include output devices such as visual, audio, and/or haptic output devices.
• a user operates the controls to provide control input signals to the simulation processing component.
• Control input signals can also be provided from a surgeon console 1 04 to one or more of the other components of the simulation system, such as the patient side cart 1 06 and/or vision side cart 108.
• teleoperated slave instrument arms of the patient side cart 1 06 can be controlled, e.g., each surgical instrument operated by one or more corresponding master controls of the surgeon console.
• the surgeon console 1 04 communicates with the simulation processing component 1 02 as indicated by connection 1 05.
• Connection 1 05 can be any type of communication channel, such as one or more wires or cables, wireless connections, etc.
• the surgeon console 104 outputs signals indicative of the manipulation of the controls of the console 1 04. For example, if a user moves levers, joysticks, or dials, selects particular buttons or touchscreen, or selects other controls, corresponding signals are provided to the simulation processing component 1 02. In some implementations, these signals can be standard signals provided to the other components of a teleoperated medical system during an actual medical procedure, such as patient side cart 1 06 and/or vision side cart 1 08, where the simulation processing component 1 02 can process these same signals. In other
• simulation signals which are specific to the simulation can be output by the surgeon console 1 04.
• the simulation processing component 1 02 can use the inputs to update a virtual environment of the simulation, for example.
• the surgeon console 1 04 also can output signals to one or more components of the simulation system 1 00, such as the patient side cart 1 06 and/or the vision side cart 1 08.
• signals received by the simulation processing component 1 00 can be relayed to these other components by the simulation processing component.
• the surgeon console 1 04 can have separate, direct connections similar to connection 1 05 with one or more of the other components of the simulation system.
• the output signals can drive the operation of these other components similarly to a teleoperated medical system that does not use a simulation processing component 1 02.
• surgeon console 1 04 receives signals on connection 1 05 from the simulation processing component 1 02.
• received signals include signals that would normally be received by the surgeon console 1 04 in an actual medical procedure, including simulation state signals used to update visual, audio, and/or haptic output devices of the surgeon console that provide a representation of the simulation state signals via video, audio, and haptic output to its user.
• these signals can be generated by the simulation processing component 1 02 to describe a current state of a simulated, virtual environment provided by the simulation processing component 1 02 and displayed at the surgeon console.
• received signals can include signals provided by one or more of the other components of the simulation system, such as signals received at the simulation processing component 1 02 from the patient side cart 106 and/or the vision side cart 1 08 and then relayed to the surgeon console 1 04 from the simulation processing component 1 02.
• the simulation processing component 1 02 can receive signals from one or more of the other components and can process or change these signals based on the simulation run by the simulation processing component. The processed signals can then be sent to the surgeon console 1 04 for its use.
• the simulation processing component 1 02 can create augmented reality data that is combined with or integrated into data received from the other components of the simulation system such as an image or video feed from an endoscope or other imaging device at the surgical site, and the combined data can then be sent to the surgeon console 1 04 as simulation state signals.
• the surgeon console 1 04 can have additional separate, direct connections similar to connection 1 05 with one or more of the other components of the simulation system to receive the signals from those other components similarly to a teleoperated medical system that does not use a simulation processing component 1 02.
• multiple master consoles 1 04 can be in
• Such multiple consoles can be each be operated by a dedicated user simultaneously during a medical procedure, e.g., to have each user control particular device instruments, to have one user assist the other in surgical exercises, etc.
• Each such surgeon console 1 04 can send signals to the simulation processing component 1 02 and can receive signals from the simulation processing component, e.g., describing a virtual environment.
• Some simulation implementations can allow a user at a console 1 04 to pass control of one or more surgical instruments (virtual and/or physical) or pass control of other components or inputs to a different user of a different console 1 04, e.g.
• console 1 04 or other device (e.g., other control panel in system 1 00).
• signals appropriate to each surgeon console can be received at that console, e.g., outputting a different visual perspective on a simulated surgical site at each console 1 04 based on which instruments that the particular console controls.
• Some implementations can include features specific to simulations having more than one console 1 04. For example, virtual pointers can be generated and displayed on display screens of the consoles 1 04, where one operator at one console 1 04 (e.g. an expert) can control the pointer and point to displayed objects as viewed by the other operator at the other console 1 04 (e.g., a new trainee).
• One or more patient side carts 1 06 can be included in simulation system 1 00 to provide realistic physical interactions of controlled devices that are made during an actual teleoperated medical procedure.
• one or more users such as trainee assistants who operate the patient side cart 1 06 can be trained during a simulated medical procedure using the actual patient-side devices used in
• Some trainees e.g., other, surgeon trainees
• Such features enable users to be realistically, accurately, and effectively trained during simulated medical procedures.
• the patient side cart 106 can be a standalone device separate from the other components of the system 1 00.
• Cart 1 06 can include a variety of different mechanisms and devices to enable teleoperated medical surgery on patients.
• the cart 1 06 includes one or more manipulable elements, such as multiple controlled manipulator arms 1 14 that each can have one or more surgical instruments removably the attached thereto.
• Such arms and their surgical instruments can be driven within particular ranges and modes of motion such as to allow a user of the surgeon console 1 04 to manipulate the instruments to perform a surgical medical operation on a patient.
• actuators e.g., motors
• the arms and/or instruments of the cart 1 06 can be controlled by signals from the console 1 04 and can drive movement of the instruments to perform surgical tasks.
• additional patient side carts 1 06 can be included in simulations.
• Some patient side carts can include teleoperated medical devices, while others can include other types of devices (other surgical instruments, video displays, operating room tables, etc.). Still others can include both teleoperated medical devices and non-teleoperated devices.
• a trainee user of the patient side cart 106 can perform a setup procedure involving the cart 106 to permit a (e.g. simulated) surgical operation to take place.
• this setup procedure can include tasks such as moving the cart to a proper position, and moving each arm 1 14 to a proper position.
• the setup of the patient side cart 1 06 can be with reference to a physical anatomical model 1 20.
• the anatomical model 1 20 can simulate a portion of a human patient or other subject, and can include various features allowing the surgical instruments of the patient side cart 1 06 to be positioned properly.
• the user places surgical instruments of the cart 1 06 within appropriate apertures of the anatomical model 1 20 (e.g., designated via port placement), so that the instruments obtain access to a physical surgical site simulated within the interior of the model 1 20.
• Other setup tasks may also be performed, such as installing the correct surgical instruments on the arms 1 14, selecting and operating particular controls of the cart 1 06 to enable needed functions, adjusting the positing of the manipulator arms to achieve patient clearance or avoid collisions, etc.
• the patient side cart 106 communicates with the simulation processing component 1 02 as indicated by connection 107.
• Connection 1 07 can be any type of communication channel, such as one or more wires or cables, wireless connections, etc.
• the patient side cart 1 06 can receive signals from the simulation processing component 1 02 which control its teleoperated functions, such as the movement of arms 1 14 and the manipulation of surgical instruments attached to the arms 1 14 and/or otherwise coupled to the cart 1 06.
• the patient side cart 1 06 can receive other signals such as simulation state signals (e.g., data) creating output from visual, audio, or other output devices on the cart 1 06 to the user of the cart.
• signals received by the patient side cart 1 06 can be generated by the simulation processing component 1 02 based on an implemented virtual environment, and/or can be provided by the surgeon console and passed through to the cart 1 06 by the simulation processing component 1 02.
• the patient side cart 106 also sends signals on connection 1 07 to the simulation processing component 1 02.
• signals can include data describing the current states of the cart 1 06, including positions and orientations of the arms 1 14 and surgical instruments of the cart 1 06 as determined by sensors of the cart 1 06.
• sensors of the cart 1 06. For example, joint position sensors, servo motor position encoders, fiber Bragg grating shape sensors, etc. can be used to determine kinematic information (position and/or orientation) associated with the manipulator arms.
• the signals can also include data describing a visual image of the physical surgical site as captured by an endoscope or other imaging instrument of the patient side cart 106 and /or other images describing the surgical site or simulated patient (e.g., rendered ultrasound images, patient vital signs, etc.).
• signals can also be sent, such as input data describing the cart user's actions or messages, audio data from a microphone or generated by interactions of the cart's devices, and other forms of data.
• signals describing states can also be sent, such as the states of particular cart controls, functions, etc.
• these signals can be standard signals provided to the surgeon console 1 04 of a teleoperated medical system for an actual medical procedure, where the simulation processing
• simulation components 1 02 can process these same signals.
• simulation signals specific to a simulation can be output by the patient side cart 1 06.
• the simulation processing component 1 02 can use the signals to update the virtual environment of the simulation, for example.
• the patient side cart 1 06 can send its signals to the simulation processing component 1 02, which generates appropriate signals in response which are sent to the surgeon console. In some cases or
• the simulation processing component can relay one or more of the signals from cart 1 06 directly to the surgeon console 1 04 via connection 1 25.
• the patient side cart 106 can have additional direct connections to the surgeon console 1 04, vision side cart 1 08, and/or other system components.
• the anatomical model 1 20 can include its own sensors and can provide signals to and/or receive signals from the simulation processing component 1 02 on a connection similar to connection 1 07.
• a connection 1 21 can provide signals between the anatomical model 1 20 and the simulation processing component 1 02.
• the model 1 20 can connect to patient side cart 1 06 which can relay signals between the model 120 and simulation processing component 1 02.
• Such sensors on the model 1 20 can allow manual surgical instruments to be tracked by the simulation, as described in greater detail below.
• system 1 00 can include other operating room equipment (e.g., operating table supporting the model 1 20, assistive tables or carts for additional surgery or support functions, etc.) which can include connections and communication to the simulation processing component 1 02 similarly to the anatomical model 1 20.
• other equipment e.g., operating table supporting the model 1 20, assistive tables or carts for additional surgery or support functions, etc.
• such other equipment can be included in and its use evaluated for simulation tasks and procedures described herein.
• One or more vision side carts 108 can be included in some implementations of simulation system 1 00 to provide output information to assistant users of the simulation system, and/or to hold equipment such as vision and data processing hardware.
• the vision side cart 1 08 can be a standalone device separate from the other components of the system 1 00.
• a vision side cart 1 08 can be used by an assistant, such as the assistant that sets up and operates the patient side cart 1 06.
• the vision side cart 1 08 includes one or more visual output devices, such as display screens, which can output a variety of information useful to the medical procedure being performed.
• the display screen can display a view of the surgical site as captured by an endoscopic camera provided on a surgical instrument of the patient side cart 1 06, which allows the assistant user to adjust the camera to positions needed for the surgical operation.
• the display screen can also display other output information such as the states of one or more controls being activated by the surgeon at the surgeon console, the states of other devices used in the medical procedure, etc.
• the vision side cart 1 08 communicates with the simulation processing component 1 02 as indicated by connection 1 09.
• Connection 1 09 can be any type of communication channel, such as one or more wires or cables, wireless connections, etc.
• the vision side cart 1 08 can receive signals from the simulation processing component 1 02 which control its functions, such as simulation state signals causing display of a virtual environment simulating the surgical site or display of images captured at the physical surgical site, display of status information related to various system components, and output of any other types of output (audio, haptic, etc.) via appropriate output devices of the cart 1 08.
• the vision side cart 1 08 can receive such signals provided by the surgeon console 1 04 and relayed through to the cart 1 08 by the simulation processing component 1 02.
• the vision side cart 1 08 also sends signals on connection 1 09 to the simulation processing component 1 02 and/or other components of the system 1 00.
• signals can include data describing the current states of controls or other input devices on the vision side cart which were activated by a user.
• the signals can include data received by the vision side cart 1 08 from other components such as patient side cart 1 06 and relayed by the cart 1 08 to the simulation processing component 1 02 and/or surgeon console 1 04.
• signals output by cart 1 08 can be standard signals provided to the surgeon console 1 04 of a teleoperated medical system, where the simulation processing component 1 02 can process these same signals.
• specific simulation signals can be output by the vision side cart 1 08.
• the simulation processing component 1 02 can use the signals to update the virtual environment of the simulation, for example.
• the vision side cart 1 08 can send its signals to the simulation processing component 1 02, which generates appropriate signals in response which are sent to the surgeon console 1 04 and/or to the patient side cart 1 06.
• the simulation processing component 1 02 can relay one or more of the signals from cart 1 08 directly to other components, such as to the surgeon console 1 04 via connection 1 27.
• the vision side cart 1 08 can have additional direct connections to the surgeon console 1 04, patient side cart 106, and/or other components.
• a variety of physical surgical instruments can be used by the simulation system 1 00 to more fully simulate an actual medical procedure.
• These surgical instruments can include complete, actual surgical instruments that are used in the actual medical procedure being simulated.
• standard manual surgical instruments such as a cannula 1 30 and laparoscopic instrument 132 can be used, which can be instruments not requiring the teleoperated patient side cart 1 06.
• surgical instruments used with the patient side cart 1 06 such as surgical instrument 1 34 and sterile adapter/ drape instrument 1 36 can be used, which are removably attached to teleoperated manipulator arms of the patient side cart 1 06.
• simulation system 1 00 can also or alternatively use non-operational "fake" surgical instruments 1 38.
• These can be instruments that are dummies used only for the simulation system and do not provide the full instrument functionality.
• the non-operational instruments 138 can include portions of instruments that can be attached to manipulator arms 1 14 like fully operational instruments, but need only be inserted in cannulas or apertures of the anatomical model 1 20.
• a shaft and end effector can be removed from a dummy instrument, and/or dummy instruments can be hollow instruments with no mechanism, or other non-operational versions of instruments that provide a user the experience of setting up and using such instruments during a simulated medical procedure.
• Fig. 2 is a block diagram illustrating an example of a simulation processing component 1 02 and communication with other components of the simulation system 1 00.
• Simulation processing component 1 02 can include an input processing block 202 which can perform various functions of the simulation. In some
• the input processing block 202 can implement one or more virtual environments for simulations provided by the simulation system.
• the virtual environment can provide a two dimensional (2D) or three-dimensional (3D) environment that can simulate a physical surgical site or a portion thereof.
• a portion of a human body can be simulated, including virtual models of skin surfaces and internal organs or other body structures, as well as virtual models of the surgical instruments and other objects used in an actual surgical operation.
• the virtual models can be changed and updated by the simulation processing block 202 based on signals 21 0 provided by the surgeon console 104 which indicate the manipulation of master controls on the surgeon console which direct how the surgical instruments on the teleoperated arms of the patient side cart are to be moved and manipulated.
• the signals 21 0 also can indicate other commands, such as entering particular usage modes, activating other surgical features (e.g., fluid spray, suction, etc.), or performing other functions.
• the input processing block 202 can receive signals 21 2 from patient side cart 1 06. These signals can include the positions and orientations of the manipulator arms and surgical instruments of the patient side cart, as well as statuses of various controls on the cart 106 as described above.
• the input processing block 202 can also receive signals 214 from the vision side cart 1 08, which can include statuses of various controls on the cart 108, etc., as described above.
• the input processing block 202 can also receive signals 21 5 from tracked devices 21 8, which for example can include one or more sensors of the anatomical model 120 that track manually operated surgical instruments.
• Other components of the simulation system (not shown) can similarly provide signals to the input processing block 202, such as operating room sensors that track component positions, etc.
• the input processing block 202 can also receive signals 21 6 from a simulator user interface (Ul) 220 in some implementations.
• the simulator interface 220 can present one or more options or selections to user(s) of the simulation system 100 to customize and/or select features of the simulation of the medical procedure.
• the simulator interface 220 can be presented on one or more of the components of the simulation system, such as surgical console 1 04, patient side cart 1 06, and/or vision side cart 1 08.
• the simulator interface can be implemented on its own dedicated device, such as a computer system (desktop computer, laptop computer, server, portable device, etc.).
• the simulator interface 220 can display options to a user, such as a number of different medical procedures to simulate, as well as various options, settings, and preferences for those medical procedures and for the components used in the simulation system. These selections can be provided in signals 21 6 to the input processing block 202. In some
• a single interface 220 can present options for simulated setup procedures as well as simulated surgical operations, thus allowing a unified interface to control simulated aspects of all stages of teleoperated medical procedures.
• the simulation processing component 1 02 can also include an output block 204.
• This block can provide signals to control or drive various components of the simulation system 1 00, as instructed by the simulation processing block 202.
• some signals can be signals to command functions on a component, such as signals controlling actuators on the patient side cart to move telemanipulator arms or to command another medical function (air suction, etc.).
• Some signals can be simulation state signals that cause an output to the user.
• the output block 204 can send a signal 230 output to the surgeon console 104 that provides video output on a display of the surgeon console, such as data causing a display of a virtual surgical site and virtual surgical instruments at the site that move in correspondence with a user's manipulation of the controls of the surgeon console 1 04.
• the output block 204 can send signal 232 to patient side cart 1 06, signal 234 to vision side cart, and signal 236 to simulator interface 220 to drive video displays on these components that are relevant to their functions.
• patient side cart 1 06 and/or vision side cart 1 08 can display a graphical virtual environment showing the surgical site based on a position of one or more endoscope instruments or other imaging instruments.
• Other visual output can be provided as well, such as status messages.
• output block 204 can send signals 235 to track devices to provide statuses, updates, etc. Other types of output can also be caused by signals to components, such as audio and haptic output.
• Simulator interface 220 can display an interface that can update its visual appearance based on input received from a user as provided in signal 236, such as a graphical user interface displaying graphical menu items and/or other selections and options, or other type of interface.
• Simulation processing component 1 02 can also include memory 206 in communication with the simulation processing block 202.
• Memory 206 can store various data needed by the simulation processing block 202 and simulation system 1 00.
• program instructions for implementing simulations and data describing one or more virtual environments, three-dimensional models, and various settings can be stored in memory 206.
• the simulation processing component 1 02 can monitor parameters based on events and actions occurring during simulated procedures, and can store such parameters in memory 206. For example, parameters such as time taken to perform a task that the procedure, positions of components during procedures, etc. can be stored, as described below.
• Fig. 3 is a flow diagram illustrating an example method 300 for providing a simulated medical procedure according to one or more implementations described herein.
• Method 300 can be controlled and/or coordinated by the simulation processing component 1 02.
• a simulated setup procedure is described for configuring one or more components of the simulation system before and in preparation for a simulated surgical operation that can be performed after the setup procedure.
• This example assumes the use of a patient side cart 1 06 having manipulator arms in the simulated setup procedure.
• Other implementations can include similar or equivalent setup components or tasks.
• the simulated setup procedure of method 300 can be performed while one or more user trainees are forming the setup tasks. For example, a single trainee can be required to perform all the tasks to obtain comprehensive training.
• multiple trainees can be simultaneously or otherwise required to perform setup tasks for the simulated procedure as in an actual surgical procedure.
• one trainee may be required to position components in operating room, anther trainee place ports, and another trainee position manipulator arms for docking.
• Advantages of the simulation system include the ability to train multiple trainees on a single system and or at the same time.
• the simulation processing component receives simulation option selections. These can be various selections to configure the setup
• Selections can include the type of surgical operation that is to be set up for simulation, such as procedures designed for general, urologic, gynecologic, transoral, cardiac, thoracic, and/or pediatric surgical operations. Selections can also include the particular system components to be used in the setup procedure, the experience level of the user trainee(s) involved, a difficulty level of the simulation (novice, standard, expert, etc.), time parameters for the procedure, etc. In some implementations, this interface can be the same interface used for the simulated surgical operations performed after setup (e.g., as described in Fig. 4).
• the simulation processing component receives and records signals indicating that the user is positioning one or more components of the simulation system.
• Such components may be required to be positioned in particular locations in the simulation area, e.g., absolute positions in the area or positions relative to each other.
• the patient side cart 1 06 can be placed relative to an operating table and/or anatomical model, and/or the vision side cart 1 08 can be positioned relative to the patient side cart 1 06, surgeon console 1 04, or other components.
• additional components can be positioned during setup simulation, such as the surgeon console and in any other components being used (anesthesia table, etc.).
• the component positions can be tracked using sensors, such as sensors for cameras positioned over the physical simulation area, sensors detecting the motion of the components, etc., and these positions can be sent to, monitored and recorded by the simulation processing component 1 02.
• sensors such as sensors for cameras positioned over the physical simulation area, sensors detecting the motion of the components, etc.
• a user can also indicate to the processing component that he or she has completed placing the components of the system, e.g., by providing input via the vision side cart 108 or other component.
• the simulation processing component 1 02 can record parameters such as the received signals and times taken to complete tasks, and can output signals causing feedback to be provided during this block.
• feedback can include a visual and/or audio displaying of instructions as to placement, graphical spatial diagrams or maps of actual and/or desired component placement, alerts when the user has deviated too much from appropriate placement, warnings when specific measures are not taken (e.g., moving the patient side cart without placing the arms up), etc.
• Feedback can be displayed on one or more of the output devices of the system components, in some implementations.
• the method can receive signals indicating that the user is positioning a model for the setup procedure. For example, in some
• static registration techniques can be used, where the user can move manipulator arms and instruments of the patient side cart 108 to touch the surface of the model at one or more known locations.
• the simulation processing component can determine the position and orientation of the model in 3-D space relative to the elements of the patient side cart, such as manipulator arms and/or surgical instruments. For example, this allows a virtual scene of the operating room and/or surgical site to be rendered and also can allows the simulator system to provide directed feedback, e.g., suggestions, evaluation and/or scoring, on which port(s) the user is currently using and how to move to the correct port, if appropriate.
• the simulation processing component 1 02 can record parameters such as sensor signals and times taken to complete tasks, and can provide feedback on user progress of the tasks performed during this block, such as updating visual displays.
• the positioning of the anatomical model can be sensed at a later time in method 300 instead of at block 306.
• the position and orientation of the model relative to the teleoperated medical device can be sensed after docking in block 31 0 using sensors of the teleoperated arms, and/or using sensors of the model.
• the simulation processing component receives and records signals indicating that the user is selecting or placing ports at a physical surgical site for use in a surgical operation.
• the ports can be placed in an anatomical model positioned at and/or including the physical surgical site.
• the ports are apertures or other locations in the model through which surgical instruments will be inserted, and the ports have specific pattern or distance requirements depending on the target anatomy and surgical operation selected for simulation.
• placing ports can include placing cannulas in selected apertures of the model (e.g., which can be detected from sensors in the model in some implementations), such as camera cannulas and operating instrument cannulas so that the desired surgical site portions are in view of an endoscopic or other camera surgical instrument and are in operating range of operating instruments to be placed in the cannulas.
• the system can detect the placement of ports using sensors within the anatomical model, and/or using sensors in the teleoperated arms after docking (described below). The user can indicate to the simulation processing component that he or she has completed placing the ports.
• the simulation processing component 102 can record parameters such as sensor signals and times taken to complete tasks, and can provide feedback on user progress or correctness of the tasks performed during this block (such as the correctness of position of placed ports), including updating visual displays.
• the simulation processing component receives and records signals indicating that the user is positioning ("docking") the manipulator arms and/or other elements of the patient side cart in appropriate positions and locations above or within selected ports of the model. For example, the user can position manipulator arms at particular angles, distances from each other, etc., in view of parameters such as mutual manipulator collision avoidance and required instrument range of motion.
• the simulation processing component receives signals from sensors in the arms of the patient side cart, which indicate the positions and orientations of the manipulator arms.
• Various implementations allow the user to manually move the arms or other elements by hand, and/or with remote control.
• the user can indicate to the simulation processing component that he or she has completed the docking.
• the simulation processing component 1 02 can record parameters such as sensor signals and times taken to complete tasks, and can provide feedback on user progress or correctness of the tasks performed during this block, such as updating visual displays.
• the simulation processing component receives and records signals indicating that a user is inserting the surgical instruments of the patient side cart in ports.
• the simulation processing component receives signals from sensors in the arms of the patient side cart and/or from sensors and surgical instruments which indicate the positions of the surgical instruments relative to the arms and/or surfaces of the model. Requirements can include particular distances or amounts of insertion, locking an instrument in place, etc.
• sensors in the model can detect instruments within cannulas.
• the surgical instruments are dummy instruments that are not functional as surgical instruments.
• sensors of the model and/or at other locations in the operating room
• the simulation processing component 1 02 can record parameters such as sensor signals and times taken to complete tasks during block 31 0.
• the processing component can also provide feedback in block 31 2. This can include the simulation processing component 102 outputting signals to cause a video display of various virtual images, progress indicators, suggestions, hints, warnings, etc. regarding instrument insertion by one or more display screens of the simulation system.
• an assistant user can view the display (e.g., at a vision side cart 1 08) to assist in determining whether arms and/or surgical instruments are properly positioned during the setup procedure.
• a display of the surgical site can include the current positions of surgical instruments and other objects at the surgical site.
• the display shows captured images of the physical surgical site at the patient side cart and/or model, such as captured by an endoscope instrument or other imaging instrument of the patient side cart (e.g. ultrasound sensors, patient vital sign sensors, etc.), model cameras or sensors, and/or other visual sensors directed at the physical site.
• This setup can be used for training on inanimate/dry-lab models or live tissue models, such as porcine or cadaveric training protocols.
• the display shows a virtual environment and virtual surgical site generated by the simulation processing component and based on detected positions of surgical instruments and/or other objects at the physical surgical site.
• the surgical instrument positions can be known from sensors in their manipulator arms, and positions of other objects at the site can be known from captured images sent by camera(s).
• the simulation processing component generates the virtual environment based on these known images and positions.
• the virtual surgical instruments and objects can be displayed to appear similar to their physical counterparts (if any), or can be displayed as virtual objects with different appearances in the virtual environment.
• the virtual environment can include display of realistic surroundings such as would be seen in an actual medical procedure.
• the background of the displayed surgical site can include body walls, blood vessels, or other realistic surroundings.
• the virtual environment can include accurate representations of the physical objects at the site, while the background and surroundings of the site can be made to look realistic as an actual medical procedure (e.g., as shown in Fig. 8B).
• the simulation processing component can output feedback information such as final parameters, metrics, score, and/or other feedback related to the setup procedure.
• feedback information may also or alternatively be displayed to the trainee during the performance of or upon completion of one or more tasks or exercises (e.g., in blocks 304-312), so that the trainee can monitor his or her progress or can compare his or her performance against other persons from a novice to expert range.
• metrics can be determined from recorded parameters and can include the times expended by an assistant for various tasks during the setup procedure, as well as a summary of the placement positions of the components and instruments used in the setup.
• An evaluation and score can also be determined by the simulation processing component based on the tasks completed by the user during the setup procedure, as described in greater detail below.
• the simulation processing component can output feedback indicating the result of evaluation, such as how well the trainee performed tasks, as well as hints or instructions for performing the tasks better. Some or all of this information can be output on one or more displays or other output devices of the simulation system.
• the method checks whether the setup is complete. For example, the simulation processing component 102 can evaluate the resulting positioning of the system components and determine whether the system
• the simulation processing component causes the user to repeat the appropriate stages or blocks of the setup procedure.
• Fig. 4 is a flow diagram illustrating an example method 400 for providing a simulated surgical operation according to one or more implementations described herein. In this example, a simulated surgical operation is described for performing surgical tasks at a surgical site (virtual site and/or physical site). Method 400 can be controlled and coordinated by the simulation processing component 1 02.
• method 400 can be performed after the setup procedure of Fig. 3.
• such implementations can offer the ability to use a simulation framework on a teleoperated medical system with a separate surgeon console and patient-side cart to monitor and track progress and to display output and feedback, all under a single software and user interface (Ul) framework.
• Ul software and user interface
• Method 400 is described assuming the setup of the simulation system components has been completed as described in Fig. 3.
• the simulated setup procedure of method 400 can be performed with one or more user trainees performing the surgical tasks.
• a single trainee can be required to perform simulated surgical tasks using the surgeon console.
• multiple trainees can be simultaneously or otherwise required to perform surgical tasks for the simulated procedure as in an actual surgical procedure.
• one (surgeon) trainee may be required to control instruments using the surgeon console, while a different (assistant) trainee may be required to control an additional manual instrument at the surgical site, or perform some other assistant function (e.g., exchange instruments, adjust positions of arms on patient side cart, adjust brightness of the endoscope feed, adjust ports, pass sutures to a teleoperated instrument using a manual laparoscopic instrument, coordinate a uterine manipulator to assist the console surgeon, etc.).
• assistant function e.g., exchange instruments, adjust positions of arms on patient side cart, adjust brightness of the endoscope feed, adjust ports, pass sutures to a teleoperated instrument using a manual laparoscopic instrument, coordinate a uterine manipulator to assist the console surgeon,
• Two (or more) surgeon trainees at two (or more) surgeon consoles can divide control of tasks, exchange control of instruments, and/or provide training to each other if, in other examples.
• Advantages of the simulation system include the ability to train multiple trainees on a single system and or at the same time.
• the simulation processing component can receive options and selections for the simulation. Such selections can include the type of surgical operation to be performed, particular stages or sub-stages of the operation to be performed, the particular components and/or instruments to be used, etc. For example, the same graphical interface used to provide selections to the setup procedure of Fig. 3 can be used for the surgical operation.
• the method checks whether the simulation is using only the surgeon console, without other components such as a patient-side cart and vision side cart. For example, the user may have designated a console-only simulation in block 402. If only the surgeon console is to be used, then in block 406, the simulation processing component 1 02 outputs signals (such as simulation state signals) to display a virtual environment on the console display device.
• This virtual environment can depict the surgical site at which the user of the console will be operating.
• a 3-D virtual environment can be displayed, including virtual anatomical structures that appear similarly to corresponding real anatomical structures.
• virtual surgical instruments controlled by the user of the console are also displayed in the virtual environment.
• the particular virtual anatomical structures and virtual surgical instruments displayed in the virtual environment can be based on the selections made in block 402, where the simulation processing component can determine the appropriate environment based on the type of procedure selected and other selections made by the user.
• the simulation processing component provides other signals to the console.
• the signals can include simulation state signals such as audio data informing the user of any events or interactions occurring in the virtual environment, haptic data for outputting haptic output at the console, and/or any other applicable data.
• the provided signals can also include performance feedback information such as metric, score, instructions, alerts, hints, or other information.
• the simulation processing component 102 receives signals from the console. These signals can include directional or positioning signals based on user manipulation of controls on the console, such as hand grips, buttons, foot pedals, and other controls.
• the simulation processing component can update the virtual environment based on the signals received from the console, including moving virtual surgical instruments to correspond with the user input as if the user were controlling physical teleoperated surgical instruments.
• Simulation processing component 1 02 can also determine interactions of the virtual surgical instruments with virtual anatomical structures, e.g., according to a physics model.
• the simulation processing component records parameters such as signals and events communicated during the simulation procedure.
• signals and events can be signals sent and received in blocks 406 - 41 0, alerts or other performance feedback provided in these blocks, user input provided in these blocks, the times expended by trainee(s) to complete surgical tasks, etc.
• the simulation processing component can check whether the simulated surgical operation is complete. For example, the user can indicate that the simulation is over via input by console controls. If the operation is not complete, the method returns to block 408 to receive signals from the console and continue the simulation in the virtual environment. If the operation is complete, the method continues to block 438, described below.
• the method finds that the simulated surgical operation is not only using a surgeon console, then the method continues to block 416, in which the method checks whether the simulation will display a virtual environment.
• a virtual environment can be displayed on a console display and other displays of the simulation system, such as a display on the vision side cart, and can be a similar virtual environment as described above for block 406. If a virtual environment is to be displayed, then in block 41 8 the simulation processing component 1 02 outputs signals (such as simulation state signals) to one or more components to display the virtual environment on displays. The method then proceeds to block 41 6, detailed below.
• the method determines that a non-virtual environment is to be displayed, then images of a physical surgical site as captured by a camera are to be received and displayed.
• the method checks if an augmented display is to be output.
• An augmented display allows the display of computer-generated graphics over the images captured by a camera. If not, the process continues to block 424. If an augmented display is to be used, in block 422 the simulation processing component processes visual overlay data to be overlaid on received images.
• visual overlays can include text, graphics, and interface objects which provide feedback information such as alerts, instructions, etc.
• the visual overlays can provide indicators to instruct where arms, surgical instruments, or other components of the system should be
• the simulation processing component 1 02 receives camera data from one or more endoscopes and/or other imaging instruments (e.g., ultrasound sensors, vital signs sensors, etc.) providing images of the physical surgical site and/or simulated patient.
• an endoscope can be one of the surgical instruments provided on an arm of the patient side cart.
• the simulation processing component 1 02 also determines and outputs signals (such as simulation state signals) to display the images of the surgical site on display devices of the system, such as display screens at the surgeon console 1 04 and the vision side cart 1 08 (if being used). If no augmented images are being used, the output signals include only the camera data. Otherwise, the augmented visual overlay signals of block 422 are combined with the camera images and the combined signals for output for display. Later in the simulation, the signals are used to update the displays.
• the simulation processing component 1 02 provides any other signals to the surgeon console and to other system components.
• the signals can include audio data for outputting audio at system components, haptic data for outputting haptic output at system components, etc.
• these signals can include control signals received from the surgeon console which the simulation processing component relays to the patient side cart and/or vision side cart.
• the signals can include position signals and / or control signals received from the patient side cart which the simulation processing component relays to the surgeon console and/or to the vision side cart.
• the provided signals can also include feedback information provided to system components, such as metrics, scores, instructions, alerts, hints, or other information (and/or feedback information can be included in augmented visual data in block 422).
• the simulation processing component 102 receives signals from the surgeon console 104 and other system components, such as the patient side cart 1 06 and the vision side cart 1 08 (if present).
• control signals from the surgeon console can indicate the manipulations of controls on the console by the console user, such as master levers used to move and otherwise manipulate the arms and surgical instruments of the patient side cart 1 06.
• Signals from the patient side cart can include position signals from sensors in teleoperated arms and surgical instruments indicating the position and orientation of those arms and instruments.
• Signals from the vision side cart can include control signals from controls on that cart operated by an assistant user.
• the simulation processing component can receive signals from the model, which can include sensor signals indicating positions of manual surgical instruments inserted in or contacting the model.
• the simulation processing component records parameters such as signals and events communicated during the simulation procedure.
• signals and events can be signals sent and received in blocks 41 8 - 428, alerts or other performance feedback provided in these blocks, user input provided in these blocks, the times expended by trainee(s) to complete surgical tasks, etc.
• the blocks 406-41 0 or blocks 41 6-428 can be implemented during a performance of one or more surgical tasks and/or exercises of the simulated surgical operation.
• a user such as a trainee surgeon can perform a simulated exercise at a simulated surgical site by teleoperating the surgical instruments inserted through the cannulas in the model, for by controlling virtual instruments.
• Such exercises can include suturing, manipulating objects, etc. at a virtual or physical surgical site, or one or more other simulated tasks.
• Assistant trainees can be performing patient-side tasks simultaneously or between tasks in some implementations.
• the simulation processing component checks whether an incorrect setup has been in place during the surgical operation simulation. This can be the case when a setup procedure was simulated before the surgical operation and included incorrect selection of model ports, positioning of system components (e.g., teleoperating arms, surgical instruments, or anatomical model), or included other incorrect settings or selections. Such incorrect setup settings can have a significant effect on a following surgical operation. For example, collisions may occur between arms or instruments, ranges of motion may be blocked, limits of motion may be prematurely reached, etc. If the simulated surgical operation was performed using this incorrect setup, then in block 434 the simulation processing component outputs feedback to the one or more users of the simulation indicating the incorrect setup and how and this incorrect setup has affected the surgical operation.
• system components e.g., teleoperating arms, surgical instruments, or anatomical model
• output feedback information can indicate that a mis- positioned arm or surgical instrument caused surgical tasks to be missed or performed poorly, unintended changes to be made to simulated patient tissue, etc.
• This feedback can also indicate the correct setup to allow the users to correct any errors.
• such feedback can be output at any point during the simulated surgical operation.
• the method continues to block 436, in which the method checks whether the simulated surgical operation is complete. For example, this can be indicated by one or more users providing input to the system to indicate the operation is over, or the simulation processing component can automatically determine that the operation is over based on evaluating component positions, the images of the surgical site, etc. If the simulated operation is not complete, the method returns to block 416 to continue displaying the surgical site environment and communicate signals between system components. If the simulated operation is complete, then the method continues to block 438. In block 438, the simulation processing component outputs feedback information such as final parameters, metrics, score, and/or other feedback to the users of the simulation.
• feedback information such as final parameters, metrics, score, and/or other feedback to the users of the simulation.
• feedback information may also be displayed to trainee(s) during the performance of or upon completion of one or more exercises (e.g., in blocks 406-41 0 or 416-428), so that a trainee can monitor his or her progress or can compare his or her performance against a database of other persons from a novice to expert skill levels.
• metrics can be determined from parameters and can include the times taken by the users for various tasks during the surgical operation, a summary of the placement positions of the components and instruments used in the surgical operation, etc.
• An evaluation and/or score can also be determined by the simulation processing component based on the surgical tasks completed by the user during the surgical operation, as described below.
• the simulation processing component can output feedback indicating the result of evaluation, such as how well the trainee performed tasks, as well as hints or instructions for performing the tasks better. Some or all of this information can be output on one or more displays and/or other output devices of the simulation system.
• the system can determine metrics and perform an automatic evaluation associated with one or more trainees' performances during the simulation based on recorded parameters.
• the system can also provide real-time performance feedback to trainees during the procedure and based on evaluations, in order to provide guidance during the procedure and for later procedures.
• an evaluation can include automatically comparing the parameters recorded during these procedures (and metrics determined therefrom) to stored reference parameters and metrics for the corresponding tasks.
• the reference parameters can be correct or optimal parameters for these tasks or parameters previously-recorded during previous simulated medical procedures.
• Parameters associated with relevant skills may be evaluated to measure trainee improvement or to compare one trainee's performance parameters to corresponding parameters demonstrated by other trainees (concurrent or historic) or by persons considered to have expert skill levels. Thus a trainee's skill level in a particular parameter may be evaluated relative to peers (e.g., to determine the trainee's progress with reference to anticipated improvement) or relative to experts (e.g., to identify deviations from a high skill level).
• patient-side skills of Fig. 3 and 4 associated with actions physically near the patient's location, e.g., manipulator arm position and orientation setup, cannula port placement, docking, assisting during surgical tasks, and the like
• surgeon-side skills associated with performing surgical tasks in the surgical operation of Fig. 4, e.g., teleoperating or manually positioning an endoscopic camera and moving tissue instruments at the surgical site can be evaluated.
• An evaluation component can measure parameters associated with the tasks performed by a trainee, such as the overall completion time of all tasks, completion time of particular tasks, the position and orientation of manipulators or instruments, as well as other parameters of the actions taken by the trainee.
• an evaluation can include determining one or more scores based on predetermined criteria associated with the parameters and the comparisons, where scores can indicate a performance level or skill of the trainee based on the performance in associated tasks. For example, a score can be based on the time needed to perform one or more tasks during the procedure and/or positioning or movement of system components during the one or more tasks.
• a trainee skill level associated with a specified parameter can be automatically scored by using kinematic and/or other sensor information obtained from a teleoperated medical system, such as from sensors of manipulator arms and of surgical instruments.
• an evaluation system may use sensor information to determine positions and orientations of instruments directed during an exercise.
• the sensor information can include kinematic information from the manipulator arms obtained during the performance of blocks 304-310 (e.g., using remote center positions of the surgical instruments and setup joint values), as well as sensor information from other sensors used in the procedure.
• a kinematic setup template can be created that defines a specific effective or ideal manipulator position and orientation for a specific surgical task. Data associated with a trainee's surgical task exercise performance is compared against the template to create a performance score.
• This comparison can be used to determine if a trainee has properly selected ports for a specific surgical task exercise, if manipulator arm setup joints and other structures are properly configured to place the associated manipulator arms at a proper position and orientation, if cannula ports are properly positioned and spaced to allow effective surgical site access with minimized manipulator collision avoidance, etc.
• the ideal template information can be, for example, clustered or averaged positions, movements, and/or placements from prior performances of trainees and/or experts, or known optimal positions for instruments, arm components, etc.
• a task exercise time parameter may be measured by starting a timer at the beginning of a cannula docking exercise and stopping the timer when the system senses that all manipulators have been properly docked to an associated cannula.
• a task exercise manipulator collision avoidance parameter may be measured by comparing sensor information from each docked manipulator arm against template sensor information to determine how close a trainee has come to placing the manipulators in prescribed ideal positions and orientations or within prescribed position and orientation envelopes.
• sensor information from the manipulator arms in conjunction with known physical dimensions of an anatomical model 1 20 can be used to determine if a trainee has properly positioned the cannulas in a correct port placement pattern in the model, or if the remote center of motion for each cannula (the location on each cannula that stays stationary in space as the manipulator arm moves) is correctly positioned so as to minimize tissue trauma at a patient's body wall.
• Scores can be determined in a variety of ways. A trainee may be scored, for example, on how well port placement is selected for a selected surgical operation, or how long it takes to determine the correct port placement. Or, a trainee may be scored on how the manipulator arms are coupled to the placed cannulas
• manipulator arm collision avoidance or how long it takes a trainee to couple the manipulator arms to cannulas placed in an anatomical model.
• Metrics also may be sampled and/or determined to indicate a trainee's performance as he or she completes the exercise, and these intermediate evaluations may be plotted against a template to obtain a score. For example, historic data may indicate that specific acts should be completed in a certain order in order to most effectively complete a task, sensor data may be used to show the actual order in which a trainee performed the acts, and differences between the recommended versus actual order of acts completed is used to determine a trainee's score.
• the system can similarly record and determine parameters such as completion time of one or more tasks of the exercise and arm positions based on kinematic data for computing metrics (e.g., movement volume, errors in the exercise, economy of motion of the instruments, etc.). Performance parameters can be measured at multiple times during the performance of surgical tasks during exercises and metrics determined from those parameters.
• parameters such as completion time of one or more tasks of the exercise and arm positions based on kinematic data for computing metrics (e.g., movement volume, errors in the exercise, economy of motion of the instruments, etc.).
• Performance parameters can be measured at multiple times during the performance of surgical tasks during exercises and metrics determined from those parameters.
• a training exercise can require that the trainee pick up a ring with an operating instrument, move the ring along the pathway to a finish position (transferring the ring to another instrument controlled by a different hand as needed) without dropping the ring, while moving the camera to keep the ring and instrument tips in the center of view at all times, and while repositioning controllers to keep the trainee's hands in central controlling positions.
• the trainee can be required to drive a needle in a predetermined pathway of suture holes in the component while keeping the site in view of the camera, or suture an opening closed with spatial requirements as to the locations of the sutures.
• Patient-side tasks during a surgical operation e.g., an assistant trainee guiding one or more instruments at the surgical site, controlling accessory equipment, etc.
• Scores and/or other results of the surgical operation evaluation indicate an estimated level or skill of the trainee for the evaluated surgical exercise.
• Some implementations can provide graphical feedback, e.g., indicating how close the operating instrument end effectors are to ideal or correct positions for the surgical task, and/or ideal locations for sutures, cuts of tissue, etc.
• Some implementations can output real-time feedback during the task performance, such as indicators of correct or incorrect sutures, instrument positions, hints to the trainee, etc.
• Some real-time feedback can be instructional, indicating how instruments should be placed, moved, or positioned.
• parameters and/or scores can be determined a first time based on a particular user's or team's performance, and the same parameters and/or scores can then be recorded at a second time during the performance of the same type of procedure. These parameters can then be compared to evaluate the same user's or team's performance for particular procedure. In other examples, the parameters can be recorded for different users or teams, and compared to evaluate and compare the different users and teams. Such scoring of a trainee based on simulation procedures allows
• a trainee can be scored in relation to other trainees or in relation to historic data in order to determine how well the trainee can perform the required task and/or to evaluate the trainee's relative learning speed and effectiveness and/or determine the trainee's skill level.
• aggregate historical scoring may reveal that trainees have difficulty performing a certain task, and so training can be modified to improve a training program for that task.
• the methods of Figs. 3 and 4 can also be used to measure and evaluate performances of multiple trainees or teams at once and in various roles during a training exercise.
• the simulation system can provide training for teams of persons, such as one or more surgeons, assistants, nurses, etc.
• one or more assistant trainees can perform patient-side tasks for the methods of Fig. 3 and or Fig. 4, and a surgeon trainee can perform surgical tasks in the method of Fig. 4 while operating a console.
• Trainees other than the surgeon can use an anatomical model to practice patient-side skills (e.g., port placement, docking, system setup, camera and instrument insertion) since they will often perform these activities in the operating room.
• the team can also train their communication to perform and coordinate various tasks such as exchange instruments, adjust ports, pass sutures using a conventional laparoscopic tool, coordinate a uterine manipulator to assist the console surgeon, etc.
• the evaluation and scoring methodology described above can be extended to evaluate the performance of operating room teams in addition to individual trainees. For example, various scores can be output indicating the performance level or skill for coordinated team tasks. Such evaluation can be assisted by automated metrics to track progress and compare to historical data similarly as described above. These features can help provide proficiency standards for teams to understand their efficiency and how they can improve.
• Fig. 5 is a diagrammatic illustration of aspects of an example system 500 which can be used for automated evaluation of simulated medical procedures using simulation system 1 00.
• a medical device 502 is used, which in this example can include an input device such as a surgeon console 1 04, and/or a teleoperated medical device such as patient side cart 1 06, or other system that is capable of providing data concerning the position and/or orientation of one or more medical Instruments.
• the medical device 502 provides parameter information 504 to be stored in a memory 506 that is included in an evaluation system 508.
• evaluation system 508 can be implemented in the simulation processing component 1 02 and memory 506 can be implemented in memory 206, for example.
• Parameter information 504 can include performance parameters for a trainee's performance and/or related data, such as kinematic information or other sensor information as described above. Information 504 may be provided, for example, via an application program interface (API) interface in the simulation system. Parameter information 504 can be provided from a patient-side cart 1 06, and/or from other components of the system, such as information describing position and/or orientation of controls for a operator (such as a surgeon or trainee) on a surgeon console 1 04, vision side cart 1 08, etc.
• API application program interface
• anatomical model information 51 0 (e.g., physical dimensions, locations of possible cannula ports, location of surgical manipulators or instruments, etc.) associated with an anatomical model 1 20 is also input to the memory 506.
• Ttemplate information 51 2 can also be input into memory 506, indicating baseline, desired, and/or correct parameters and data for comparison to trainee performance parameters.
• Other parameter information can also be stored in memory 506, such as event data, e.g., recorded times related to trainee tasks and task completions, etc., and which can be collected and/or determined by other components of system 1 00 or 500 such as processor 514, sensors of the system, etc.
• memory 506 can be one or more physical memory locations that can store information that evaluation system 508 uses to carry out an evaluation of a trainee's performance.
• processor 514 can be one or more information processing devices (e.g., microprocessor(s) or other processing circuitry) that can be used to carry out the evaluation.
• the evaluation results such as one or more scores, guidance feedback, and/or other information, can be output via an output device 516, such as a visible display on a display screen or other display device, a physical printout from a printer, or other output.
• the individual exercise results may be added to historic data 520 (e.g., depending on an input at operator selection input 51 8), which in turn may be used to modify template information 51 2.
• an operator input device 51 8 enables a training system operator to input various selections related to training exercises, such as identifying a particular surgical exercise task to be carried out, and/or identifying a particular anatomical model that is being used.
• the evaluation system can automatically select the appropriate information (e.g., proper template information 51 2) to use to carry out the
• Embodiments of a evaluation system 508 may be embedded in teleoperated medical systems (e.g., with outputs displayed via the system's displays) or may be implemented, for example, on a separate small computer system, such as a laptop computer or other electronic device. Such evaluation systems may also be networked to a central database to facilitate data collection from a number of medical devices and from a population of medical personnel (e.g., surgeons) and to facilitate data and/or scoring comparison within the trainee or surgeon population.
• a central database to facilitate data collection from a number of medical devices and from a population of medical personnel (e.g., surgeons) and to facilitate data and/or scoring comparison within the trainee or surgeon population.
• Scoring aspects for training can be adapted for training in such manual tasks, such as ability to reach locations at the surgical site, instrument interference, camera position, surgeon comfort, etc.
• Automated scoring aspects can be based on sensing a position of one or more components, such as cannulas, surgical instruments, etc. by various sensors in an anatomical model and/or in other locations as described above.
• Various implementations can provide results of evaluations and/or guidance feedback to trainees' during and after the simulated procedures, indicating differences to ideal or desired positions, times, metrics and/or scores, trainee level and/or skill, and suggestions and/or instructions for the correct or desired results.
• graphical diagrams can be displayed on a display device indicating how close the manipulator arms are positioned to ideal or correct positions for the surgical task.
• some implementations can output real-time feedback during the performance of tasks, such as indicators of correct or incorrect placements and positions of surgical instruments, hints to the trainee, graphical indications of correct positioning and orientation and the acceptable range of motions and placements for particular instruments, etc.
• Some real-time feedback can be instructional, indicating where and when in the procedure that instruments should be placed or positioned.
• the system can provide tutorials to persons, demonstrating how to select ports in a model, position components, and dock manipulator arms.
• Figs. 6A and 6B are examples of training image screens 600 which can be displayed on one or more display screens of a simulation system as described above.
• the images on screen 600 can be displayed to assist and guide the user in placing the manipulator arms of the patient side cart during a setup procedure.
• display screen 600 shows an image 602 of an example
• the image 602 includes images of three manipulator arms 604.
• a user physically moves the physical arms corresponding to the arm images 604, where this motion is sensed by the simulation processing component (or other processing component) via arm sensors, and processing component causes the arm images 604 displayed on screen 600 to move in correspondence with the physical arms.
• Indications can also be displayed to indicate the status of the arms relative to desired positions in a particular stage or block of a setup procedure.
• the image 602 can indicate that the position of one or more of the arms of the physical side cart is currently incorrect or suboptimal.
• display screen 600 can display an enclosed line or border 606 around an area of the image 602 that is incorrect in position.
• the left manipulator arm 604 is shown to be incorrect in an area of its joints which are circled by the line 606.
• the line 606 thus directs the viewer's attention to the incorrectly positioned area.
• a more precise indication such as a highlight 608 of a particular joint can specifically point out the incorrect positioning.
• the highlight 608 can be of a particular color, pattern, or other distinguishing mark.
• a legend 61 0 can indicate that particular problem that indicated by highlight 608, which in this case is that the left manipulator arm (Patient Side Manipulator, PSM) 604 is not facing forward enough, e.g., toward an anatomical model. Additional explanations of the incorrect positioning can be displayed in some implementations, if desired by the user.
• PSM Patient Side Manipulator
• Fig. 6B shows another example of display screen 600, in which a different portion of the physical patient side cart is incorrectly positioned as indicated in the displayed image 602 of the patient side cart.
• a line 61 6 indicates an area of the image 602 having inaccurate positioning, which in this example is an endoscopic center arm (ECM), i.e., the center arm 604 of the patient side cart.
• ECM endoscopic center arm
• a highlight 61 8 indicates the particular joint of the arm 604 which should be corrected.
• legend 61 0 informs the viewer that highlight 61 8 indicates that the center endoscopic manipulator 604 is not positioned in a "sweet spot" which allows the instrument of that harm to provide accurate or optimal views when inserted in a patient or model for the currently selected surgical operation.
• Other types of lines, borders, and/or indicators can be displayed in other implementations, including visual, audio, haptic, or other forms.
• Virtual reality or other generated images such as those of screen 600, and/or augmented reality (AR) ghost images overlaid on camera images, can be displayed on system display devices to indicate or highlight system areas of concern or interest to users.
• manipulator arm setup joints can have incorrect positions highlighted, as described above.
• reachability limits of manipulator arms and/or instruments can be displayed.
• spatial areas where internal or external collisions between arms and/or instruments can be highlighted as zones for the user to be aware of.
• feedback information such as suggestions can be displayed on screen 600.
• text suggestions can indicate an estimated amount of movement which would cause a highlighted arm to the mood into a correct position.
• Graphical suggestions can display correct positions (e.g., in a different color or other format) on the same display 600 relative to the current, incorrect positions. Broader hints can also be provided to allow the user to exercise judgment or make decisions. Such suggestions can guide the user in a training exercise to learn the correct way to perform tasks.
• a variety of other types of feedback information can also be displayed on one or more display screens during setup procedures and surgical operations to provide guidance on system setup and accurate positioning of the manipulator arms, as well as guidance on surgical tasks.
• text information messages such as instructions and alerts can be displayed to inform users of correct or incorrect actions or positioning during procedures.
• Some feedback can be provided in other forms, such as audio output or haptic output (e.g., on motor-controlled manipulator arms and/or surgical instruments).
• the simulation system can thus provide guidance and feedback to trainees for system setup and skills during procedures such as dry-lab or wet-lab patient- side training scenarios. This can also reduce the burden on training assistants to constantly catch mistakes during simulation procedures, especially when training multiple trainees simultaneously.
• the system can also provide guidance and feedback during surgical operations and tasks, e.g., to assistant users operating the patient side cart (and other components) and/or manual instruments at an
• the simulation can include a setup procedure using the patient side cart 1 06, vision side cart 1 08, and/or any other system components needed for simulating a setup for a surgical operation.
• only the patient side cart 106 is used in the setup simulation.
• a user can set up the position of the cart, the arms of the cart, and the surgical instruments of the cart, and the simulation processing component 1 02 can read the positions of these elements and provide current simulation state and feedback as to how well the setup was performed.
• an output device such as a display screen on the patient side cart or on another component, and/or audio speakers, can be used to output representations of the current states of the setup procedure and/or feedback regarding user performance of the setup procedure.
• Some implementations of the simulated setup procedure can include the use of an anatomical model.
• An assistant trainee can set up an anatomical model on an operating table.
• the model can be an inanimate model made of a rigid material and can be approximately shaped like a portion of a human patient. This model can be set up with a particular configuration for a particular surgical operation.
• exercise devices can be placed within the model, such as beads or rings to be manipulated on wires, rubber or foam pieces of material to be sutured, cut, or otherwise manipulated with surgical instruments.
• the model can be a biological specimen such as a porcine or cadaveric model, and / or one or more biological specimens can be placed within an inanimate model.
• the assistant trainee can position a patient side cart in an operating position next to the anatomical model.
• the simulation processing component 1 02 can receive signals indicating the position of the patient side cart 1 06 and the simulation processing component can provide signals to a display such as a display screen on the patient side cart 1 06 or vision side cart 1 08 which provide indications of the current states of the procedure and feedback to the assistant trainee as to the correctness of the positioning of the patient side cart relative to the anatomical model.
• the assistant can position the arms of the patient side cart to contact the model surface at multiple points on the surface, which locates the model relative to the patient side cart. The assistant trainee can then position the arms and instruments of the patient side cart relative to the anatomical model. The assistant can select appropriate apertures in the model, place cannulas into the selected apertures of the model, and then place the surgical instruments into the appropriate cannulas.
• fully functional surgical instruments are provided on the arms of the patient side cart, and the trainee can insert the surgical instruments in ports on the model.
• one or more dummy instruments are provided on the arms of the patient side cart. These dummy instruments can include base portions of instruments which can be inserted into cannulas, but do not include end effectors such as claws, scissors, or scalpels.
• manual instruments can also be used in the medical procedure in connection with the anatomical model.
• the assistant trainee can place manual instruments in appropriate apertures of the model and the positions of these instruments are tracked by the simulation processing component.
• a vision side cart 1 08 can also or alternatively be used in the setup simulation.
• the vision side cart is used with an anatomical model and a patient side cart in the setup procedure.
• the user can view a model and/or the arms and instruments of a teleoperated medical device on a display screen of the vision side cart as captured by one or more cameras, e.g., positioned over and/or within the model.
• the user can view a display screen of the vision side cart to determine if surgical instruments have been correctly positioned, based on the view of an endoscope which provides images of the surgical site to the display screen after the user has positioned the endoscope instrument.
• a virtual environment can be displayed that models the anatomical model, physical surgical site, and/or teleoperated arms and instruments.
• the user can be required to use controls on the vision side cart to control one or more functions in the simulated procedure.
• the vision side cart is used in the setup procedure without the anatomical model and/or without the patient's side cart.
• a user can be tested in positioning of vision side cart within the operating area, and/or the positioning of the vision side cart relative to other components such as a surgeon console.
• one or more surgeon consoles can be included in a simulated setup procedure.
• the positioning of the surgeon console within the operating area can be simulated.
• a surgeon trainee can also be required to perform some tasks during the setup procedure, such as port selection on an anatomical model.
• the surgeon console can be used singly, or in conjunction with other components, such as a patient side cart, vision side cart, and/or anatomical model.
• a setup procedure can include only the surgeon console and an anatomical model, where the positioning of the components in the operating area, and the setup of instruments on the anatomical model are simulated.
• a virtual surgical site based on the physical side of the model can be displayed by the surgeon console while a user sets up manual instruments in the model.
• any other components of the simulation system can also be used singly or in conjunction with other components in a setup simulation.
• the simulation processing component can monitor the used system components and the assistant can be required to position each system component correctly within an operating area or room.
• the setup procedure can be the only simulation performed. In other implementations, a simulation of a surgical operation can be performed after the setup procedure, as in an actual surgical procedure.
• a simulated surgical operation can be performed on its own, without a simulation of a setup procedure.
• the simulation of the surgical operation can be performed using a variety of implementations.
• only the surgeon console is used and the simulation component provides a virtual environment simulation in which virtual surgical instruments are displayed manipulating virtual structures at a virtual surgical site based on user input provided by the surgeon trainee operating the console.
• the teleoperated surgical instruments can be virtual instruments displayed in a virtual environment by the simulation processing component 1 02 and controlled by the console trainee.
• One or more real manual instruments can be inserted in the model and controlled by an assistant trainee, where the manual instrument position is tracked by the simulation processing component using sensors of the model, allowing the simulation processing component to display a virtual version of the manual instrument tip, or the image-captured manual instrument, within the virtual environment alongside the virtual teleoperated surgical instruments.
• surgeon console and the patient side cart 106 are used in the surgical operation.
• a virtual environment is displayed by the simulation processing component 1 02, e.g., on a display of the surgeon console 1 04.
• the trainee operating the surgeon console provides input controlling the physical arms and surgical instruments of the patient side cart, however, the simulation processing component displays corresponding virtual versions of these instruments at the surgical site on the displays of the simulation system.
• real or dummy instruments can be used for the surgical instruments of the patient side cart.
• Figs. 7 A and 7B One example of such an implementation is shown below with respect to Figs. 7 A and 7B.
• surgeon console and patient side cart are used in a simulated surgical operation in which the physical surgical site is displayed on a display device of the surgeon console.
• real, fully functional surgical instruments are inserted in the physical model, including one or more endoscopes having cameras which capture images of the physical surgical site (or other imaging instruments).
• the images are displayed on the console display.
• the console user thus sees the actual instruments he or she is manipulating.
• the simulation system can coordinate the simulation, including record parameters, provide guidance and evaluations, etc.
• Figs. 9A and 9B One example of such an implementation is shown below with respect to Figs. 9A and 9B.
• images displayed to users can be combinations of generated virtual graphics and captured images of a physical surgical site.
• generated virtual instruments can be displayed alongside captured images of other, physical instruments, or images of physical instruments can be displayed alongside a virtual background generated to look like an actual surgical site.
• images of the physical surgical site can be combined with augmented images that are displayed over portions of the images of the physical surgical site.
• graphics can be overlaid on the image of the physical site to provide feedback information such as statuses, instructions, alerts, and other information before, during, and after the medical procedure.
• a vision side cart is included in the system. Any of the above displays of the surgical site can also be displayed on one or more displays on the vision side cart that is viewed by an assistant user.
• the surgeon console and vision side cart can display different images or views in some implementations.
• some display screens of the simulation system can display an endoscopic or camera view of the physical surgical site, while other display screens of the system can display a virtual environment that corresponds to the physical surgical site.
• a camera view of the physical site can be displayed by the vision side cart for the assistant user to operate manual instruments at the site.
• a virtual environment corresponding to the physical site can be displayed on the surgeon console.
• the vision side cart can display instructional feedback instead of or in addition to images of the surgical site displayed by the surgeon console.
• Fig. 7 A shows one example of a simulation system 700 including examples of several components described herein.
• a surgeon console 704 can provide controls for a user such as a surgeon or surgeon trainee, who sits at the console to manipulate the controls, and can also include the display screen (shown in Fig. 5B).
• a patient side cart 706 includes a number of manipulator arms 714 which include surgical instruments at the ends of the arms and which are responsive to the controls operated by the user at the surgeon console 504.
• An operating room table 722 is positioned adjacent to the patient side cart 706, and can include an anatomical model 720 which can receive the surgical instruments of the patient side cart (the model in this example is covered by the cloth over the operating table).
• a vision side cart 708 can include a display screen 726 and other components, such as electronic equipment.
• the display screen 726 displays a virtual surgical site generated by the simulation processing component.
• the model 720 can be only a surface or object having one or more apertures and not having any interior physical surgical site, where the virtual surgical site on screen 726 is not based on any physical corresponding site.
• the virtual surgical site shown on screen 726 can correspond at least partially to a physical site included within the model 720.
• the simulation processing component 1 02 can be located in one or more of the components of system 700, such as the surgeon console 704, the patient side cart 706, etc., or can be located in its own housing (not shown).
• FIG. 7B an example display screen 740 is shown that is provided on the surgeon console 704.
• two stereoscopic display screens 740 can be provided to show a 3-D view, and/or screen 740 can be a touch- responsive screen.
• display screen 740 displays a virtual environment generated by the simulation processing component 1 04.
• virtual instrument tips 742 e.g., end effectors or other end portions
• These displayed virtual instrument tips 742 also track the physical instrument tips at the patient side cart 706, which are moved within the model 720.
• Objects in the environment are also displayed, such as a thread 744 grasped by the instrument tips 742 used in suturing a portion of the object 746.
• the virtual thread is generated within the virtual
• the object 746 is generated as a new virtual object different from any physical objects within the model 720.
• the virtual thread 744 can correspond to a physical thread being manipulated within the model 720 by the physical instrument tips.
• the manipulated object 746 can also correspond to a physical object within the model 720,
• the display screen 726 on the vision side cart 708 can display the same environment displayed on the screen 740 of the console 704. This allows an assistant user to view the scene that the console user is viewing, allowing greater assistance during the operation procedure.
• an endoscope or other imaging device on the patient side cart 706 can capture images of the physical site within the model 720, and these images of the actual physical site can be displayed on display screen 740 and and/or display screen 726 instead of a generated virtual environment, or in combination with some virtual, generated objects.
• Fig. 8 is a perspective view of an example teleoperated medical device 800 that can be included in the patient side cart 1 06, similar to the patient side cart 706 shown in Fig. 7A, and an example anatomical model.
• Device 800 can include multiple manipulator arms, where each arm is coupled to one or more surgical instruments.
• each arm can be considered a teleoperated manipulator that can be coupled ("docked") to each port or cannula in a model for patient, and the manipulator controls both the cannula and the instrument that extends through the cannula and into the model or patient to reach the physical surgical site.
• one instrument 802 can be a camera or endoscope instrument and the three other instruments 804, 806, 808 can be surgical operation instruments.
• Model 820 can include multiple holes 822 and a top surface simulating a surface of a patient and through which cannulas and surgical instruments are inserted. In some implementations, one cannula and instrument can be inserted in each hole, while in other implementations, multiple cannulas and/or instruments can be inserted through a single hole (e.g., single-site).
• the model 820 can include a hollow space underneath or within, which can hold one or more physical surgical sites 824 at which physical exercises can take place manipulating exercise objects, such as flexible materials, thread, beads on wires, etc.
• Model 820 is placed on an operating table (such as table 722 described above) at a location corresponding to a patient's position on the table.
• different surgical operations may require various different port placements, and a user being trained may have to position the device 800 in one location for one surgical operation (e.g., at the foot of the operating table, simulating a location between the patient's legs) and in a second location for another surgical operation (e.g., to the side the operating table).
• Some examples of anatomical model 820 and exercises are described in copending Patent Application no. 1 3/968,253, entitled, "Anatomical Model and Method for Surgical Training," which is incorporated herein by reference in its entirety.
• Fig. 9A shows another example 900 of a simulation system including examples of several of the components described herein.
• a surgeon console 904 can provide controls for a user and can also include one or more display screens (example shown in Fig. 9B).
• a patient side cart 906 includes a number of manipulator arms 914 which include surgical instruments at the ends of the arms and which are responsive to the controls operated by the user at the surgeon console 904.
• An operating room table 922 is positioned adjacent to the patient side cart 906, and can include an anatomical model 920 similarly as described above.
• a vision side cart 908 can include a display screen 926 and other components, such as electronic equipment.
• the display screen 926 displays a virtual environment similar to the environment displayed on the screen of the surgeon console as described below in Fig. 9B.
• a component corresponding to simulation processing component 1 02 can be located in one or more of the components of system 900 similarly as described above.
• Simulation system 900 can also include manual surgical instruments, such as manual instrument 930 which is shown as a laparoscopic instrument.
• a manual instrument 930 can be guided and manipulated by an assistant user into or relative to model 920 during simulated surgical operations, while a surgeon user controls teleoperated surgical instruments using the surgeon console 904.
• the surgeon trainee an assistant trainee can train together during a simulation.
• the surgeon trainee can operate the surgeon console 904 and can operate one or more manual instruments 930, e.g. Where one or more telemanipulated instruments can be operated by the simulation system
• Some implementations can enable the simulation of a surgical operation using teleoperated instruments, and then enable the simulation of that same surgical operation using one or more manual instruments. The results of these two simulations can then be compared by the system, and results summarized.
• manual instrument 930 Some implementations including manual instrument 930 are described below.
• FIG. 9B an example display screen 940 is shown that can be provided on the surgeon console 904.
• display screen 940 displays a virtual environment generated by the simulation processing component 1 04, which can be a 2D or 3D environment, and can be displayed on a touch-responsive screen in some implementations.
• the virtual environment presents a realistic background of the interior simulating an actual patient surgical site within an actual patient, including body tissue and other body components instead of the generated exercise environment of Fig. 7B.
• Virtual instrument tips 942 are displayed and are moved on the display screen 940 based on user manipulation of the controls at console 904.
• the displayed instrument tips 942 also track the physical instrument tips of the patient side cart 906, which are moved at a physical surgical site within the model 920.
• Objects in the virtual environment are also displayed, such as a ring 944 grasped by an instrument tip or end effector 942a and moved along a track 946 following a virtual object 948 as an exercise.
• the ring 944 and wire track 946 can have physical corresponding objects provided in the model 920 which are manipulated by the physical instruments corresponding to the virtual instruments 942.
• none of the virtual objects need correspond to physical instruments, where the physical instruments of the patient side cart can be dummy instruments.
• just the virtual instruments can correspond with physical instruments, which interact with nothing in the model.
• instrument tip 942b can be grasping a virtual object 950 which has no physical corresponding object at the physical site in the model 920.
• haptic output can be provided on the controls of the surgeon console 904 using one or more actuators of the surgeon console, to provide the user the sensation of manipulating the object 950.
• an instrument tip 952 can be displayed on screen 840 and within the virtual environment.
• tip 952 can correspond to a manual instrument, such as manual instrument 930 shown in Fig. 9A, that has been inserted in the anatomical model 920.
• the physical end or tip of instrument 930 can be tracked within model 920 as described in some implementations herein, and its corresponding virtual tip 952 moved accordingly within the virtual environment of screen 940.
• the virtual tip 952 can be displayed to interact with virtual objects that correspond with physical objects in the model 920, and/or with virtual objects with no corresponding physical objects.
• Figs. 1 0A-1 0C illustrate examples related to tracking instruments within an anatomical model.
• Fig. 1 0A is a diagrammatic illustration of an example
• Manipulator arms 1 006a, 1006b, and 1 006c of the patient side cart 1002 include operating instruments that are surgical instruments 1 008a, 1 008b, and 1 008c, respectively, and manipulator arm 1 006d includes an operating instrument that is an endoscopic instrument 1 008d.
• Each of the instruments 1 008a-d is inserted in an associated cannula 1 01 0a, 1 01 0b, 101 0c, or 1 01 Od, respectively (e.g., the instrument can be a trocar within a cannula 1 01 0, or a cannula 1 01 0 can be part of a trocar 1 008 in some examples, such as for an initial insertion in the model 1 01 0).
• the cannulas 101 0 are inserted in apertures of the model 1 004.
• the endoscope instrument 1 008d can have its own sensing reference origin 1 014 relative to which it can sense the instruments and cannulas inserted in the model 1004.
• the endoscope camera can capture images of the cannulas 1 010 within the model when they are moved into the view of the camera.
• the model 1 004 can also include its own sensing system for tracking instruments inserted in (or otherwise interacting with) the model 1004.
• one or more sensors are provided within the model 1 004 to sense the cannulas 101 0.
• a camera system 1 020 is positioned on the interior base of the model 1004 to sense the interior of the modeH 004.
• a camera system 1 020 can be positioned near or within a patient side element (PSE) such as model 1004, or can be positioned at other locations of the bottom or sides of the model.
• PSE patient side element
• Camera system 1 020 thus continually capture images showing the positions of cannulas 101 0 being inserted in the model, as well as images showing the positions of surgical instruments 1008 inserted through the cannulas 1 01 0.
• Camera system 1 020 thus has its own sensing reference origin 1 022 which is the reference point for images captured by the camera system.
• sensing reference origin 1 022 is the reference point for images captured by the camera system.
• two cameras are shown in the camera system 1 020 to allow stereo triangulation in determining positions of the cannulas 1 01 0 in the anatomic model.
• camera system 1 020 can include a single camera, or other types of sensors to capture position or motion of cannulas and instruments.
• Fig. 1 0B shows example views 1 050 of a camera system within an
• anatomical model such as camera system 1 020 with an model 1 004 of Fig. 10A.
• the camera system 1020 includes two cameras, and a left view 1 052 is the view of one of the cameras, and a right view 1 054 is the view of the other camera.
• the top surface 1 056 and bottom surface 1 058 are shown, as well as apertures 1 060 in the top of the model.
• Cannulas 1 010 can be viewed inserted through specific holes of the model. In this example having two cameras, stereo triangulation can be used to accurately determine the position of each cannula 1 01 0 with reference to the origin system of the cameras.
• each cannula 1 01 0 can be distinguished from each other cannula 1 01 0 with individual marks or other characteristics.
• each cannula 101 0 can have a different exterior color to allow easy distinguishment of each cannula 1 01 0 by the sensing system 1 020.
• Fig. 1 0C shows a plan view of the external surface of the top surface of model 1 004, including holes 1 060 in the surface.
• Marks 1 070 indicate particular holes through which the cannulas 1 01 0 were detected by the camera system 1020 to have been inserted.
• Such a view can be created by the simulation processing component 1 04 based on the sensing views of the camera system 1020 and (for example) a 3-D computer-aided design (CAD) model of the anatomical model 1 004 used in visualization software.
• the view of Fig. 1 0C can be used to display port placement for the model for instructional or guidance purposes during a simulated medical procedure. For example, the view showing marked used ports can be displayed next to a similar view that displays the correct ports to be used in the particular medical procedure being simulated.
• Manual instruments can be tracked by sensing system 1 020 similarly to the cannulas and teleoperated instruments described above.
• a manual laparoscopic tool can be tracked.
• Other instruments can include a uterine manipulator, retraction tool, needle-passing tools, another manipulator arm or instrument attached to a separate component of the simulation system, or other instruments where an instrument or device separate from the patient side cart is being tracked and incorporated into the simulation environment.
• Figs. 1 1 A and 1 1 B are diagrammatic illustrations of one example of the use of an anatomical model 1 1 00 in simulated medical procedures that includes the use of both teleoperated and manual surgical instruments.
• Fig. 1 1 A is an exterior view of the model 1 1 00 and inserted instruments
• Model 1 1 B is an interior view of the model 1 1 00.
• Model 1 100 can be similar to models described above and includes apertures 1 1 02 in an upper shelf portion of the model, through which cannulas 1 104 have been inserted during a setup procedure.
• Teleoperated surgical instruments such as laparoscopic instruments and endoscopes, can be inserted in the cannulas 1 1 04.
• manual surgical instruments such as manual laparoscopic instrument 1 1 10
• sensors can be provided within the model 1 100 to sense the cannulas 1 1 04 and manual instruments such as instrument 1 1 0.
• camera system 1 1 1 2 is positioned on the interior base of the model 1 1 00 to sense the interior of the model 1 1 00 similarly as described for Fig. 1 0A. Camera system 1 1 1 2 can thus capture images showing when cannulas 1 1 04 have been inserted in the model, as well as images showing when surgical instruments have been inserted in the cannulas 1 104.
• dummy instruments can be used, e.g., which do not extend into the interior hollow portion of model 1 1 00.
• electromagnetic sensors can be used to sense cannulas and manual surgical instruments.
• other optical sensors can be used to sense cannulas and manual surgical instruments.
• Fig. 1 2 is a flow diagram illustrating an example method for using an anatomical model with both teleoperated surgical instruments and manual surgical instruments in one or more simulated surgical procedures, with reference to Figs. 1 1 A-1 1 B.
• blocks 1202 to 1 208 can be performed during a simulated setup procedure
• blocks 1 210 and 121 2 can be performed during a simulated surgical operation (block 1 21 0 can also be performed during a simulated setup procedure).
• the simulation processing component 1 02 can receive a position of the model 1 1 00 relative to the teleoperated arms of the patient side cart. For example, a teleoperated instrument at the end of one of the arms can be moved to contact (register) the model in multiple locations to establish the location of the mode in 3-D space.
• block 1 202 can be omitted or performed at a later time, e.g., the model location can be determined relative to the teleoperated instruments after docking instruments to cannulas in block 1 206 below by using sensors in the teleoperated arms.
• the simulation processing component senses the insertion of cannulas 1 1 04 in the model 1 1 00 and the simulation processing component estimates the position and orientation of the cannulas 1 1 04.
• sensors like camera system 1 1 1 2 can send signals to the simulation processing component.
• the simulation processing component senses docking and insertion of teleoperated dummy instruments in cannulas 1 1 04, e.g., based on signals from the sensors in the manipulator arms of the patient side cart. In other implementations, full surgical instruments can be docked and inserted in the cannulas 1 1 04.
• the sensors (such as camera system 1 1 1 2) and simulation processing component sense insertion of one or more manual instruments in cannulas 1 1 04, such as instrument 1 1 1 0.
• Blocks 1 206 and 1 208 can be performed in any order and/or at least partially simultaneously.
• the simulation processing component generates a virtual environment and generates virtual surgical instruments in the virtual environment corresponding to the teleoperated surgical instruments and the manual surgical instruments.
• the simulation processing component runs the simulated surgical operation based on console signals, sensed teleoperated instruments, and sensed manual surgical instruments.
• method 1 200 can use the sensing system of the anatomical model and/or the teleoperated medical device in conjunction with displaying a virtual environment.
• a generic picture of the model can be displayed without the teleoperated arms and without the exact
• the port locations placed by a user can be identified using cameras inside the model without using the teleoperated arm kinematics. Instruction can be given to the user to adjust incorrect port locations before beginning to dock the teleoperated arms to the model.
• the arm kinematics can be used to estimate the position and orientation of the model relative to the teleoperated medical device. (Other implementations can use sensors placed in or on the anatomical model to estimate the pose and location of the model relative to the patient side cart, rather than using the teleoperated arm sensors.) An entire scene of the surgical site and/or operating room can then be displayed to trainees.
• sensors in the model can track surgical instruments to provide an estimate of where the model is relative to the teleoperated device and this estimate of location can be updated during a procedure or operation to continuously provide accurate relative model location in case the model is bumped or moved by trainees.
• Figs. 1 3A and 1 3B are diagrammatic illustrations of an example of the use of an anatomical model 1300 in simulated medical procedures that include the use of manual surgical instruments. Model 1 300 can be similar to the models described above.
• Fig. 1 3A is an exterior view of the model 1 300 and an inserted instrument
• Fig. 1 3B is an interior view of the model 1 300.
• Model 1 300 includes apertures 1 302 in an upper portion of the model.
• Particular apertures 1 306 can be designated as remote centers for teleoperated instruments typically inserted through these apertures, but no cannulas for teleoperated instruments need be placed in this implementation.
• One or more cannulas, such as cannula 1 304, are inserted in model 1300 during a setup procedure in apertures where manual instruments are to be inserted.
• Manual surgical instruments, such as manual laparoscopic instrument 1 31 0, can be inserted in the cannula 1 304.
• Sensors can be provided within the model 1 300 to sense the cannulas such as cannula 1 304 and manual instruments such as instrument 1 31 0.
• camera system 1 31 2 is positioned on the interior base of the model 1300 to sense the interior of the model 1300.
• Camera system 1 31 2 can capture images showing when cannula 1 304 has been inserted in the model, as well as images showing when manual surgical instruments have been inserted in the cannula 1 304.
• the apertures 1 306 for teleoperated surgical instruments can be located by the simulation processor based on the known geometry of the model and the particular medical procedure being simulated.
• these particular aperture locations can be assumed arm remote centers, and no cannulas need to be tracked, nor do any teleoperated surgical instruments need to be docked with the model.
• this implementation can be used for simulations that include the use of manual surgical instruments and do not need use of a patient side cart, e.g., the teleoperated surgical instruments can all be virtual instruments generated in a virtual
• electromagnetic sensors can be used to sense cannulas and manual surgical instruments.
• other optical sensors can be used to sense cannulas and manual surgical instruments.
• Fig. 14 is a flow diagram illustrating an example method 1400 for using manual surgical instruments in a simulated medical procedure, with reference to Fig. 1 3.
• blocks 1402 to 1406 can be performed during a simulated setup procedure
• blocks 1408 and 1410 can be performed during a simulated surgical operation (block 1408 can also be performed during a setup procedure).
• the simulation processing component 1 02 assumes the position and orientation of the model 1 300, including assuming the positions of the surgical site and the apertures in the model (remote centers of teleoperated instruments) through which teleoperated surgical instruments would be inserted. To do this, the simulation processing component knows the geometry of the model and its apertures and physical surgical site location, as well as the particular apertures used in the surgical operation being set up.
• the simulation processing component knows the geometry of the model and its apertures and physical surgical site location, as well as the particular apertures used in the surgical operation being set up.
• the simulation processing component knows the geometry of the model and its apertures and physical surgical site location, as well as the particular aperture
• the processing component senses the insertion of cannulas 1 304 in the model 1 300 using sensors of the model 1 300 and the simulation processing component estimates the position and orientation of the cannulas 1304.
• the simulation processing component senses insertion of one or more manual instruments in cannulas 1 304, such as instrument 131 0.
• the simulation processing component generates a virtual environment and generates virtual surgical instruments in the virtual environment corresponding to teleoperated surgical instruments and the manual surgical instruments. The relative position between the manual surgical instruments and the assumed aperture locations for the teleoperated instruments in the model 1 300 enable relative positioning of these instruments in the virtual environment.
• the simulation processing component runs the simulated surgical operation and updates the virtual
• console signals to move the virtual teleoperated surgical instruments
• sensed manual surgical instruments to move the virtual teleoperated surgical instruments
• a simulation processing component can interact with a surgeon console (e.g., with master controllers) and/or a patient side cart (e.g., with slave manipulator arms and instruments).
• the console master can drive the slave arms with or without instruments (or with dummy instruments) on the patient side cart.
• the simulation system can simulate and provide guidance and other feedback on system setup and accurate positioning of the manipulator arms prior to a simulated surgical operation. This can be used to provide standardized and consistent training to surgeons using inanimate training exercises or wet-lab exercises.
• a simulated surgical operation can follow the simulated setup procedure, which can allow the entire medical procedure to be simulated. This allows trainees to see the consequences of improperly-performed tasks. For example, improper or incorrect tasks performed in a setup procedure may have repercussions in a following surgical operation, and the simulation system herein simulates this entire effect to allow trainees to learn and improve.
• the simulation system can display a virtual environment (e.g., ignore endoscope feed and instruments if installed), combined or augmented environment (e.g., endoscope feed with generated graphical visual overlays or virtual
• virtual or augmented images can be output through display systems (such as using TilePro from Intuitive Surgical, Inc.) on the surgeon console during any training exercise to provide instruction or performance metrics.
• VR virtual reality
• AR augmented reality
• ghost images overlaid on camera images
• system display devices can indicate or highlight system areas of concern or interest, such as patient cart setup joints having incorrect positions highlighted, reachability limits of instruments displayed, and/or internal/external collisions. This can reduce the burden on training assistants to catch mistakes during training procedures.
• the system can record the kinematics and events of the master console(s) and teleoperated slave medical device(s) during completion of inanimate training or wet-lab exercises by a console surgeon to compute training metrics and display such metrics using a similar interface as purely virtual training exercises.
• the system can record the kinematics and events of teleoperated devices during completion of patient-side exercises and setup procedures using actual instruments, and compute training metrics and display such metrics using a similar interface as purely virtual training exercises. Further, the system can record the kinematics and events of the masters and slaves during completion of exercise modules on porcine models (for example, during offsite training) to provide metrics and display such metrics.
• porcine models for example, during offsite training
• All data from any training environment or configuration can be recorded and stored locally or remotely in the same way to improve accessibility of data, monitoring of surgeon training and performance of trainee personnel during simulated procedures, standardization of exercises, and feedback to surgeon during training (e.g., to improve surgeon training).
• the simulation system can centralize most training content to one software platform separate from the system
• Teleoperated medical device surgery offers an unprecedented ability to record, track, and monitor surgery and surgeon training unlike any pre-existing form of surgery. Implementations described herein can make effective use of this capability and the data that can be harvested e.g., for simulation and training purposes. Some additional advantages of various implementations of teleoperated and non-teleoperated systems can include the following. Features described herein can centralize user training and evaluation on a single system, e.g., a single teleoperated medical system. Some systems can provide the ability to use a single simulation framework on teleoperated medical system with a separate surgeon console and patient side cart to monitor and track progress and to display feedback all under a single software and user interface (Ul) framework.
• Ul software and user interface
• Some systems can provide the ability to provide augmented reality output and feedback during wet-lab or porcine model exercises and dry-lab exercises using a teleoperated medical system. Some systems can provide the ability to combine training data using a single software and hardware architecture used for various types of training exercises including virtual environment exercises, inanimate exercises, wet-lab or porcine models, etc.
• One or more features can allow any training exercise or offsite lab exercise to be conducted using the single simulation architecture to provide real-time (during a procedure) and end-of- exercise metrics to guide training and learning.
• Features herein can improve accessibility of training data, especially for tasks not normally implemented on a simulator system.
• Features can improve standardization of training since the system can be used for several types of training tasks.
• Features can improve surgeon training at offsite training labs by quantifying and delivering feedback in addition to that provided by training personnel to help trainees learn.
• features can help training personnel to better manage multiple surgeon trainings simultaneously (e.g., dual surgeon console trainings).
• features can improve surgeon training conducted by clinical sales representatives (CSRs) or other instructors by simulating tasks performed during setup procedures, and by providing feedback determined by the system and displayed in real-time for setup exercises.
• CSRs clinical sales representatives
• features described herein can expand the capability of teleoperated and non-teleoperated medical simulation systems to support inanimate training exercises, wet-lab training scenarios, and VR-based training exercises.
• a single simulation system can administer and record all training performed by surgeons, e.g., with their CSRs, with dedicated training specialists (TSs), or independently.
• the simulation system can be used to simulate all interactions with the system outside of actual surgery.
• blocks described in the various methods herein can be performed in a different order than shown and/or simultaneously (partially or completely) with other blocks in the same method (or other methods), where appropriate.
• blocks can occur multiple times, in a different order, and/or at different times in the methods.
• spatially relative terms such as “beneath”, “below”, “lower”, “above”, “upper”, “proximal”, “distal”, and the like— may be used to describe one element's or feature's relationship to another element or feature as illustrated in the figures.
• These spatially relative terms are intended to encompass different positions (i.e., locations) and orientations (i.e., rotational placements) of a device in use or operation in addition to the position and orientation shown in the figures. For example, if a device in the figures is turned over, elements described as “below” or “beneath” other elements or features would then be “above” or “over” the other elements or features.
• the exemplary term “below” can encompass both positions and orientations of above and below.
• a device may be otherwise oriented (rotated 90 degrees or at other orientations) and the spatially relative descriptors used herein interpreted accordingly.
• descriptions of movement along and around various axes includes various special device positions and orientations.
• the singular forms “a”, “an”, and “the” are intended to include the plural forms as well, unless the context indicates otherwise.
• Components described as coupled may be electrically or mechanically directly coupled, or they may be indirectly coupled via one or more intermediate components.

Landscapes

• Engineering & Computer Science (AREA)
• Physics & Mathematics (AREA)
• General Physics & Mathematics (AREA)
• Health & Medical Sciences (AREA)
• Theoretical Computer Science (AREA)
• Business, Economics & Management (AREA)
• Educational Technology (AREA)
• Educational Administration (AREA)
• General Health & Medical Sciences (AREA)
• Medical Informatics (AREA)
• Computational Mathematics (AREA)
• Mathematical Optimization (AREA)
• Mathematical Physics (AREA)
• Pure & Applied Mathematics (AREA)
• Mathematical Analysis (AREA)
• Algebra (AREA)
• Medicinal Chemistry (AREA)
• Chemical & Material Sciences (AREA)
• Pulmonology (AREA)
• Radiology & Medical Imaging (AREA)
• Manipulator (AREA)
• Instructional Devices (AREA)
• Surgery (AREA)
• Life Sciences & Earth Sciences (AREA)
• Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
• Robotics (AREA)
• Biomedical Technology (AREA)
• Heart & Thoracic Surgery (AREA)
• Molecular Biology (AREA)
• Animal Behavior & Ethology (AREA)
• Public Health (AREA)
• Veterinary Medicine (AREA)

Priority Applications (10)

Applications Claiming Priority (2)

Related Child Applications (2)

Publications (1)

Family

ID=53403746

Family Applications (1)

Country Status (6)

Cited By (29)

Families Citing this family (294)

Citations (6)

Family Cites Families (112)

• 2014
  • 2014-12-19 KR KR1020167019144A patent/KR102366023B1/ko active Active
  • 2014-12-19 JP JP2016541234A patent/JP6659547B2/ja active Active
  • 2014-12-19 WO PCT/US2014/071521 patent/WO2015095715A1/en not_active Ceased
  • 2014-12-19 CN CN201480076076.0A patent/CN106030683B/zh active Active
  • 2014-12-19 KR KR1020227005342A patent/KR102405656B1/ko active Active
  • 2014-12-19 US US15/106,254 patent/US10510267B2/en active Active
  • 2014-12-19 CN CN202011072053.6A patent/CN112201131B/zh active Active
  • 2014-12-19 EP EP14871282.1A patent/EP3084747B1/en active Active
  • 2014-12-19 EP EP22205491.8A patent/EP4184483B1/en active Active
• 2019
  • 2019-09-26 US US16/584,564 patent/US11468791B2/en active Active
• 2020
  • 2020-02-06 JP JP2020018974A patent/JP6916322B2/ja active Active
• 2021
  • 2021-07-15 JP JP2021117360A patent/JP7195385B2/ja active Active
• 2022
  • 2022-09-02 US US17/902,678 patent/US12456392B2/en active Active