BR112012011323B1 - minimally invasive surgical system - Google Patents

Abstract

method and system for detecting the presence of hand in a minimally invasive surgical system. the present invention relates to a minimally invasive surgical system, a hand tracking system tracks a location of a sensor element mounted on part of a human hand. a system control parameter is generated based on the location of the human hand. the operation of the minimally invasive surgical system is controlled using the system control parameter. thus, the minimally invasive surgical system includes a hand tracking system. the hand tracking system tracks a location of part of a human hand. a controller supporting the hand tracking system converts the location into a system control parameter, and injects a command based on the system control parameter into the minimally invasive surgical system.

Description

Descriptive Report of the Invention Patent for MINIMUM INVASIVE SURGICAL SYSTEM.

BACKGROUND

Field of the Invention [001] The present invention relates to the control of minimally invasive surgical systems and is related, more particularly, to the use of a surgeon's hand movement in the control of a minimally invasive surgical system.

Related Technique [002] Methods and techniques for tracking hand positions and gestures are known. For example, some video game controllers use hand track input. For example, the Nintendo Wii® gaming platform supports wireless remote controls for position and orientation reading (Wii is a registered trademark of Nintendo of America Inc, Redmond, Washington, USA). The use of gestures and other physical movements, such as shaking a stick or waving a magic wand, represents the fundamental game element of this platform. The Sony Playstation Move has similar features to the Nintendo Wii® gaming platform.

[003] The CyberGlove® wireless motion capture data glove from CyberGlove Systems includes eighteen data sensors with two fold sensors on each finger, four abduction sensors and sensors that measure thumb crossing, palm arch, flexion of the wrist and abduction of the wrist. (CyberGlove® is a registered trademark of CyberGlove Systems LLC of San Jose, CA). When a three-dimensional tracking system is used with the CyberGlove® motion capture data glove, information on x, y, z, yaw, pitch, OK, rolling position and hand orientation is available. The motion capture system for the CyberGlove® motion capture data glove was used in evaluating digital prototypes,

Petition 870190091580, of 9/13/2019, p. 5/92

2/81 biomechanics of virtual reality and animation.

[004] Another data glove with forty sensors is the ShapeHand data glove from Measurand Inc. The ShapeClaw portable, lightweight hand motion capture system from Measurand Inc. includes a flexible tape system that captures the movement of the indicator and thumb, along with the position and orientation of the hand and forearm in space.

[005] In In-Cheol Kim and Sung-Il Chien, Analysis of 3D Hand

Trajectory Gestures Using Stroke-Based Composite Hidden Markov Models, Applied Intelligence, Vol.15 No. 2, p.131-143, SeptemberOctober 2001, Kim and Chien explore the use of three-dimensional trajectory input with a Polhemus sensor for gesture recognition. Kim and Chien propose this form of entry because three-dimensional trajectories offer a greater power of discrimination than two-dimensional gestures, which are predominantly used in video-based approaches. For their experiments Kim and Chien make use of a Polhemus magnetic position tracking sensor attached to the back of a Fakespace PinchGlove. PinchGlove provides a means for the user to signal the beginning and end of a gesture, while the Polhemus sensor captures the three-dimensional trajectory of the user's hand.

[006] In Elena Sanchez-Nielsen, et al., Hand Gesture

Recognition for Human-Machine Interaction, Journal of WSCG, Vol. 12, No. 1-3, ISSN 1213-6972, WSCG'2004, 2-6 February 2003, Plzen, Czech Republic, proposes a vision system in real time for application in visual interaction environments through hand gesture recognition using generic hardware and low-cost sensors, such as a personal computer and a webcam. In Pragati Garg, et al., Vision Based Hand Gesture Recognition 49 World Academy of Science, Engineering and Technology, 972-977 (2009), presented

Petition 870190091580, of 9/13/2019, p. 6/92

3/81 a review of vision-based hand gesture recognition. One conclusion presented was that most approaches are based on several underlying considerations that may be appropriate in a controlled laboratory environment, but that are not generalized in arbitrary environments. The authors stated that computer vision methods for hand gesture interfaces must surpass current performance in terms of robustness and speed to achieve interactivity and usability. In the medical field, gesture recognition was considered to leaf through sterile X-ray images. See Juan P. Wachs, et al., A Gesture-based Tool for Sterile Browsing of Radiology Images, Journal of the American Medical Informatics Association (2008; 15: 321-323, DOI 10.1197 / jamia.M24). SUMMARY [007] In one aspect, a hand tracking system in a minimally invasive surgical system tracks a location on part of a human hand. A minimally invasive surgical system control parameter is generated based on the location of the part of the human hand. The operation of the minimally invasive surgical system is controlled using the system control parameter.

[008] In one aspect, sensor elements mounted on part of a human hand are tracked to obtain locations on the part of the human hand. A position and orientation for a control point are generated based on the location. The teleoperation of a device in a minimally invasive surgical system is controlled based on the position and orientation of the control point. In one aspect, the device is a teleoperated slave surgical instrument. In another aspect, the device is a virtual substitute presented in a video image of a surgical field. Examples of a virtual substitute include a virtual slave surgical instrument, a virtual hand

Petition 870190091580, of 9/13/2019, p. 7/92

4/81 and a virtual remote medicine device (telestration).

[009] In an additional aspect, a handle closing parameter is generated in addition to the position and orientation of the control point. The handle of a terminal effector of the teleoperated slave surgical instrument is controlled based on the handle closing parameter.

[0010] In another aspect, the system control parameter is a position and orientation of a control point used for the teleoperation of the slave surgical instrument. In yet another aspect, the system control parameter is determined from two hands. The system control parameter is a position and orientation from a control point to one of the two hands, and a position and orientation from a control point to the other of the two hands. The control points are used for the teleoperation of an endoscopic camera manipulator in the minimally invasive surgical system.

[0011] In yet another aspect, sensor elements mounted on part of a second human hand are tracked, in addition to the sensor elements on the part of the human hand. A position and orientation for a second control point are generated based on the location of the second human hand part. In this respect, both the control point and the second control point are used to control teleoperation.

[0012] In yet another aspect, sensor elements mounted on fingers of a human hand are tracked. A movement between the fingers is determined, and the orientation of a teleoperated slave surgical instrument in a minimally invasive surgical system is controlled based on the movement.

[0013] When the movement is a first movement, the control includes rolling one end of a surgical instrument handle

Petition 870190091580, of 9/13/2019, p. 8/92

5/81 slave around your pointing direction. When the movement is a second movement different from the first movement, the control includes the yaw movement of the slave surgical instrument's handle.

[0014] A minimally invasive surgical system includes a hand tracking system and a controller coupled to the hand tracking system. The hand tracking system tracks locations of a plurality of sensor elements mounted on part of a human hand. The controller transforms the locations into a position and orientation from a control point. The controller sends a command to move a device in the minimally invasive surgical system based on the control point. Once again, in one aspect, the device is a teleoperated slave surgical instrument, while in another aspect, the device is a virtual substitute presented in a video image of a surgical field.

[0015] In one aspect, the system also includes a master finger tracking device including the plurality of tracking sensors. The master finger tracking device also includes a compressible body, a first finger loop attached to the compressible body and a second finger loop attached to the compressible body. A first tracking sensor in the plurality of tracking sensors is attached to the first finger loop. A second tracking sensor in the plurality of tracking sensors is attached to the second finger loop.

[0016] Therefore, in one aspect, a minimally invasive surgical system includes a master finger tracking device. The master finger tracking device includes a compressible body, a first finger loop attached to the compressible body and a second finger loop attached to the compressible body. A first tracking sensor is attached to the first finger loop. A second tracking sensor is attached to the second finger loop.

Petition 870190091580, of 9/13/2019, p. 9/92

6/81 [0017] The compressible body includes a first end, a second end and an external outer surface. The outer outer surface includes a first portion that extends between the first and second ends and a second portion, opposite and away from the first portion, that extends between the first and second ends.

[0018] The compressible body also has a length. The length is selected to limit a separation between a first finger and a second finger on the human hand.

[0019] The first finger loop is attached to the compressible body adjacent to the first end and extends around the first portion of the outer outer surface. With the placement of the first finger loop on a first finger of a human hand, a first part of the first portion of the outer outer surface contacts the first finger.

[0020] The second finger loop is attached to the compressible body adjacent to the second end and extends around the first portion of the outer outer surface. By placing the second finger loop on a second finger of the human hand, a second part of the first portion of the outer outer surface contacts the second finger. With the movement of the first and second fingers towards each other, the compressible body is positioned between the two fingers, so that the compressible body creates resistance to movement.

[0021] A thickness of the compressible body is selected so that when a tip of the first finger only touches a tip of the second finger, the compressible body is less than completely compressed. The compressible body is configured to provide haptic feedback corresponding to the clamping force of a terminal effector of a teleoperated slave surgical instrument.

[0022] In one respect, the first and second tracking sensors

Petition 870190091580, of 9/13/2019, p. 10/92

7/81 are passive electromagnetic sensors. In an additional aspect, each passive electromagnetic tracking sensor has six degrees of freedom.

[0023] One method of using the master finger tracking device includes tracking a first location of a sensor mounted on a first finger of a human hand and a second location of another sensor mounted on a second finger. Each location has N degrees of freedom, where N is an integer greater than zero. The first location and the second location are mapped to a checkpoint location. The control point location has six degrees of freedom. The six degrees of freedom are less than or equal to 2 * N degrees of freedom. The first location and the second location are also mapped to a parameter with a single degree of freedom. The teleoperation of a slave surgical instrument in a minimally invasive surgical system is controlled based on the location of the control point and the parameter.

[0024] In a first aspect, the parameter is a closing distance of the handle. In a second aspect, the parameter comprises an orientation. In another aspect, N is six, while in a different aspect, N is five.

[0025] In yet another aspect, sensor elements mounted on part of a human hand are tracked to obtain a plurality of locations on the part of the human hand. A hand gesture from a plurality of known hand gestures is selected based on the plurality of locations. The operation of a minimally invasive surgical system is controlled based on the hand gesture.

[0026] The hand gesture can be any of a hand gesture pose, a hand gesture trajectory or a combination of a hand gesture pose and a hand gesture trajectory. When the gesture

Petition 870190091580, of 9/13/2019, p. 11/92

8/81 hand is a hand gesture pose and the plurality of known hand gestures includes a plurality of known hand gesture poses, a minimally invasive surgical system user interface is controlled based on the hand gesture pose .

[0027] Furthermore, in one aspect, when the hand gesture is a hand gesture pose, the hand gesture selection includes the generation of a set of characteristics observed from the plurality of tracked locations. The set of characteristics observed is compared with sets of characteristics of the plurality of known hand gesture poses. One of the known hand gestures is selected as the hand gesture pose. The selected known hand gesture pose is mapped to a system command, and the system command is triggered in the minimally invasive surgical system.

[0028] In yet another aspect, when the hand gesture includes a hand gesture trajectory, the user interface of the minimally invasive surgical system is controlled based on the trajectory of the hand gesture.

[0029] In the minimally invasive surgical system with the hand tracking systems and the controller, the controller turns the locations into a hand gesture. The controller sends a command to modify a minimally invasive operating system operation mode based on the hand gesture.

[0030] In yet another aspect, a sensor element mounted on part of a human being is tracked to obtain a location of the part of the human hand. Based on location, the method determines whether a position of the human hand part is within a threshold distance from a position of a master tool handle of a minimally invasive surgical system. The operation of the minimally invasive surgical system is controlled based on a result

Petition 870190091580, of 9/13/2019, p. 12/92

9/81 of the determination. In one aspect, the teleoperation of a teleoperated slave surgical instrument coupled to the handle of the master tool is controlled based on a result of the determination. In another aspect, the monitor of a user interface or the monitor of a substitute visual is controlled based on the result of the determination. [0031] In one aspect, the position of the human hand part is specified by a position of the control point. In another aspect, the position of the part of the human hand is a position of the index finger. [0032] A minimally invasive surgical system includes a hand tracking system. The hand tracking system tracks a location of part of a human hand. A controller uses location to determine whether a surgeon's hand is close enough to a master tool handle to allow for a particular operation of the minimally invasive surgical system. [0033] A minimally invasive surgical system also includes a controller coupled to the hand tracking system. The controller converts the location into a system control parameter and injects a command based on the system control parameter into the minimally invasive surgical system.

BRIEF DESCRIPTION OF THE DRAWINGS [0034] Figure 1 is a high-level diagrammatic view of a minimally invasive teleoperated surgical system including a hand tracking system.

[0035] Figures 2A to 2G are examples of various configurations of a master tool handle with hand tracking used to control a teleoperated slave surgical instrument from the minimally invasive teleoperated surgical system of figure 1.

[0036] Figures 3A to 3D are examples of hand gesture poses used to control system modes in the minimally invasive teleoperated surgical system of figure 1.

Petition 870190091580, of 9/13/2019, p. 13/92

10/81 [0037] Figures 4A to 4C are examples of hand gesture trajectories that are also used to control system modes in the minimally invasive teleoperated surgical system of figure 1.

[0038] Figure 5 is an illustration of placing fiducial markers for hand tracking in a camera-based tracking system.

[0039] Figures 6A and 6B are more detailed diagrams of the surgeon's console in figure 1 and include examples of coordinate systems used in hand tracking by the minimally invasive teleoperated surgical system in figure 1.

[0040] Figure 7 is a more detailed illustration of a master tool handle used in the hand and of the locations and coordinate systems used in hand tracking by the minimally invasive teleoperated surgical system in Figure 1.

[0041] Figure 8 is a process flow chart of a process used in the screening system to track fingers and used to generate data for the teleoperation of a slave surgical instrument in the minimally invasive teleoperated surgical system of figure 1.

[0042] Figure 9 is a more detailed process flow chart of the MAP LOCATION DATA process to CONTROL POINT AND HANDLE PARAMETER of figure 8.

[0043] Figure 10 is a process flow chart of a process used in the tracking system to recognize hand gesture poses and hand gesture trajectories.

[0044] Figure 11 is a process flow chart of a process used in the screening system to detect the presence of hand.

[0045] Figure 12 is an illustration of an example of a master finger tracking device.

[0046] Figure 13 is an illustration of a video image, presented on a display device, including a visual

Petition 870190091580, of 9/13/2019, p. 14/92

11/81 substitute, which, in this example, includes a virtual phantom instrument, and a teleoperated slave surgical instrument.

[0047] Figure 14 is an illustration of a video image, presented on a display device, including substitute visuals, which, in this example, includes a pair of virtual hands, and teleoperated slave surgical instruments.

[0048] Figure 15 is an illustration of a video image, presented on a display device, including substitute visuals, which, in this example, includes a virtual remote medicine device and a virtual ghost instrument, and teleoperated slave surgical instruments.

[0049] In the drawings, the first digit of a three-digit reference number indicates the figure number of the figure in which the element with that reference number first appeared, and the first two digits of a four-digit reference number indicate the figure number of the figure where the element with that reference number first appeared.

DETAILED DESCRIPTION [0050] As used here, a location includes a position and an orientation.

[0051] As used here, a hand gesture, sometimes called a gesture, includes a hand gesture pose, a hand gesture path and a combination of a hand gesture pose and a hand gesture path.

[0052] Aspects of this invention increase the ability to control minimally invasive surgical systems, for example, the da Vinci® minimally invasive teleoperated surgical system marketed by Intuitive Surgical, Inc. of Sunnyvale, California, by using hand location information in control of the minimally invasive surgical system. A measured location of one or more

Petition 870190091580, of 9/13/2019, p. 15/92

12/81 fingers is used to determine a system control parameter which, in turn, is used to trigger a system command in the surgical system. System commands depend on the location of the person whose hand location is being tracked, that is, whether the person is on a surgeon's console.

[0053] When the measured locations are four fingers of a person who is not on the surgeon's console, the system commands include a command to change the orientation of a part of a teleoperated slave surgical instrument based on a combination of orientation of the hand and relative movement of two fingers of a hand, and a command to move a tip of a teleoperated slave surgical instrument, so that the movement of the tip follows the movement of a part of the hand. When the measured locations are four fingers of a person's hand on the surgeon console, the system commands include commands that allow or prevent the movement of a slave surgical instrument from continuing to follow the movement of a master tool handle. When the measured locations are four fingers of a person who is not on the surgeon's console or four fingers of a person on the surgeon's console, system commands include commanding the system, or a part of the system, to perform a action based on a hand gesture pose, and command the system, or part of the system, to perform an action based on a hand gesture trajectory.

[0054] Figure 1 is a high-level diagrammatic view of a minimally invasive teleoperated surgical system 100, for example, the da Vinci® Surgical System, including a hand tracking system. There are other parts, cables and others associated with the da Vinci® Surgical System, but these are not illustrated in figure 1 to avoid deviations from exposure. Additional information regarding surgical systems

Petition 870190091580, of 9/13/2019, p. 16/92

Minimally invasive 13/81 can be found, for example, in US patent application No. 11 / 762,165 (filed on June 13, 2007, which features a Minimally Invasive Surgical System), and in US patent No. 6,331,181 ( issued on December 18, 2001, which presents Surgical Robotic Tools, Data Architecture and Usage), both of which are incorporated herein by reference. See also, for example, US patents No. 7,155,315 (filed on December 12, 2005; which features a Referenced Control in a Minimally Invasive Surgical Apparatus) and 7,574,250 (filed on February 4, 2005) 2003; which presents an Image Displacement Apparatus and Method for a Telerobotic System), both of which are incorporated herein by reference.

[0055] In this example, system 100 includes a cart 110 with a plurality of handlers. Each manipulator and the teleoperated slave surgical instrument controlled by that manipulator can be coupled and decoupled from master tool manipulators on the 185 surgeon console and, in addition, they can be coupled and decoupled from a non-energized and mechanically non-energized master tracking handle base 170, sometimes called the master finger tracking handle 170.

[0056] A stereoscopic endoscope 112 mounted on the manipulator 113 provides an image of the surgical field 103 within the patient 111, which is displayed on monitor 187 and on the monitor of the surgeon console 185. The image includes images of any of the slave surgical devices on the field of view of the stereoscopic endoscope 112. The interactions between the master tool manipulators on the surgeon console 185, the slave surgical devices and the stereoscopic endoscope 112 are the same as in a known system and are therefore known to those skilled in the art.

Petition 870190091580, of 9/13/2019, p. 17/92

14/81 [0057] In one aspect, the surgeon 181 moves at least one finger from the surgeon's hand, which in turn causes a sensor on the master finger tracking handle 170 to change location. The hand tracking transmitter 175 provides a field so that the new finger position and orientation are read by the master finger tracking handle 170. The new position and orientation read are provided to the hand tracking controller 130.

[0058] In one aspect, as explained more fully below, the hand tracking controller 130 maps the position and orientation read at a control point position and a control point orientation in a surgeon 181 eye coordinate system The hand tracking controller 130 sends this location information to the system controller 140 which, in turn, sends a system command to the teleoperated slave surgical instrument coupled to the master finger tracking handle 170. As explained more fully below, using the master finger tracking handle 170, the surgeon 181 can control, for example, the handle of a terminal effector of the teleoperated slave surgical instrument, as well as the bearing and yaw of a handle coupled to the terminal effector.

[0059] In another aspect, hand tracking of at least part of the hand of the surgeon 181 or the hand of the surgeon 180 is used by the hand tracking controller 130 to determine whether a hand gesture pose is performed by the surgeon, or a combination of a hand gesture pose and a hand gesture trajectory is done by the surgeon. Each hand gesture pose and each trajectory combined with a hand gesture pose is mapped to a different system command. System commands control, for example, changes in system mode and control other aspects of the minimally invasive surgical system 100.

Petition 870190091580, of 9/13/2019, p. 18/92

15/81 [0060] For example, instead of using pedals and foot switches, as in a known minimally invasive surgical system, a hand gesture, a hand gesture pose or a hand gesture trajectory is used , (i) to start tracking movements between the master tool handle and the associated teleoperated slave surgical instrument, (ii) to activate the master clutch (which decouples the master control from the slave instrument), (iii) to control the endoscopic camera ( which allows the master to control the movement or characteristics of the endoscope, such as focus or electronic zoom), (iv) to exchange the robotic arm (which exchanges a particular master control between two slave instruments), and (v) to exchange TILEPRO® ( that alternates the display of a window and auxiliary video on the surgeon's monitor) (TILEPRO is a registered trademark of Intuitive Surgical, Inc. of Sunnyvale, CA, USA).

[0061] When there are only two master tool handles in system 100, and surgeon 180 wants to control the movement of a slave surgical instrument other than the two teleoperated slave surgical instruments coupled to the two master tool handles, the surgeon can lock one or both teleoperated slave surgical instruments, instead of using a first hand gesture. The surgeon then associates one or both of the master tool handles with other slave surgical instruments attached to the manipulating arms using a different hand gesture which, in this implementation, makes an exchange association between the master tool handle and another teleoperated slave surgical instrument. . Surgeon 181 performs an equivalent procedure when there are only two master finger tracking handles on the 100 system.

[0062] In yet another aspect, a hand tracking unit

186 mounted on the surgeon console 185 tracks at least part of the surgeon's hand 180 and sends the location information read to the

Petition 870190091580, of 9/13/2019, p. 19/92

16/81 hand tracking controller 130. Hand tracking controller 130 determines when the surgeon's hand is close enough to the master tool handle to allow the system to track, for example, the movement of the slave surgical instrument following the movement of the master tool handle. As explained more fully below, in one aspect, the hand tracking controller 130 determines the position of the surgeon's hand and the position of the corresponding master tool handle. If the difference in the two positions is within a predetermined distance, for example, less than a threshold separation, monitoring is allowed, otherwise monitoring is inhibited. Thus, the distance is used as a measure of the presence of the surgeon's hand with respect to the master tool handle on the surgeon console 185. In another aspect, when the position of the surgeon's hand with respect to the position of the master tool handle is less that the threshold separation, the display of a user interface on a monitor is inhibited, for example, the display device is turned off. Conversely, when the position of the surgeon's hand relative to the position of the master tool handle is greater than the threshold separation, the user interface is displayed on the display device, for example, it is turned on.

[0063] Detecting the presence of the surgeon's hand is a long-standing problem. Presence detection has been attempted many times using different contact reading technologies, such as capacitive switches, pressure sensors and mechanical switches. However, these approaches are inherently problematic, because surgeons have different preferences as to how and where to hold the master tool handle. Distance is used as a measure of presence is advantageous, because this type of presence detection allows the surgeon to lightly touch the master tool handle and then momentarily break physical contact to

Petition 870190091580, of 9/13/2019, p. 20/92

17/81 adjust the master tool handle, but do not restrict how the surgeon holds the master tool handle with your fingers. Control of Surgical Instrument Through Hand Tracking [0064] An example of a non-energized and mechanically unfounded master finger tracking handle 270, sometimes called a master finger tracking handle 270, is illustrated in figures 2A to 2D in different configurations, which are more fully described below. The master finger tracking handle 270 includes sensors mounted on fingers 211, 212, sometimes called sensors mounted on fingers and thumb 211, 212, which independently track the location (position and orientation in an example) of each of a tip of the index finger 292B and a tip of the thumb 292A, that is, track the location of two fingers of the surgeon's hand. Thus, the location of the hand itself is tracked, as opposed to tracking the location of master tool handles in a known minimally invasive surgical system.

[0065] In one aspect, the sensors track six degrees of freedom (three of translation and three of rotation) for each finger of the hand on which the sensor is mounted. In another aspect, the sensors track five degrees of freedom (three of translation and two of rotation) for each finger of the hand on which the sensor is mounted.

[0066] In yet another aspect, the sensors track three degrees of freedom (three of translation) for each finger of the hand on which the sensor is mounted. When two fingers are each tracked with three degrees of freedom, the full six degrees of freedom of translation is sufficient to control a slave surgical instrument that does not include a wrist mechanism.

[0067] A foam padded connector 210 is connected between the sensors mounted on the fingers and thumb 211, 212. The connector 210 restricts the thumb 292A and the indicator 292B, that is, the fingers of the

Petition 870190091580, of 9/13/2019, p. 21/92

18/81 hand 291R, within a fixed distance, that is, there is a maximum separation distance between the fingers of the hand 291R on which the master finger tracking handle 270 is mounted. When the thumb 292A and index finger 292B are moved from maximum separation (figure 2A) to a completely closed configuration (figure 2D), the padding provides positive feedback to help surgeon 181 control the clamping force of a terminal effector of a teleoperated slave surgical instrument attached to the screening handle master finger 170. [0068] For the position shown in figure 2A, with the thumb 292A and the index finger 292B separated by the maximum distance allowed by the master finger tracking handle 270, the minimum clamping force. Conversely, in the position shown in figure 2D, where the thumb 292A and index finger 292 are as close as allowed by connector 210, that is, separated by the minimum distance allowed by the master finger tracking handle 270, the clamping force is maximum . Figures 2B and 2C represent positions that are mapped to intermediate clamping forces.

[0069] As explained more fully below, the locations (positions and orientations) of the thumb 292A and indicator 292B in figures 2A to 2D are mapped into a handle closing parameter, for example, a normalized handle closing value that is used to control the handle of the teleoperated slave surgical instrument coupled to the master finger tracking handle 270. Specifically, the locations read from the thumb 292A and the indicator 292B are mapped in the handle closure parameter by the hand tracking controller 130.

[0070] Thus, a location of a part of the hand of the surgeon 181 is tracked. Based on the location tracked, a system control parameter of the minimally invasive surgical system 100, that is, a handle closure parameter, is generated by the controller

Petition 870190091580, of 9/13/2019, p. 22/92

19/81 hand tracking 130, and provided to system controller 140. System controller 140 uses the handle closure parameter in generating a system command that is sent to the teleoperated slave surgical instrument. The system command instructs the teleoperated surgical instrument to configure a terminal effector to have a handle closure corresponding to the handle closure parameter. Therefore, the minimally invasive surgical system 100 uses the handle closure parameter to control the operation of the teleoperated slave surgical instrument of the minimally invasive surgical system 100.

[0071] Likewise, the locations (position and orientation) of the thumb 292A and indicator 292B in figures 2A to 2D are mapped into a position of the control point and an orientation of the control point by the hand tracking controller 130. The position of the control point and the orientation of the control point are mapped in an eye coordinate system to the surgeon 181 and then supplied to the system controller 140 by means of a command signal. The position of the control point and the orientation of the control point in the eye coordinate system are used by system controller 140 for teleoperating the slave surgical instrument attached to the master finger tracking handle 170.

[0072] Once again, a location of part of the surgeon's hand 181 is tracked. Based on the location tracked, another system control parameter of the minimally invasive surgical system 100, that is, the position and orientation of the control, is generated by the hand tracking controller 130. The hand tracking controller 130 transmits a command signal with the position and orientation of the control point to system controller 140. System controller 140 uses the position and orientation control point when generating a system command that is sent to the

Petition 870190091580, of 9/13/2019, p. 23/92

20/81 teleoperated slave surgical instrument. The system command instructs the teleoperated surgical instrument to position the teleoperated surgical instrument based on the position and orientation of the control point. Therefore, the minimally invasive surgical system 100 uses the position and orientation of the control point to control the operation of the teleoperated slave surgical instrument of the minimally invasive surgical system 100.

[0073] In addition to determining the closure of the handle based on the positions of the sensors 211, 212, another relative movement between the indicator 292B and the thumb 292A is used to control the yaw movement and the rolling movement of the slave surgical instrument. When the finger 292B and thumb 292A are rubbed crosswise, as when rotating an axis, which is represented by the arrows in (figure 2E) around an imaginary axis 293, it produces the bearing of the tip of the slave surgical instrument, when whereas, when the index and thumb slide up and down along the length of each other, which is represented by the arrows in (figure 2F) along an axis in the pointed direction represented by the arrow 295, yaw movement along the X axis of the slave surgical instrument. This is achieved by mapping the vector between the positions of the tip of the index and the tip of the thumb to define the X axis of the control point orientation. The position of the control point remains relatively stationary, as the finger and thumb are sliding symmetrically along the 295 axis. Although the movements of the index and thumb are not completely symmetrical movements, the position still remains sufficiently stationary for the user can easily correct any disturbance that may occur.

[0074] Once again, locations of part of the hand of the surgeon 181 are tracked. Based on the tracked locations,

Petition 870190091580, of 9/13/2019, p. 24/92

21/81 another system control parameter, that is, the relative movement between two fingers of the 291R surgeon's hand, is generated by the hand tracking controller 130.

[0075] The hand tracking controller 130 converts the relative motion into an orientation for the teleoperated slave surgical instrument attached to the master finger tracking handle 170. The hand tracking controller 130 sends a command signal with the orientation to the system controller 140. Although this orientation is an absolute orientation mapping, system controller 140, in one respect, uses this ratchet entry during teleoperation in the same way as an orientation input from any other cardan-supported master tool handle passive. An example of a ratchet is described in US patent application No. 12 / 495,213 assigned in common (deposited on June 30, 2009, which presents a Ratchet For Master Alignment of a Teleoperated Surgical Instrument), which is incorporated herein by reference in its wholeness.

[0076] System controller 140 uses guidance in generating a system command that is sent to the teleoperated slave surgical instrument. The system controller instructs the teleoperated surgical instrument to rotate the teleoperated surgical instrument based on the orientation. Therefore, the minimally invasive surgical system 100 uses the movement between the two fingers to control the operation of the teleoperated slave surgical instrument of the minimally invasive surgical system 100.

[0077] When the movement is a first movement, for example, cross-rubbing the finger 292B and thumb 292A, as when rotating an axis, the orientation is a bearing, and the system command results in a bearing of the tip of the handle of the slave surgical instrument along its pointed direction. When the

Petition 870190091580, of 9/13/2019, p. 25/92

22/81 movement is a second movement different from the first movement, for example, sliding the index and thumb up and down along the length of each other (figure 2F), the orientation is a turn, and the system command results in a yaw motion of the slave surgical instrument handle.

[0078] In yet another aspect, when the surgeon switches the mode of operation of the system to a gesture recognition mode, both hands are tracked, and control points and orientations for both hands are generated based on positions and orientations hand-mounted sensors in one aspect. For example, as shown in figure 2G, the tips of the thumb and forefinger of each hand are touched together to form a circular shape. The read position of each hand is mapped by the hand tracking controller 130 into a pair of checkpoint positions. The control point pair is sent with a camera control system event to system controller 140.

[0079] Thus, in this respect, a location of a part of each hand of the surgeon 181 is tracked. Another system control parameter of the minimally invasive surgical system 100, that is, the pair of control point positions, based on at the tracked location it is generated by the hand tracking controller 130. The hand tracking controller 130 sends the pair of checkpoint positions with a camera control system event to the system controller 140.

[0080] In response to the camera control system event, system controller 140 generates a camera control system command based on the pair of control point positions. The camera control system command is sent to a teleoperated endoscopic camera manipulator in the 100 minimally invasive surgical system. Therefore, the minimally surgical system

Petition 870190091580, of 9/13/2019, p. 26/92

Invasive 23/81 100 uses the pair of control point positions to control the operation of the teleoperated endoscopic camera manipulator of the 100 minimally invasive surgical system.

System Control Through Hand Gesture Poses and Hand Gesture Trajectories [0081] In this aspect, after being placed in a gesture detection operation mode, the hand tracking controller 130 detects a hand gesture pose, or a hand gesture pose and a hand gesture trajectory. Controller 130 maps hand gesture poses to certain system mode control commands, and similarly, controller 130 maps hand gesture trajectories to other system mode control commands. It should be noted that the mapping of poses and trajectories is independent and, therefore, this is different from, for example, manual sign language tracking. The ability to generate system commands and control the system 100 using hand gesture poses and hand gesture trajectories, instead of manipulating switches, numerous pedals and others, as in known minimally invasive surgical systems, provides greater ease of use of system 100 for the surgeon.

[0082] When the surgeon is standing, the use of hand gesture poses and hand gesture trajectories to control the system 100 makes it unnecessary for the surgeon to take the patient's eyes and / or the viewing screen to look for a pedal or switch when the surgeon wants to change the system mode. Finally, the elimination of the various switches and pedals reduces the space required by the minimally invasive teleoperated surgical system.

[0083] The particular set of hand gesture poses and hand gesture trajectories used to control the minimally invasive surgical system 100 is not critical, as long as each hand gesture pose and

Petition 870190091580, of 9/13/2019, p. 27/92

24/81 each hand gesture trajectory is unambiguous. Specifically, a hand gesture pose should not be interpreted by the hand tracking controller 130 as one or more other hand gesture poses in the set of poses, and a hand gesture path should not be interpreted as more than one. hand gesture trajectory in the set of trajectories. Thus, the hand gesture poses and the hand gesture trajectories discussed below are illustrative only and are not intended to be limiting.

[0084] Figures 3A to 3D are examples of hand gesture poses

300A to 300D, respectively. Figures 4A to 4C are examples of hand gesture trajectories. It should be noted, for example, that the configuration in figure 2A looks similar to that in figure 3A, but the mode of operation of the minimally invasive surgical system 100 is different when using the two configurations.

[0085] In figure 2A, the teleoperated minimally invasive slave surgical instrument is coupled to the master finger tracking handle 170 and the system 100 is in tracking mode, so that the movement of the teleoperated minimally invasive slave surgical instrument follows the tracked movement of the surgeon's hand. In figures 3A to 3D and 4A to 4C, the surgeon puts the system 100 in gesture recognition mode and then makes one of the hand gesture poses or hand gesture trajectories illustrated. Hand gesture poses and hand gesture trajectories are used to control system modes and are not used in accompanying operation mode. For example, system modes controlled with hand gesture poses are to enable, disable and switch between visual displays, engage visual display and draw / erase remote medicine.

[0086] In the 300A hand gesture pose (figure 3A), the thumb 292A and the index finger 292 are separated beyond a master clutch threshold, for example, the separation between the two fingers of the 291R hand is

Petition 870190091580, of 9/13/2019, p. 28/92

25/81 greater than 115 mm. The hand gesture pose 300B (figure 3B) with the index finger 292B extended and the thumb 292A bent is used to signal the hand tracking controller 130 that the surgeon is plotting a hand gesture path (see figures 4A and 4B ). The 300C hand gesture pose (figure 3C) with the 292A thumb up and the folded 292B indicator is used to turn on a user interface and switch between modes on the user interface. The 300D hand gesture pose (3D figure) with the 292A thumb down and the bent 292B indicator is used to turn off the user interface. Other hand gesture poses could include an A-OK hand gesture pose, an L-shaped hand gesture pose, and others.

[0087] The hand tracking controller 130, in one aspect, uses a multidimensional feature vector to recognize and identify a hand gesture pose. Initially, a plurality of hand gesture poses is specified. Next, a set of characteristics is specified that includes a plurality of characteristics. The feature set is designed to uniquely identify the hand gesture pose in the plurality of poses.

[0088] A hand gesture pose recognition process is trained using a training database. The training database includes a plurality of cases from each of the hand gesture poses. The plurality of cases includes feature vectors for the poses made by countless different people. A set of characteristics is generated for each of the cases in the training database. These feature sets are used for training a multidimensional Bayesian classifier, as explained more fully below.

[0089] When the surgeon 180 wants to enter the operation mode by hand gesture, the surgeon activates a switch, for example, presses a pedal, and then makes a hand gesture pose with hair

Petition 870190091580, of 9/13/2019, p. 29/92

26/81 minus a hand. It should be noted that, although this example requires a single pedal, it allows the elimination of the other footrest pedals of the known minimally invasive surgical system and therefore still has the advantages described above. The hand tracking unit 186 sends signals representing the positions and orientations read from the thumb and forefinger of the surgeon's hand or hands or to the hand tracking controller 130.

[0090] Using the tracking data for the surgeon's fingers, the hand tracking controller 130 generates a set of observed features. The hand tracking controller 130 then uses the trained multidimensional Bayesian classifier and a Mahalanobis distance to determine the possibility, that is, the probability, that the set of observed characteristics is a set of characteristics of a hand gesture pose. in the plurality of poses. This is done for each of the hand gesture poses in the plurality of poses.

[0091] The hand gesture pose in the plurality of poses that is selected by the hand tracking controller 130 as the observed hand gesture pose is the one with the shortest Mahalanobis distance, if the Mahalanobis distance is less than the Mahalanobis distance maximum in the training database for this hand gesture pose. The selected hand gesture pose is mapped to a system event. Hand tracking controller 130 injects the system event into system controller 140.

[0092] System controller 140 processes the system event and issues a system command. For example, if the hand gesture pose 300C (figure 3C) is detected, system controller 140 sends a system command to monitor controller 150 to connect the user interface. In response, monitor controller 150 runs at least a portion of user interface module 155 on

Petition 870190091580, of 9/13/2019, p. 30/92

27/81 processor 151 to generate a user interface on the monitor of the surgeon console 185.

[0093] Thus, in this respect, the minimally invasive surgical system 100 tracks a location of part of a human hand. Based on the tracked location, a system control parameter is generated, for example, a hand gesture pose is selected. The hand gesture pose is used to control the user interface of the minimally invasive surgical system 100, for example, to display the user interface on the monitor of the surgeon console 185.

[0094] The control of the user interface is only illustrative and is not intended to be limiting. A hand gesture can be used to perform any of the mode changes in a known minimally invasive surgical system, for example, master clutch, camera control, camera focus, changing the manipulator arm and others.

[0095] If the hand gesture pose recognition process determines that the observed hand gesture pose is the hand gesture pose for a hand gesture path, a system event is not injected by the hand tracking controller hand 130 based on pose recognition. Instead, a hand gesture trajectory recognition process begins.

[0096] In this example, the 300B hand gesture pose (figure 3B) is the pose used to generate a hand gesture path. Figures 4A and 4B are two-dimensional examples of hand gesture trajectories 400A and 400B that are made using the hand gesture pose 300B. Figure 4C shows other two-dimensional examples of hand gesture trajectories that can be used.

[0097] In one aspect, the hand gesture trajectory recognition process uses a Hidden Markov Model Λ. To generate the

Petition 870190091580, of 9/13/2019, p. 31/92

28/81 probability distributions for the Markov Hidden Model Λ, a training database is required. Before obtaining the training database, a set of hand gesture trajectories is specified. In one aspect, the sixteen hand gesture trajectories in figure 4C are selected.

[0098] In one aspect, numerous test subjects are selected to perform each of the hand gesture trajectories. In one example, each test subject performed each trajectory a predetermined number of times. The position and orientation data of each subject for each trajectory performed were saved in the training database. In one respect, as explained more fully below, the training database was used to train a discrete left-right Markov Hidden Model using an iterative Baum-Welch method.

[0099] When a surgeon makes a trajectory, the data is converted into an observation sequence O by the hand tracking controller 130. With observation sequence O and the Markov Hidden Model Λ, the hand tracking controller 130 determines which hand gesture trajectory corresponds to the observed sequence of symbols. In one aspect, the hand tracking controller 130 uses the direct recursion algorithm with the Markov Hidden Model Λ to generate the total probability of the observed symbol sequence. The trajectory of the hand gesture with the highest probability is selected if that probability is greater than a threshold probability. If the highest probability is less than the threshold probability, no trajectory of the hand gesture is selected, and processing ends.

[00100] The selected hand gesture path is mapped to a system event. Hand tracking controller 130 injects the system event into system controller 140.

[00101] System controller 140 processes the system event

Petition 870190091580, of 9/13/2019, p. 32/92

29/81 and issues a system command. For example, if the selected hand gesture path is mapped to an event to change the lighting level in the surgical field, system controller 140 sends a system event to a controller in the illuminator to change the lighting level.

Presence Detection Through Hand Tracking [00102] In one aspect, as indicated above, the hand positions of the surgeon 291R, 291L (figure 6A) are tracked to determine whether teleoperation of a minimally invasive surgical system 100 is allowed and, in some aspects, whether it is to display a user interface to the surgeon. Again, the hand tracking controller 130 tracks at least part of a hand of the surgeon 180B (figure 6A). The hand tracking controller 130 generates a location for a master tool handle, for example, the master tool handle 621 (figure 6B), which represents the master tool handles 621L, 621R (figure 6A), and a location for the part of the hand. The hand tracking controller 130 maps the two locations in a common coordinate frame and then determines the distance between the two locations in the common coordinate frame. Distance is a system control parameter for a minimally invasive surgical system that is based on the tracked location of the surgeon's hand.

[00103] If the distance is less than a safe threshold, that is, less than the maximum allowed separation between the hand part and the master tool handle, teleoperation of a minimally invasive surgical system 100 is allowed and, otherwise, the teleoperation is inhibited. Likewise, in the aspect that uses presence detection to control the display of a user interface, if the distance is less than a safe boundary, that is, less than the maximum allowed separation between the hand part and the handle. master tool, the

Petition 870190091580, of 9/13/2019, p. 33/92

30/81 display of a user interface on a monitor of the minimally invasive surgical system 100 is inhibited and, otherwise, display of the user interface is allowed.

[00104] Thus, the distance is used to control the teleoperation of the minimally invasive surgical system 100. Specifically, the hand tracking controller 130 sends a system event to the system controller 140 indicating whether teleoperation is allowed. In response to the system event, system controller 140 configures system 100 to allow or inhibit teleoperation.

[00105] Hand Location Tracking Technologies [00106] Before considering the various aspects of hand tracking described in more detail, an example of a tracking technology is described. This example is illustrative only, and in view of the description below, any tracking technology that provides the required hand or finger location information can be used.

[00107] In one aspect, pulsed DC electromagnetic tracking is used with sensors mounted on two fingers of a hand, for example, the thumb and index finger, as shown in figures 2A to 2D and figure 7. Each sensor measures six degrees of freedom and, in one aspect, it has a size of eight millimeters by two millimeters by one and a half millimeters (8 mm x 2 mm x 1.5 mm). The tracking system has a hemisphere of 0.8 m right-handed workspace and a position reading resolution of 0.5 mm and 0.1 degree. The refresh rate is 160 Hertz and has a read latency of four milliseconds. When integrated into a system, additional latency may be incurred due to additional communication and filtering. An effective command latency of up to 30 milliseconds proved to be acceptable.

[00108] In this respect, the tracking system includes an electromagnetic hand tracking controller, sensors for use in

Petition 870190091580, of 9/13/2019, p. 34/92

31/81 master finger tracking and a hand tracking transmitter. A tracking system suitable for use in one embodiment of this invention is available from Ascension Technology Corporation of Burlington, Vermont, USA, as a 3D guidance trakSTAR® system with a medium range transmitter (trakSTAR® is a registered trademark of Ascension Technology Corporation .) The transmitter generates pulsed DC magnetic fields for high precision tracking in medium ranges, which are specified as 78 centimeters (31 inches). This system provides dynamic tracking with 240 to 420 updates / second for each sensor. The outputs of the miniaturized passive sensors are not affected by noise sources in the power line. A clear line of sight between the transmitter and the sensors is not required. There is total attitude tracking and no inertial drift or optical interference. There is high immunity to metals and no distortion by non-magnetic metals.

[00109] Although an electromagnetic tracking system with finger coverage is used here, this is only illustrative and is not intended to be limiting. For example, a pen-shaped device could be held by the surgeon. The pen-shaped device is a finger piece with three or more non-collinear fiducial markers on the outer surface of the device. Typically, to make at least three fiducial markers visible from any point of view, more fiducial markers are used due to self-occlusion. The fiducial markers are sufficient to determine six degrees of freedom (three of translation and three of rotation) of the movement of the finger piece and, therefore, of the hand that holds the pen-shaped device. The pen-shaped device also reads the grip in one aspect.

[00110] The pen-shaped device is viewed by two or more cameras of known parameters to locate the markers

Petition 870190091580, of 9/13/2019, p. 35/92

32/81 fiducial in three dimensions and infer the three-dimensional pose of the finger piece. Fiducial markers can be implemented, for example, as 1) retroreflective spheres with lighting close to the chamber; 2) concave or convex semi-spheres with lighting close to the camera; or 3) active markers, such as an LED (flashing). In one aspect, almost infrared illumination of the finger piece is used, and filters are used to block the visible spectrum in the chamber, to minimize distraction by the background cluster.

[00111] In another aspect, a data glove 501 (figure 5) or bare hand 502 is used, and fiducial markers 511 are attached to the thumb and index finger of the glove 501 (and / or other fingers of the glove) surgeon will use and / or directly to the skin of the hand 502. Again, redundant markers can be used to accommodate self-occlusion. Fiducial markers can also be placed on other fingers, to allow for more user interface features through specifically defined hand gestures.

[00112] The three-dimensional locations of the fiducial markers are computed by triangulation of multiple cameras with a common field of view. The three-dimensional locations of the fiducial markers are used to infer the three-dimensional pose (translation and orientation) of the hand and also the size of the handle.

[00113] The marker locations need to be calibrated before use. For example, the surgeon can show the hand with markers in different poses for the camera. The different poses are then used for calibration.

[00114] In yet another aspect, hand tracking without markers is used. An articulated hand movement can be tracked using images seen by one or more cameras and processing these images using computer software. Running computer software does not need to track all

Petition 870190091580, of 9/13/2019, p. 36/92

33/81 degrees of freedom of the hand to be useful. The running software only needs to track the part related to the two fingers of a hand to be useful in controlling a surgical tool, as shown here.

[00115] In camera-based tracking, the accuracy of measurements depends on the accuracy of location of the markers in the image; the precision of three-dimensional reconstruction due to the geometry of the camera; and redundant data, such as more than a minimum number, for example, three, fiducial markers, more than a minimum number (one or two) of cameras, and the mean time and filtering.

[00116] The accuracy of three-dimensional reconstruction is largely based on the accuracy of the camera calibration. Some fiducial markers attached to known locations on the surgeon console can be used to determine the extrinsic parameters (rotation and translation) of multiple cameras with respect to the surgeon console. This process can be done automatically. Active fiducial markers can be used as fiducial calibration markers, as these markers are only switched on during a calibration process and before the procedure. During the procedure, the calibration fiducial markers are turned off to avoid confusion with the fiducial markers used to locate the surgeon's hands. The relative extrinsic parameters can also be computed by observing a moving marker in the common field of view of the cameras.

[00117] Other screening technologies that are suitable for use include, but are not limited to, inertial screening, depth chamber scanning and fiber fold reading.

[00118] As used here, a sensor element, sometimes called a tracking sensor, can be a sensor for any of the hand tracking technologies described above, for example, a

Petition 870190091580, of 9/13/2019, p. 37/92

34/81 passive electromagnetic sensor, a fiducial marker or a sensor from any of the other technologies.

Coordinate Systems [00119] Before considering the various processes described in more detail, an example of a 185B surgeon console (figure 6A) is considered, and several coordinate systems are defined for use in the following examples. The 185B surgeon console is an example of the 185B surgeon console. The 185B surgeon console includes a 610 three-dimensional viewer, sometimes called a 610 viewer, 620L, 620R master tool manipulators with 621L, 621R master tool handles, and a base 630. Master tool handle 621 (figure 6B) is a more detailed diagram of master tool handles 621L, 621R.

[00120] The master tool handles 621L, 621R of the master tool manipulators 620L, 620R are held by the surgeon 180B using the index and thumb, so that orientation and gripping involve intuitive pointing and tightening movements. Master tool manipulators 620L, 620R in combination with master tool handles 621L, 621R are used to control teleoperated slave surgical instruments, teleoperated endoscopes and others, in the same way as known master tool manipulators in a known minimally invasive teleoperated surgical system. Likewise, the position coordinates of master tool manipulators 620L, 620R and master tool handles 621L, 621R are known for the kinematics used to control slave surgical instruments.

[00121] In normal view operating mode, viewer 610 displays three-dimensional images of surgical field 103 of stereoscopic endoscope 112. Viewer 610 is positioned on

Petition 870190091580, of 9/13/2019, p. 38/92

35/81 console 185B (figure 6B) close to the surgeon's hands, so that the image of the surgical field seen in viewer 610 is oriented so that the surgeon 180B feels that he is really looking directly down into the surgical field 103. The surgical instruments in the image they appear to be located substantially where the surgeon's hands are located and oriented substantially as the surgeon 180B would expect based on the position of his hands. However, the 180B surgeon may not be able to see your hands or the position or orientation of the 621L, 621R master tool handles, although he does see the surgical field image displayed in viewer 610. [00122] In one aspect, the master tool manipulators 620L, 620R are moved from directly in front of the surgeon 180B and under the viewer 610 to be positioned on the base 630, and so that they are no longer positioned under the viewer 610, that is, the master tool manipulators are parked outside the hand gesture path. This creates an unobstructed volume under the viewer 610 in which the surgeon 180B can make hand gestures, either or both between hand gesture poses or hand gesture trajectories.

[00123] In the aspect of figure 6A, three coordinate systems are defined in relation to the 185B surgeon console: a 660 display coordinate system, a world coordinate system 670 and a 650 coordinate tracking system. note that equivalent coordinate systems are defined for surgeon 181 (figure 1), so that the mapping described more fully below can be done for tracking data from the master finger tracking handle 170 or the master tool handles 621L, 621R . See, for example, U.S. Patent Application No. 12 / 617,937, entitled Surgeon Patient Side Interface for a Minimally Invasive Teleoperated Surgical Instrument, filed

Petition 870190091580, of 9/13/2019, p. 39/92

36/81 on November 13, 2009, which is incorporated herein by reference in its entirety.

[00124] In the 660 display coordinate system, the surgeon 180B is looking down on the Zview Z axis. The Yview Y axis points up on the monitor. The Xview X axis points to the left on the monitor. In the world 670 coordinate system, the Zworld Z axis is a vertical axis. The Xworld X axis and the Yworld Y axis are in a plane perpendicular to the Zworld Z axis.

[00125] Figure 6B is a more detailed illustration of master tool handle 621 and master tool manipulators 620. Coordinate systems 680, 690 are discussed more fully below with respect to method 1100 of figure 11.

Surgical Instrument Control Process Using Hand Tracking [00126] Figure 7 is an illustration of sensor 212 mounted on indicator 292B with a location 713 in the tracking coordinate system 750 and a sensor 211 mounted on the thumb 292A with a location 711 on coordinate tracking system 750. Sensors 211 and 212 are part of the electromagnetic tracking system described above. The thumb 292A and index finger 292B are examples of fingers on the right hand 291R. As previously indicated, a part of a human hand includes at least one finger of the hand. As is known to those skilled in the art, the fingers, sometimes called digits or phalanges, of the hand are the thumb (first finger), index (second finger), middle finger (third finger), ring finger (fourth finger) and little finger (fifth finger).

[00127] Here, the thumb and forefinger are used as examples of two fingers on a human hand. This is only illustrative and is not intended to be limiting. For example, the thumb and middle finger could be used in place of the thumb and index finger. This description is

Petition 870190091580, of 9/13/2019, p. 40/92

37/81 directly applicable also to the use of the middle finger. Likewise, the use of the right hand is only illustrative. When similar sensors are used in the left hand, the present description is directly applicable also to the left hand.

[00128] A cable 741,742 connects sensors 211,212 from the master finger tracking handle 270 to the hand tracking controller 130. In one aspect, cable 741, 742 carries position and orientation information from sensors 211, 212 to the hand screening 130.

[00129] The use of a cable to transmit the position and orientation data read to the hand tracking controller 130 is only illustrative and is not intended to be limiting to that specific aspect. In view of this exposure, those skilled in the art can select a mechanism for transmitting position and orientation data read from the master finger tracking handle or master finger tracking handles to the hand tracking controller 130 (for example, using wireless connection).

[00130] The cable 741, 742 does not inhibit the movement of the master finger tracking handle 270. Since the master finger tracking handle 270 is mechanically not grounded, each master finger tracking handle is effectively unrestricted for position movements and orientation within the workspace achieved by the surgeon and workspace of the hand-held tracker (for example, left-right, up-down, in-out, rolling, pitching and yawing in a coordinate system Cartesian).

[00131] In one aspect, as described above, each sensor 211, 212 on the master finger tracking handle 270 reads three degrees of translation and three degrees of rotation, that is, six degrees of freedom. Thus, the data read from the two sensors represents twelve degrees of freedom. In another

Petition 870190091580, of 9/13/2019, p. 41/92

38/81 aspect, each sensor 211,212 in the master finger tracking handle 270 reads three degrees of translation and two degrees of rotation (yaw and pitch), that is, five degrees of freedom. Thus, the data read from the two sensors represents ten degrees of freedom.

[00132] Using the control point position and control point orientation based on the tracked locations to control a teleoperated slave surgical instrument requires six degrees of freedom (three of translation and three of orientation), as described more fully below . Thus, in the aspects where each sensor has five or six degrees of freedom, sensors 211, 212 provide redundant degrees of freedom. As described above and more completely below, the redundant degrees of freedom are mapped into parameters used to control aspects of the teleoperated slave surgical instrument other than position and orientation.

[00133] In yet another aspect, each sensor 211, 212 reads only three degrees of freedom of translation, which therefore represent six degrees of freedom. This is sufficient to control three degrees of translation, rolling and closing of the handle of a slave surgical instrument that does not include a handle mechanism. The following description is used to generate the location of the control point using the six degrees of freedom. The orientation of the control point is taken as the orientation of the slave surgical instrument. The handle closing parameter is determined as described below using the location of the control point and the orientation of the control point. The bearing is determined as described above using the relative movement of the thumb and forefinger.

[00134] In any aspect where the sensors read six degrees of freedom, or where the sensors read five degrees of freedom, the indicator sensor 212 generates a signal that represents a position of the

Petition 870190091580, of 9/13/2019, p. 42/92

39/81 Pindex index finger and a Rindex index finger orientation in the 750 tracking coordinate frame. The 211 thumb sensor generates a signal representing a Pthumb thumb position and an Rthumb thumb orientation in the 750 tracking coordinate frame. In one aspect, the Pindex and Pthumb positions are taken to be aligned with the center of the user's nail on the 292A thumb, respectively.

[00135] In this example, the Pindex and Pthumb positions are each represented as a three by one vector in the 750 coordinate tracking frame. The Pindex and Pthumb positions are in the tracking coordinates.

[00136] The Rindex and Rthumb orientations are each represented as a three by three matrix in the 750 tracking coordinate table, that is,

R = ^Λ index

Kindest 1

Kindex21

Kjndex31

Kjndex12

K-í _{n (} j _ex 22

Kjndex31

Kjndexl3

Kindex23

Kjndex33

K thumb

Kthumbll

K (humb21

K _t humb31

Kthumbl2

Kthumb22

Kthumb31

K (humbl3)

K (humb23

K (humb33 [00137] A P _cp control point position is centered between the 292B indicator and the 292A thumb. The P _cp control point position is in the 790 control point frame, but is specified in the tracking coordinates. The Z axis of the control point frame 790 extends through the position of the control point P _cp in the pointed direction, as described more fully below.

[00138] Likewise, as explained below, the indicator

Petition 870190091580, of 9/13/2019, p. 43/92

40/81

292B and thumb 292A are mapped on the jaws of a slave surgical instrument, but the two fingers are more right-handed than the jaws on the instrument. The Y axis of the control point frame 790 corresponds to the pin used to close the instrument's clamp. Thus, the Y axis of the control point frame 790 is perpendicular to a vector between the indicator 292B and the thumb 292A, as described below.

[00139] The position of the control point P _cp is represented as a three by one vector in the trace coordinates of the trace coordinate frame 750. The orientation of the control point R _cp is represented as a three by three matrix in the tracking coordinates, that is,

Rcpll Rcpl2 Rcpl3 Rcp = Rcp21 Rcp22 Rcp23 Rcp31 Rcp31 Rcp33

[00140] Figure 8 is a process flow chart for mapping the location of part of a hand in a handle closure parameter used to control the grip of a slave surgical instrument, for example, one of the teleoperated slave surgical instruments in the figure 1. This mapping also maps a temporal change in location in a new parameter for closing the handle and a corresponding location of a slave instrument tip and the speed of movement to that location.

[00141] Initially, with the entry in process 800, the process RECEIVE HAND LOCATION DATA 810 receives the position and orientation of the index finger (Pindex and Rmdex) and the position and orientation of the thumb (Pthumb and Rthumb), which, in this example, are stored

Petition 870190091580, of 9/13/2019, p. 44/92

41/81 as data 811. The position and orientation of the index finger (Pindex and Rindex) and the position and orientation of the thumb (Pthumb and Rthumb) are based on data from the tracking system. Process 810 transfers to the process MAPPING LOCATION DATA IN CONTROL POINT AND PICKUP PARAMETER 820.

[00142] The MAPPING LOCATION DATA IN CONTROL POINT AND HANDLE 820 PARAMETER generates a Pcp control point position, an Rcp control point orientation and a ggrip handle closure parameter using the index finger position and orientation (Pindex and Rindex) and the position and orientation of the thumb (Pthumb and Rthumb). The position of the Pcp control point, the orientation of the Rcp control point and the ggrip handle closing parameter are stored as 821 data.

[00143] In one aspect, the control point mapping performed in the 820 process is defined to emulate key properties of the placement of control points in known master tool manipulators. Thus, the response to the movement of the thumb and forefinger will be familiar and intuitive for users of the teleoperated minimally invasive surgical system known with a surgeon console similar to the 180B surgeon console (figure 6A).

[00144] Figure 9 is a more detailed process flowchart for an aspect of the MAPPING LOCATION DATA process IN CONTROL POINT AND PICKUP 820 PARAMETER. First, in process 820, the process MAPPING HAND POSITION DATA IN POINTS CONTROL 910 generates a location of the position of the Pcp control point from the position of the index finger and the position of the Pthumb thumb, that is,

Pcp = 0.5 * (Pthumb + Pindex) [00145] The position of the Pcp control point is the average of the position of the Pindex indicating and position of the Pthumb thumb. The MAP DATA process

Petition 870190091580, of 9/13/2019, p. 45/92

42/81

OF HAND POSITION AT CONTROL POINTS 910 transfers ο processing to the GENERATE CONTROL POINT 920 GUIDANCE process.

[00146] As indicated above, the Z axis of the control point orientation is aligned in the pointed direction. In this aspect of the process GENERATE CONTROL POINT 920 ORIENTATION, Rodriquez's axis / angle formula is used to define the direction vector pointed on the Z Zhait axis for the orientation of the control point as a half rotation between the direction vector pointed by the Z index indicator and the direction vector pointed by the thumb Z _thumb from the orientation of the thumb Rthumb, the direction vector pointed by the thumb Zthumb is:

Z thumb ⁼ [Κ thumb 13 ^ thumb23 ^ thumbSs] [00147] Likewise, from the orientation of the Rindex index finger, the direction vector pointed by the Z _index indicator is:

Λ

Z index ^- [- ^ indexl3 ^ index23 ^ indexSsl [00148] The vector ω is a vector perpendicular to the direction vector pointed by the Z _index indicator _{and the} direction vector pointed by the thumb Z _thumb the vector ω is defined as the cross product of the direction vector pointed by the Z _index indicator θ direction vector pointed by the thumb Z _thumbi that is, ^{ω =} Z index ^X Z thumb [00149] The angle Θ is the magnitude of the angle between the direction vector

Petition 870190091580, of 9/13/2019, p. 46/92

43/81 pointed by the Z _index indicator _{and the} direction vector pointed by the Zthumb thumb. The angle Θ is defined as:

sin ' ¹ sin

MifHd]>

otherwise [00150] With the ω axis and the θ angle, the direction vector pointed on the ^{Z Z} haiffle axis:

^ half

index [00151] Thus, process 910 generated the position of the control point Pc _P , and the initial part of process 920 generated the approximate pointed direction of the Z axis in the control point frame 790. You can proceed with the interpellation of the vectors orientation of the index and thumb to generate vector axes of Xcp and _cp control point units in a similar manner and then re-orthogonalize them to produce a control point orientation matrix.

[00152] However, greater dexterity of teleoperation can be achieved from the tracked locations of the fingers using the following mapping. This mapping uses the relative positions of the forefinger and thumb to effectively roll and yaw the control point, just like manipulating a small cardan support between the fingers. The rest of process 920 is performed as follows to generate a complete set of vector axes of orthonormal control point units X _cp , ^ cpe ^z _cp .

Petition 870190091580, of 9/13/2019, p. 47/92

44/81 £ _ P index P thumb || P index ~ P thumb ||

Υcp ~ ^ half X ^X cp

 _P = vy _cp [00153] With these vectors, the orientation of the control point R _cp is:

Rcpll R _C pl2 Rcpl3 Rcp [ ^X Cp ycp ^z cp] Rcp21 Rcp22 Rcp23 RcpJl Rcp31 Rcp33_

[00154] Now with processes 910 and 920, process 820 mapped the positions and orientations of the index and thumb (Pindex, Rindex), (Pthumb, Rthumb) in the position and orientation of the control point (P _cp , R _cp ) . Process 820 has yet to generate the ggrip handle closing parameter. Thus, the process GENERATE CONTROL POINT 920 ORIENTATION transfers the processing to the process GENERATE CLOSE PARAMETER OF HANDLE 930.

[00155] In process 930, the closure of the handle is determined by the distances from the position of the index finger Pmdex and the position of the thumb Pthumb from the axis of the central line defined by the position of the control point P _cp and direction of the axis ZZ _cp | _SS0 allows the handle closure to be invariant to sliding when the thumb and forefinger are touching.

[00156] Thus, the finger position indicator and Pindex Pthumb thumb position in the Z axis are mapped in the frame 790. The position Pindex_ roj _P is the projection of Pindex finger position indicator on the Z axis of the frame 790, and Pthumb_ _P roj position is the projection of the Pthumb position of the thumb on the Z axis of frame 790.

Petition 870190091580, of 9/13/2019, p. 48/92

45/81

Pindex_proj Pep + ( ^Z ep * (Pindex Pep)) ^Z cp

P thumb proj ⁼ Pep ⁺ ( ^Z cp * (Pthumb ^- Pep)) * ^Z cp [00157] The Pindex_proj position and the Pthumb_proj position are used to evaluate an evaluation handle closing distance d _va i, that is, going to P index Pindex_proj

P thumb P thumb_proj [00158] Here, the double parallel lines are known representatives of the Euclidean binormal distance. The closing distance of the evaluation handle d _va i is limited by a maximum distance threshold d _max and a minimum distance threshold dmin. As shown in figure 7, the foam-padded connector 210 between sensors 211, 212 restricts the fingers within a fixed separation, that is, between a maximum distance threshold d _max and a minimum distance threshold dmin. In addition, a neutral distance from that corresponding to the separation distance when the two fingers are just touching.

[00159] For a particular set of sensors and a connector, a maximum distance threshold d _max , a minimum distance threshold dmin and the neutral distance of are empirically determined. In one respect, three different combinations of sensors and a connector are presented for small, medium and large hands. Each combination has its own maximum distance threshold d _max , minimum distance threshold dmin and neutral distance of, as the length of connector 210 is different in each of the combinations.

[00160] Process 930 compares the distance d _va i with the minimum distance threshold dmin. If the comparison finds that the distance d _va i is less than a minimum distance threshold dmin, the handle closing distance d is established at the minimum distance threshold dmin. Otherwise, process 930 compares the distance d _va i with the maximum distance threshold d _max . If the comparison enters that the distance d _va i is

Petition 870190091580, of 9/13/2019, p. 49/92

46/81 greater than the maximum distance threshold dmax, the handle closing distance d is established at the maximum distance threshold dmax. Otherwise, the closing distance d is established at the distance d _va i. [00161] The test performed at distance d _va i to determine the closing distance of handle d is summarized as follows:

^ min ^ val (d min d max d will) ^ max > go otherwise [00162] Next on process 930, the parameter of

closing the handle g _gr i _P :

grip dd ₀ dmax do d> d ₀ d- <l, _ni „

U d min otherwise [00163] Thus, a closing distance of handle d between the maximum distance threshold dmax and the distance is mapped to a value between zero and one. A handle closing distance d between the minimum distance threshold dmin and the distance of is mapped to a value between minus one and zero.

[00164] A value of one for the handle closing parameter g _g rip is obtained when the indicator 292B and the thumb 292A are separated by the maximum length allowed by connector 210 (figure 2A). A value of zero for the handle closure parameter g _gr i _P is obtained when the tip of the indicator 292B and the tip of the thumb 292A are just touching (figure 2C). Values in a range between zero and one control the opening / closing of the end effector jaws of a

Petition 870190091580, of 9/13/2019, p. 50/92

47/81 slave surgical instrument. A value of minus one for the ggrip handle closing parameter is obtained when the indicator 292B and the thumb 292A are touching, and the connector 210 is completely compressed between the indicator 292B and the thumb 292A (figure 2D). Values in the range between zero and minus one control the jaw strength of the closed jaws of the end effector. The connector 210 provides a passive haptic track for closing the jaw.

[00165] This example of mapping the closing distance of handle d into a value in one of the two ranges is only illustrative and is not intended to be limiting. The example is illustrative of mapping the closing distance of handle d to a value in a first range of closing parameters of the handle ggrip to control the opening / closing of the jaws of a terminal effector of a slave surgical instrument when the closing distance of the handle d is greater than the neutral distance of. Here, opening / closing means opening and closing the jaws. The closing distance of handle d is mapped to a value in a second range of closing parameters of the ggrip handle to control the jaw strength of the closed jaws of the effector end when the closing distance of handle d is less than the neutral distance of the handle. .

[00166] Thus, process 820 mapped the position and orientation of the index finger (Pindex, Rindex) and the position and orientation of the thumb (Pthumb, Rthumb) in a position and orientation of the control point (Pcp, Rcp) and parameter for closing the ggrip handle that is stored as data 821. Process 820 transfers to the MAP MAP THE WORLD COORDINATES 830 process (figure 8).

[00167] The 830 WORLD COORDINATE MAPPING process receives data 821 and maps the data 821 to a world coordinate system (see world coordinate system 670 (figure 6A)). Specifically, the position and orientation of the

Petition 870190091580, of 9/13/2019, p. 51/92

48/81 control (P _cp , R _cp ) and the handle closing parameter g _gr i _P are mapped in position and control point orientation of the world coordinates (P _cp _wc, Rc _P _wc) using a homogeneous transform of four by four ^wc Ttc that maps the coordinates in the tracking coordinate system 750 (figure 7B) into coordinates in the world coordinate system 670, for example, wc

wc tc

WC J ^l tc [00168] where:

[00169] ^wc Rtc maps an orientation in the tc tracking coordinates into an orientation in the wc world coordinates, and [00170] ^wc t _tc translates a position in the tc tracking coordinates into a position in the wc world coordinates.

[00171] The handle closure parameter g _gr i _P is not changed by this mapping. The data in the wc world coordinates are stored as data 831. Process 830 transfers to the MAP COORDINATES OF EYES 840 process.

[00172] The 840 EYE COORDINATE MAPPING process receives the data 831 in the world coordinates wc and maps the data 831 in an eye coordinate system (see the eye coordinate system 660 (figure 6A)). Specifically, the position and orientation of the world coordinate control point (Pcp_wc, Rcp_wc) and the handle closure parameter g _gr i _P are mapped in position and orientation of the eye coordinate control point (Pcp_ec, Rcp_ec) using a homogeneous four-by-four transform ^{and ec} T _wc that maps the coordinates in the world coordinate system 670 (figure 6A) to coordinates in the eye coordinate system 660, for example,

Petition 870190091580, of 9/13/2019, p. 52/92

49/81 ec rp

You wc wc

1 [00173] where:

[00174] ^ec R _wc maps an orientation in the world coordinates wc in an orientation in the coordinates of the eyes c, and [00175] ^ec t _wc is a translation of a position in the coordinates of the world wc in a position in the coordinates of the eyes ec.

[00176] Once again, the handle closing parameter g _gr i _P is not changed by the mapping. The data in the eye coordinates are stored as 841 data. The 840 process transfers to the GENERATE SPEED 850 process.

[00177] In process 800, mapping processes 830 and 840 are described as two different processes just for ease of illustration. In one aspect, the mapping processes 830 and 840 are combined, so that the control point data in the tc tracking coordinates are mapped directly into data in the c coordinates of the eyes using a homogeneous four-by-four transform ^{and ec} Ttc that maps the coordinates in the tracking coordinate system 650 (figure 6A) in coordinates in the eye coordinate system 660, for example, eci> wc / ec >> ^wc t _i_ ^ec f 1 ^ tc v Mc ^- · ec rp _ i _tc -

1 [00178] In this aspect, the position of the control point in the coordinates of the eyes Pcp_ec is;

Pcp_ ec TtcPcp_tc ’[00179] and the orientation of the control point in the coordinates of

Petition 870190091580, of 9/13/2019, p. 53/92

50/81 eyes Rcp. ec is:

R = ^CC R ^wc RR ⁿ cpec ^lx \ vc ^ tc ^ cptc [00180] In some ways, the mapping of world coordinates can be eliminated. In this case, the control point data is mapped directly from the tracking coordinate system to the eye coordinate system without using a world coordinate system.

[00181] For teleoperation, position, orientation and speed are necessary. Thus, the GENERATE SPEED 850 process generates the necessary speeds. Speeds can be generated in a number of ways. Some implementations, such as inertial and gyroscopic sensors, can measure differential signals differently to produce a linear speed and an angular speed of the control point. If speeds cannot be measured directly, process 850 estimates speeds from location measurements in the eye coordinate system in one aspect.

[00182] Speeds can be estimated using finite differences in the eye coordinate system in the sampling interval. For example, the linear velocity V _cp _ec is estimated as:

₌ P _C p_ _ec ( ^tl ) -P _C pe _C ( ^t °) ^V cp_ec - _A.

[00183] and the angular velocity u _cp _ec is estimated as:

Ra = Rcp ec (tO) ’* Rcp ec (tl)

Ra32 ^_ Ra23 ω _Δ -0.5 * R _A i3 R _à 3i

Ra21 Ra12 ^ * cp_ec

Rcp ec (t0) * io _A

At

Petition 870190091580, of 9/13/2019, p. 54/92

51/81 [00184] In another aspect of the GENERATE SPEED 850 process, the linear speed of the control point V _cp _tc and the angular speed of the control point u _cp _tc are read in the trace coordinates of the trace coordinate system 750 ( figure 7). In this respect, the linear velocity of the control point V _cp _tc directly read and the angular velocity of the control point u _cp _tc directly read are rotated from the tracking coordinate system 750 to the eye coordinate system 660 using an ^ec Rt rotation _c . Specifically, using the rotation mappings defined above:

^ec T? - ecp wcp ⁿ wc ^ tc V = ^T cp_ec = ^ec RV tc cp_tc ^ cpec ' ^{= eCR} tC® _c p_ _tc

[00185] The GENERATE SPEED 850 process transfers to the SEND CONTROL COMMAND 860 process. The 860 process sends an appropriate system control command to the slave surgical instrument based on the position, orientation, speeds and handle closing parameter stored as data 851.

[00186] In one aspect, processes 810 to 850 are performed by hand tracking controller 130 (figure 1). Controller 130 runs finger tracking module 135 on processor 131 to perform processes 810 to 850. In that respect, finger tracking module 135 is stored in memory 132. Process 850 sends a system event to the controller system 140 which, in turn, performs process 860.

[00187] It should be noted that the handheld tracking controller 130 and the system controller 140 can be implemented in practice by any combination of hardware, software that runs on a processor and firmware. Likewise, the functions of these

Petition 870190091580, of 9/13/2019, p. 55/92

52/81 controllers, as described here, can be performed by a unit or be divided between different components, each one being able to be implemented, in turn, by any combination of hardware, software that runs on a processor and firmware. When divided between different components, the components can be centralized in one location or distributed by the system 100 for purposes of distributed processing.

Hand Gesture Pose and Gesture Trajectory Control Process [00188] Figure 10 is a process flow chart of an aspect of a 1000 hand gesture pose and hand gesture trajectory control system 100. In one aspect as described above, a 1050 hand gesture pose recognition process uses a multidimensional Bayesian classifier, and a 1060 hand gesture path recognition process uses a discrete Markov Hidden Model Λ.

[00189] As described above, figures 3A to 3D are examples of hand gesture poses. To train the 1050 hand gesture pose recognition process, numerous hand gesture poses are specified. The number of hand gesture poses used is limited by the ability to define unique poses that can be unambiguously identified by the 1050 recognition process, and by the surgeon's ability to reliably remember and reproduce each of the different gesture poses hand.

[00190] In addition to defining the hand gesture poses, a set of characteristics is defined including a plurality of characteristics fi, where i varies from 1 to n. The number n is the number of characteristics used. The number and type of characteristics are selected so that each of the hand gesture poses in the set of allowable poses can be accurately identified. In one respect, the number n is six.

Petition 870190091580, of 9/13/2019, p. 56/92

53/81 [00191] The following is an example of a characteristic with n characteristics:

^- Z index * thumb ^- || p thumb P index || ^ 3 ~ (P thumb ~ P index) * index ~ || P thumb P index ^{+ f} 3 index I fs ⁼ Z thumb · [θ θ 1] f - γ · 7 ^Γ n thumb index [00192] The fi characteristic is the product of points of the pointed direction of the Z _index of the indicator 292B and the pointed direction of the thumb _thumb of the 292A. The f ₂ feature is the distance between the 292B indicator and the 292A thumb. The Í3 characteristic is the distance of the thumb 292A projected in the pointed direction Z _index of the indicator 292B. The Í4 characteristic is the distance of the thumb 292A from the axis along the pointed direction Z _index of the indicator 292B. The fs feature is the Z component of the _thumb pointing Z _thumb of the 292A thumb. The feature f _n is the product of the normal vector points the thumb of thumb ^x 292A thumb and pointed toward the Z _index indicator 292B.

[00193] Before using the 1000 method, it is necessary to develop a training database of hand gesture poses. Countless different users produce each hand gesture pose at least once, and the position and orientation data for each hand gesture pose for each user is measured using the tracking system. For example, each person in a group of people does each of the allowable hand gesture poses. The positions and orientations of the index and thumb (Pindex, Rindex), (Pthumb, Rthumb) are saved for

Petition 870190091580, of 9/13/2019, p. 57/92

54/81 each of the hand gesture poses for each person in the group in the training database.

[00194] Using the training database, a set of characteristics {fi} is generated for each hand gesture pose for each user. The training feature vector set for each hand gesture pose is then used to compute an average f.

¹ and a £ fi covariance.

[00195] Thus, the training database is used to obtain an average and a covariance of the characteristic vectors for each trained gesture. In addition, for each hand gesture pose, a Mahalanobis distance d (fi) (see description below) is generated for each trainer, and the maximum Mahalanobis distance d (fi) for each hand gesture pose is saved as a threshold for that hand gesture pose.

[00196] It is also possible to use the Mahalanobis distance measure to verify that all the trained gestures are sufficiently different and unambiguous for the given set of characteristics used. This can be accomplished by testing the Mahalanobis distance from f.

characteristic vector mean of a given gesture ¹ and the characteristic vector means of all other permissible gesture poses. This test distance should be much greater than the maximum training distance threshold used for this gesture.

[00197] As is known to those skilled in the art, a specification of a Hidden Markov Model requires the specification of two parameters of the model N, M and three measures of probability A, B, π. The Hidden Model of Markov A is represented as:

A = (A, B, π) [00198] The model parameter N is the number of states in the model, and the model parameter M is the number of observation symbols per state. The three probability measures are the distribution of

Petition 870190091580, of 9/13/2019, p. 58/92

55/81 state transition probability A, observation gesture probability distribution B and initial state distribution π.

[00199] In one aspect, for a discrete hidden Markov Model, the transition probability distribution A is an N x N matrix. The observation gesture probability distribution B is an N x M matrix, and the state distribution initial π is an N x 1 matrix.

[00200] Given an observation sequence O and the Hidden Markov Model Λ, the probability of observation sequence O, given the Hidden Markov Model Λ, that is, P (O | Λ), is evaluated in process 1000, as more fully described below.

[00201] To generate the probability distributions for the Hidden Markov Model Λ, a training database is required. Before obtaining the training database, a set of hand gesture trajectories is specified.

[00202] Numerous test subjects j are selected to make each of the hand gesture trajectories. While in figure 4C the sixteen hand gesture trajectories are presented in a projected two-dimensional shape, the test subjects are not restricted when they perform the various hand gesture trajectories, which allows for some three-dimensional variations. In one aspect, each subject performed each hand gesture trajectory k times, this produces j * k training sequences per hand gesture trajectory.

[00203] In one aspect, a discrete left-right Markov Hidden Model was used. The Markov Hidden Model Λ was chosen so that the probability P (O | Λ) was locally maximized using an iterative Baum-Welch method. See, for example, Lawrence R. Rabiner, A Tutorial on Hidden Markov Models and Selected Applications in Speech Recognition, Proceedings of the IEEE, Vol. 77, No. 2, pp. 257-286 (Feb. 1989), which is hereby incorporated by reference as a demonstration of knowledge of Hidden Models of

Petition 870190091580, of 9/13/2019, p. 59/92

56/81

Markov for those versed in models. In one respect, the iterative method was interrupted when the model converged within 0.1 percent for three successive iterations.

[00204] The probability of initial state π was established so that the model always started with state one. The transition probability matrix A was initialized with random entries, which were stored in descending order based on row by row. To comply with the left-to-right structure, all entries on the lower diagonal of the transition probability matrix A were set to zero. In addition, transitions greater than two states were not allowed by setting the entries to zero when (i - j)> 2 for all rows i and columns j. The transition probability matrix A was normalized at the end based on row by row.

[00205] Initialization for observation probability matrix B divided the observation sequence evenly based on the desired number of states. Consequently, each state may initially observe one or more symbols with a probability based on a local frequency count. This matrix was also normalized based on row by row. See, for example, N. Liu, R.I.A. Davis, BC Lovell, PJ Kootsookos, Effect of Initial HMM Choices in Multiple Sequence Training for Gesture Recognition, International Conference on Information Technology, 5-7, Las Vegas, pages 608-613 (April 2004), which is incorporated herein by reference as a demonstration of initialization procedures for Hidden Markov Models known to those skilled in the art. A Hidden Markov Model was developed for each of the hand gesture trajectories.

[00206] Back to method 1000, the verification process of ENABLED GESTURE MODE 1001 determines whether the surgeon

Petition 870190091580, of 9/13/2019, p. 60/92

57/81 enabled the system 100 gesture recognition operation mode. In one aspect, to enable the gesture recognition mode, the surgeon presses a pedal on the surgeon console 185 (figure 1A). If the gesture recognition mode is enabled, the verification process 1001 transfers to the process RECEIVE HAND LOCATION DATA 1010 and, otherwise, returns by RETURN 1002.

[00207] The process RECEIVE HAND LOCATION DATA 1010 receives the position and orientation of the index finger (Pindex, Rindex) and the position and orientation of the thumb (Pthumb, Rthumb) for the gesture being made by the surgeon. As indicated above, the position and orientation of the index finger (Pindex, Rindex) and the position and orientation of the thumb (Pthumb, Rthumb) are based on data from the tracking system. Process 1010 transfers to the process GENERATE

CHARACTERISTICS 1011.

[00208] In the process GENERATE CHARACTERISTICS 1011, the position and orientation of the index finger (Pindex, Rindex) and the position and orientation of the thumb (Pthumb, Rthumb) are used to generate each of the characteristics fi_o a fn_o in an observed characteristic vector thread. The process GENERATE CHARACTERISTICS 1011 transfers to the process COMPARE CHARACTERISTICS WITH KNOWN POSES 1012.

[00209] The process COMPARE CHARACTERISTICS WITH KNOWN POSES 1012 compares the vector of observed wire characteristic with the set of trained characteristics {fi} for each pose. This process determines the probability that the observed feature vector is included in a set of characteristics of the training data set {fi} for a particular hand gesture pose, that is, matches the training data set. This can be expressed as:

Petition 870190091580, of 9/13/2019, p. 61/92

58/81

P (fi_o | O) [00210] where the set of characteristics of the training data set {fj} is of the object class Ω.

[00211] In this example, the probability P (fi_ _o | O) is:

-fi) ' ^Σ π ¹ - ^f i) (2 _π Γ | Σ _η | “[00212] where N is the dimensionality of the characteristic vector, for example, n in the example above.

[00213] A statistic used to characterize this probability is the Mahalanobis distance d (fj_ _o ), which is defined as:

d (f _i _ _o ) = íj; E _fi ⁱ f _i _ _o

[00214] where '-O ^- Lo í. the Mahalanobis distance is known to those skilled in the art. See, for example, Moghadam, Baback and Pentland, Alex, Probabilistic Visual Learningor Object Representation, IEEE Transactions On Pattern Analysis and Machine Intelligence, Vol. 19, No. 7, pp. 696 to 710 (July 1997), which is incorporated by reference.

[00215] Using the eigenvectors Φ and the eigenvalues A of covariance Σα ' ¹ θ used in a diagonalized form, so that the Mahalanobis distance d (fi_ _o ) is:

Λ (Γ | _ _ο ) = Γ _ί ^Γ ο [φΛ- ¹ Φ ^τ ] fLo [00216] where

Lo. The diagonalized shape allows the

Petition 870190091580, of 9/13/2019, p. 62/92

59/81 Mahalanobis distance d (fi_ _o ) is expressed in terms of the sum:

[00217] In this example, this is the expression that is evaluated to determine the Mahalanobis distance d (fi_ _o ). Therefore, process 1011 generates a Mahalanobis distance d (fi_ _o ). When finished, process 1012 transfers it to the SELECT POSE 1013 process.

[00218] In the SELECT POSE 1013 process, the hand gesture pose with the shortest Mahalanobis distance d (fi_ _o ) is selected if the Mahalanobis distance d (fi_ _o ) is less than the maximum Mahalanobis distance in the training database for that hand gesture pose. If the Mahalanobis distance d (fi_ _o ) is greater than the maximum Mahalanobis distance in the hand gesture pose training database, no hand gesture pose is selected. The SELECT POSE 1012 process transfers it to the TEMPORAL FILTER PROCESS 1014.

[00219] The TEMPORAL FILTER process 1014 determines whether the result of process 1013 provided the same result consecutively a predetermined number of times. If process 1013 provided the same result the predetermined number of times, the process TEMPORAL FILTER 1014 transfers it to the GESTURE POSE verification process 1015 and, otherwise, returns. The predetermined number of times is selected so that the TIME FILTER 1014 process prevents transient oscillations or detections when switching between hand gesture poses.

[00220] The GESTURE POSE verification process 1015 determines whether the selected hand gesture pose is the hand gesture pose used in a hand gesture trajectory. If the selected hand gesture pose is the hand gesture pose used on a trajectory

Petition 870190091580, of 9/13/2019, p. 63/92

60/81 hand gesture, the GESTURE POSE verification process 1015 transfers the processing to the SPEED SEQUENCE GENERATION 1020 process and otherwise transfers the processing to the POSE CHANGE 1016 verification process.

[00221] The POSE CHANGE 1016 verification process determines whether the hand gesture pose has changed since the last pass by method 1000. If the selected hand gesture pose is the same as the result of the temporal filtered gesture pose immediately previous, the POSE CHANGE verification process 1016 returns by RETURN 1003 and, otherwise, transfers it to the MAPPING IN SYSTEM EVENT 1030 process.

[00222] the MAPPING IN SYSTEM EVENT 1030 process maps the selected hand gesture pose to a system event, for example, the system event assigned to the hand gesture pose is sought. Upon finding the system event, the MAP IN SYSTEM EVENT 1030 process transfers processing to the INJECT SYSTEM EVENT 1031 process.

[00223] In one aspect, the INJECT SYSTEM EVENT process 1031 sends the system event to an event handler on system controller 140 (figure 1). In response to the system event, system controller 140 sends an appropriate system command to controllers and / or another device in system 100. For example, if the hand gesture pose is assigned to an interface turn event event user, system controller 140 sends a command to monitor controller 150 to turn on the user interface. The monitor controller 150 performs the user interface module part 155 on the processor 150 required to power the user interface.

[00224] When the hand gesture pose is the hand gesture pose

Petition 870190091580, of 9/13/2019, p. 64/92

61/81 used to make a trajectory, the processing in method 1000 transfers from the process of checking POSE DE GESTO 1015 to the SPEED SEQUENCE GENERATION process 1020. In one aspect, the main characteristic used for recognition of the management gesture trajectory hand is a unit velocity vector. The unit velocity vector is invariant for the starting position of the gesture. In addition, a normalized velocity vector takes into account variations in the size or speed of the gesture. Thus, in process 1020, the control point samples are converted into a normalized control point speed sequence, that is, a unit speed vector sequence.

'cp and <A) ^{- x} cp ecrí) ^-x cp ech ^- 1) ^- í ^ ^X cp «ιΆλυρ ec' ^ ^z cp ecí ^V cn ec (t) u (t) = | fort = 1,2, ..., Nl | ^V cp_ec (f) | [00225] Upon completing the GENERATE SPEED SEQUENCE 1020 process, the 1020 process transfers the processing to the CONVERT SPEED SEQUENCE TO SYMBOL SEQUENCE 1021 process. As noted above, the discrete Markov Hidden Model A requires a sequence of discrete symbols as input. In process 1021, the discrete symbols are generated from the normalized control point speed sequence using vector quantization.

[00226] In one aspect, vector quantization was performed using a modified K averaging cluster with the proviso that the process stops when the cluster assignments stop changing. Although grouping of K means is used, the process leverages the fact that the characteristics are unit vectors. In this case, vectors are grouped that are of similar direction. This is done using the dot product between each unit characteristic vector and the normalized cluster center vectors as the

Petition 870190091580, of 9/13/2019, p. 65/92

62/81 similarity.

[00227] The grouping is initialized with random vector assignments in thirty-two clusters, and the overall process is iterated multiple times, when the best grouping result is selected based on a cost metric within the maximum total grouping. It should be noted, in this case, that the cost within the grouping is based on a similarity measure. Each of the resulting groupings is assigned a unique index, which serves as a symbol for the Hidden Markov Model. An input vector is then mapped to its nearest cluster average, and the corresponding index for that cluster is used as the symbol. In this way, a sequence of unit velocity vectors can be translated into a sequence of indices or symbols.

[00228] In one aspect, grouped vectors are assigned a symbol based on a two-dimensional vector quantization code book in eight fixed directions. Thus, process 1020 generates a sequence of symbols observed and transfers to the process GENERATE PROBABILITY OF MANAGEMENT 1023.

[00229] In one aspect, to determine which gesture corresponds to the observed sequence of symbols, the process GENERATE PROBABILITY OF GESTURE 1023 uses the direct recursion algorithm with the Markov Hidden Models to find the probability that each gesture corresponds to the sequence of symbols observed. The direct recursion algorithm is described in Rainer, A Tutorial on Hidden Markov Models and Selected Applications in Speech Recognition, which was previously incorporated here by reference. When completing the process GENERATE PROBABILITY OF MANAGEMENT 1023, the processing transfers to the process SELECT PATH 1024.

[00230] In the process SELECT PATH 1024, the path

Petition 870190091580, of 9/13/2019, p. 66/92

63/81 of the hand gesture with the highest probability among the permissible trajectory gesture models in the Markov Hidden Model. This probability must also be greater than a given threshold to be acceptable. If the highest probability is not greater than the threshold, no trajectory of the hand gesture is selected. This threshold should be adjusted to maximize recognition accuracy, while avoiding false recognition.

[00231] Upon completion, the SELECT PATH 1024 process transfers the processing to the PATH FOUND 1025 verification process. If the SELECT PATH 1024 process selected a hand gesture path, the PATH FIND 1025 verification process transfers the processing to the MAPPING IN SYSTEM 1030 EVENT process and, otherwise, returns by RETURN 1004.

[00232] The MAPPING IN SYSTEM EVENT 1030 process maps the selected hand gesture path in a system event, for example, the system event assigned to the hand gesture path is sought. Upon finding, the system event, the MAPPING IN SYSTEM EVENT 1030 process transfers the processing to the INJECT SYSTEM EVENT 1031 process.

[00233] In one aspect, the INJECT SYSTEM EVENT process 1031 sends the system event to the event handler on system controller 140 (figure 1). In response to the system event, system controller 140 sends an appropriate system command to the appropriate controller (s) or device. For example, if the system event is assigned an action on the user interface, system controller 140 sends a command to monitor controller 150 to perform that action on the user interface, for example, changing the field display mode surgical.

Presence Detection Process

Petition 870190091580, of 9/13/2019, p. 67/92

64/81 [00234] In yet another aspect, as described above, the tracked position of at least a part of the 180B surgeon's hand is used to determine whether the hand is present in a 621 master manipulator end effector. Figure 11 is a process flow chart of an aspect of a presence detection process 1100 performed, in one aspect, by the hand tracking controller 130 in system 100. Process 1100 is performed separately for each of the surgeon's hands, in one aspect.

[00235] In the process GET ANGLES OF ARTICULATION 1110, the articulation angles of the master tool manipulator 620 (figure 6B) are measured. The process GET ANGLES FROM ARTICULATION 1110 transfers the processing to the process GENERATE KINEMATICS AHEAD 1111.

[00236] As the lengths of the various links in the master tool manipulator 620 are known, and the position of the base 629 of the master tool manipulator 620 is known, geometric relationships are used to generate the location of the master tool handle 621 in the system coordinates of master workspace 680. Thus, process GENERATE KINEMATICS FRONT 1111 generates the Pmtm position of master tool handle 621 in the coordinate system of master workspace 680 using process angles 1110. The process GENERATE KINEMATICS AHEAD 1111 transfers the processing to the MAP THE WORLD COORDINATES 1112 process.

[00237] The MAP WORLD COORDINATE 1112 process maps the Pmtm position in the coordinate system of the master workspace 680 to a position P _m tm_wc in the world coordinate system 670 (figure 6A). Specifically,

Pintin_wc ^- T _ws * pmtm [00238] where ^WC Tws is a rigid homogeneous transformation of

Petition 870190091580, of 9/13/2019, p. 68/92

65/81 four by four, which maps the coordinates in the workspace coordinate system 680 to coordinates in the world coordinate system 670.

[00239] When finished, the MAPPING WORLD COORDINATES 1112 process transfers the processing to the GENERATE SEPARATION OF HAND OF EFFECTOR TERMINAL 1130 process.

[00240] With the return to the process RECEIVE HAND LOCATION DATA 1120, the process RECEIVE HAND LOCATION DATA 1120 receives (retrieves) the position and orientation of the index finger (Pindex, Rindex) and the position and orientation of the thumb (Pthumb , Rthumb). The position and orientation of the index finger (Pindex, Rindex) and the position and orientation of the thumb (Pthumb, Rthumb) are based on data from the tracking system. The RECEIVE HAND LOCATION DATA 1120 process transfers processing to the GENERATE HAND POSITION 1121 process.

[00241] The GENERATE HAND POSITION 1121 process maps the position and orientation of the index finger (Pindex, Rindex) and the position and orientation of the thumb (Pthumb, Rthumb) to a position and orientation of the control point in the tracking coordinate system as described above, and that description is hereby incorporated by reference. The Phand position is the position of the control point in the tracking coordinates. The GENERATE HAND POSITION 1121 process transfers processing to the MAP THE WORLD COORDINATES 1122 process.

[00242] The use of the position of the control point in the presence detection is only illustrative and is not intended to be limiting. In view of this exposure, presence detection could be done, for example, using the position of the tip of the indicator and using the position of the tip of the thumb, or using only one of these positions. The processes described below are equivalent for each of the various positions

Petition 870190091580, of 9/13/2019, p. 69/92

66/81 associated with a part of a human hand.

[00243] The MAP WORLD COORDINATE 1122 process maps the Phand position in the tracking coordinates to a Phand_wc position in the 670 world coordinate system (Fig 6A). Specifically,

P _ WCrp £ ¹ hand_wc ~ ¹ tc Phand [00244] where ^wc Tt _c is a rigid homogeneous transformation of four by four, which maps the coordinates in the tracking coordinate system 650 to coordinates in the world coordinate system 670.

[00245] When finished, the MAPPING WORLD COORDINATE 1122 process transfers the processing to the GENERATE HAND SEPARATION process in TERMINAL 1130 EFFECTOR.

[00246] The process GENERATE HAND TERMINAL separation effector 1130 generates a separation distance d between _sep Pmtm_wc position in the world coordinate system 670 and Phand_wc position in the world coordinate system 670. In one aspect, the distance dsep separation is:

^d sep

P mtmivc Phand_ _wc [00247] When finished, the process GENERATE HAND SEPARATION IN EFFECTOR TERMINAL 1130 transfers the processing to the SAFE DISTANCE verification process 1131.

[00248] The SAFE DISTANCE verification process 1131 compares the separation distance dsep at a safe distance threshold. This threshold should be small enough to be conservative, while still allowing the surgeon to change footprints or manipulate the most distal end of the terminal effector. If the dsep separation distance is less than the safe distance threshold, the SAFE DISTANCE verification process 1131 transfers to the

Petition 870190091580, of 9/13/2019, p. 70/92

67/81 HANDLING PRESENCE process 1140. Conversely, if the separation distance dsep is greater than the safe distance threshold the SAFE DISTANCE verification process 1131 transfers it to the HANDLING PRESENCE process 1150.

[00249] The HANDCall PRESENCE 1140 process determines whether system 100 is in teleoperation. If system 100 is in teleoperation, no action is required, and teleoperation is allowed to continue, and thus process 1140 transfers to start process 1100 again. If the system 100 is not teleoperating, the HAND CONNECTED PRESENCE process 1140 sends a present hand event to the INJECT SYSTEM EVENT process which, in turn, sends the present hand event to the system controller 140.

[00250] The HAND OFF PRESENCE 1150 process determines whether system 100 is in teleoperation. If system 100 is not teleoperating, no action is required and thus process 1150 transfers to start process 1100 again. If system 100 is in teleoperation, the HAND OFF PRESENCE process 1150 sends a hand event not present to the INJECT SYSTEM EVENT process which, in turn, sends the hand event present to system controller 140.

[00251] System controller 140 determines whether the present hand event or the non present hand event requires any change in the system's operating mode and sends an appropriate command. In one aspect, system controller 140 enables teleoperation in response to a present hand event, for example, allows teleoperation, and disables teleoperation in response to a non present hand event if a teleoperated minimally invasive surgical instrument is attached to the handle of the master tool. As is known to those skilled in the art, a surgical instrument

Petition 870190091580, of 9/13/2019, p. 71/92

68/81 minimally invasive teleoperated is attachable to and detachable from a master tool handle.

[00252] In other respects, the present hand event and the non present hand event are used by system controller 140 in combination with other events to determine whether to allow teleoperation. For example, detecting the presence of the surgeon's head can be combined with detecting the presence of the surgeon's hand or hands in determining whether to allow teleoperation.

[00253] Likewise, as described above, the present hand event and the non present hand event are used by system controller 140 to control the monitor from a user interface on a minimally invasive surgical system monitor. When system controller 140 receives the hand event not present, if the user interface is not powered on, system controller 140 sends a command to monitor controller 150 to turn on the user interface. Monitor controller 150 runs the part of user interface module 155 on processor 150 required to power up the user interface. When system controller 140 receives the present hand event, if the user interface is on, system controller 140 sends a command to monitor controller 150 to shut down the user interface. Monitor controller 150 performs the part of user interface module 155 on processor 150 necessary to shut down the user interface.

[00254] The present hand event and non present hand event can be used by system controller 140 in combination with other events to determine whether to display the user interface. Thus, the user interface monitor control and teleoperation control are examples of system mode control using presence detection and are not intended to be limiting to these two.

Petition 870190091580, of 9/13/2019, p. 72/92

69/81 specific system control modes.

[00255] For example, presence detection could be used to control a substitute visual, such as those described more fully below. Likewise, combinations of the various modes, for example, teleoperation and substitute visual display, could be controlled by system controller 140 based on the present hand event and the non present hand event.

[00256] Likewise, hand presence detection is useful for eliminating dual purpose of the 621L, 621R master tool handles, for example, by pushing a pedal and then using the 621L, 621R master tool handles to control a user interface that appears on the 185B surgeon console. When the master tool handles are dual-purpose, for example, used to control both a surgical instrument and a user interface, the surgeon typically has to press a pedal to switch to user interface operating mode. If, for some reason, the surgeon does not press the pedal, but believes that the system has switched to a user interface operating mode, the movement of the master tool handle may result in an undesirable movement of the surgical instrument. The 1100 presence detection process is used to prevent this problem and to eliminate the dual purpose of the master tool handles.

[00257] With the presence detection process 1100, in an example, when the hand presence event off is received by system controller 140, system controller 140 sends a system command to lock the 6201 master tool handlers , 620R (figure 6A) in place and sends a system command to monitor controller 150 to present the user interface on the 185B surgeon console monitor. The movement of the surgeon's hand is tracked and is used to control elements in the

Petition 870190091580, of 9/13/2019, p. 73/92

70/81 user, for example, move a slide switch, change the display and others. As indicated above, the control point is mapped in the eye coordinate frame and, therefore, can be associated with the location of an element in the user interface. The movement of the control point is used to manipulate this element. This is done without the surgeon having to activate a pedal and is done so that the surgeon cannot inadvertently move a surgical instrument. This eliminates the problems associated with using the master tool handles to control both the surgical instrument and the user interface.

[00258] In the example above, the world coordinate table is an example of a common coordinate table. The use of the world coordinate table as the common coordinate table is illustrative only and is not intended to be limiting.

[00259] Master Finger Tracking Handle [00260] Figure 12 is an illustration of an example of a 1270 Master Finger Tracking Handle. The 1270 Master Finger Tracking Handle is an example of a 170 Finger Tracking Handle. , 270.

[00261] The master finger tracking handle 1270 includes a compressible body 1210 and two finger loops 1220, 1230. The compressible body 1210 has a first end 1213 and a second end 1214. A body section 1215 extends between the first end 1213 and second end 1214.

[00262] The compressible body 1210 has an external outer surface. The outer outer surface includes a first portion 1216 and a second portion 1217. The first portion 1216, for example, an upper portion, extends between the first end 1213 and the second end 1214. The second portion 1217, for example, a portion bottom, extends between the first end 1213 and the second end 1214. The second portion 1217 is opposite and is

Petition 870190091580, of 9/13/2019, p. 74/92

71/81 removed from first portion 1216.

[00263] In one aspect, the outer outer surface is a surface of a fabric wrap. The fabric is suitable for use in an operating room. The fabric wrap closes a compressible foam. The foam is selected to provide resistance to compression and expansion when compression is released. In one aspect, several foam strips were included within the fabric wrap. The foam must also be able to bend so that a first portion 1216 is positioned between the first and second fingers of a human hand, when the tip of the first finger is moved towards the tip of the second finger.

[00264] Body section 1215 has a length L between the finger strap 1220 and the finger strap 1230. As explained above, the length L is selected to limit the separation between a first finger on the strap 1220 and a second finger on the handle 1230 (see figure 2A). [00265] In one aspect, the body section 1215 has a thickness T. As shown in figure 2C, thickness T is selected in the I mode, when the master finger tracking handle 1270 is configured so that region 1236 on the second portion 1217 of the outer outer surface adjacent to end 1214 and region 1226 in second portion 1217 adjacent to end 1213 are just touching, the second portion 1217 along length L is not in complete contact with itself.

[00266] The first finger loop 1220 is attached to the compressible body 1210 adjacent to the first end 1213. The loop 1220 extends around the region 1225 of the first portion 1216 of the outer outer surface of the compressible body 1210. With the placement of the first loop of finger 1220 on the first finger of the human hand, region 1225 contacts the first finger, for example, a first part of the first portion 1216 of the outer outer surface contacts the thumb.

Petition 870190091580, of 9/13/2019, p. 75/92

72/81 [00267] In this example, the finger strap 1220 has two ends: a first end of fabric 1221 A and a second end of fabric 1221 B. End 1221 A and end 1221 B are ends of a strip of fabric which is attached to body 1210. A piece of loop fabric 1222 B is attached to an inner surface of end 1221 B, and a piece of hook fabric 1222 A is attached to an outer surface of end 1221 A. An example of fabric hook and loop fabric is a nylon fastening tape that consists of two strips of nylon fabric: one with small hook threads, and the other with a coarse surface. The two strips form a strong bond when pressed together. An example of a commercially available fixing tape and VELCRO® fixing tape (VELCRO ® is a registered trademark of Velcro Industries B.V).

[00268] The second finger loop 1230 is attached to the compressible body 1210 adjacent to the second end 1214. The loop 1230 extends around a region 1235 of the first portion 1216 of the outer outer surface of the compressible body 1210. With the placement of the second finger loop 1230 on the second finger of the human hand, region 1235 contacts the second finger, for example, a second part of the first portion 1216 of the outer outer surface contacts the index finger. The second part 1235 of the first portion is opposite to and is removed from the first part 1225 of the first portion.

[00269] In this example, the finger loop 1230 also has two ends: a first end of fabric 1231 A and a second end of fabric 1231 B. End 1231 A and end 1231 B are ends of a strip of fabric that is attached to body 1210. A piece of loop fabric 1232 B is attached to an inner surface of end 1231 B, and a piece of hook fabric 1232 A is attached to the outer surface of surface 1231 A.

Petition 870190091580, of 9/13/2019, p. 76/92

73/81 [00270] A first location tracking sensor 1211 is attached to the first finger loop 1220. A second location tracking sensor 1212 is attached to the second finger loop 1230. The location tracking sensors can be any of the sensor elements described above. In one example, the location tracking sensors 1211, 1212 are passive electromagnetic sensors. Visual Substitute System [00271] In one aspect, the hand tracking control system is used to control any of a plurality of visual substitutes that can be used by a surgeon to act in place of another surgeon. For example, when surgeon 180 (figure 1A) is being guided by surgeon 181 using the master finger tracking handle 170, surgeon 181 uses the master finger tracking handle 170 to control a replacement look for a surgical instrument while whereas surgeon 180 uses the master tool handle to control a teleoperated slave surgical instrument.

[00272] Alternatively, the surgeon 181 can operate remotely or can control a virtual hand on the monitor. Likewise, surgeon 181 can demonstrate how to manipulate the master tool handle on the surgeon console by manipulating a virtual image of the master tool handle 621 that is displayed on the monitor. These substitute visual examples are illustrative only and are not intended to be limiting.

[00273] In addition, the use of the master finger tracking handle 170 while outside the surgeon console is also illustrative and is not intended to be limiting. For example, with the presence detection system described above, a surgeon on a surgeon console could move the hand of a master tool handle and then use that hand to remotely control another surgeon when the hand is tracked by the tracking system. of hand.

Petition 870190091580, of 9/13/2019, p. 77/92

74/81 [00274] To facilitate remote control, a visual replacement module (not shown) is processed as part of a one-way vision processing subsystem. In this regard, the performing module receives the position and orientation of the control point from the hand of the remote controller and provides stereo images, which are combined with the images from the endoscopic camera in real time and displayed in any combination of the 185 surgeon console, monitor of the assistant and monitor of the surgeon patient side interface 187.

[00275] When surgeon 181 initiates the remote control by performing a predefined action, for example, a hand gesture pose, a substitute visual system cycle is activated, for example, the substitute visual module is executed in a processor module . The particular action, for example, a hand gesture pose, used as the predefined action is not essential, as long as the system controller 140 (figure 1) is configured to recognize that action.

[00276] In one aspect, the replacement look is a virtual ghost instrument 1311 (figure 13) controlled by the master finger tracking handle 170, while the teleoperated slave surgical instrument 1310 is controlled by one of the master tool manipulators on the console surgeon 185. Surgeon 181 sees both instruments 1310 and 1311 on the display device 187, while surgeon 180 sees both instruments 1310 and 1311 on the stereoscopic monitor on the surgeon console 185. Using the virtual ghost instrument 1311 as a substitute look is illustrative only and is not intended to be limiting to that particular image. In view of this exhibition, other images can be used for the substitute visual, which facilitates the differentiation between an image that represents the substitute visual and an image of the actual terminal effector of the teleoperated slave surgical instrument.

[00277] The virtual ghost instrument 1311 looks similar to

Petition 870190091580, of 9/13/2019, p. 78/92

75/81 real instrument 1310, except that the virtual ghost instrument 1311 is displayed in a way that clearly distinguishes the virtual ghost instrument 1311 from the real instrument 1310 (for example, a transparent or translucent ghost image, a distinct color image and others). The control and operation of the 1311 virtual phantom instrument is the same as described above for a real teleoperated surgical instrument. Thus, surgeon 181 can manipulate the virtual ghost instrument 1311 using the master finger tracking handle 170 to demonstrate proper use of the teleoperated slave surgical instrument 1310. Surgeon 180 can mimic the movement of the virtual ghost instrument 1311 with the 1310 instrument.

[00278] Phantom virtual instruments are described more fully in the publication of US patent application jointly assigned No. US 2009/0192523 Al (filed March 31, 2009; which exposes the Synthetic Representation of a Surgical Instrument), which it is incorporated here by reference in its entirety. See also US Patent Application No. 12 / 485,503 (filed June 16, 2009, which exposes a Virtual Measurement Tool for Minimally Invasive Surgery); US patent application No. 12 / 485,545 (filed June 16, 2009, which exposes a Virtual Measurement Tool for Minimally Invasive Surgery); US patent application publication No. US 2009/0036902 Al (filed on August 11, 2008; which exposes Interactive User Interfaces for Minimally Invasive Robotic Surgical Systems); US Patent Application Publication No. US 2007/0167702 Al (filed December 30, 2005; which exposes Medical Robotic System that Provides Three-dimensional Remote Medicine); US Patent Application Publication No. US 2007/0156017 Al (filed December 30, 2005; which exposes Stereo Remote Medicine to

Petition 870190091580, of 9/13/2019, p. 79/92

76/81

Robotic Surgery) and publication of US patent application No. US 2010/0164950 Al (filed on May 13, 2009; which exposes Efficient 3-D Remote Medicine for Local Robotic Control), each incorporated herein by reference in its wholeness.

[00279] In another aspect, the substitute look is a pair of virtual hands 1410, 1411 (figure 14) controlled by the master finger tracking handle 170 and a second master finger tracking handle, which is not visible in figure 1. Teleoperated slave surgical instruments 1420, 1421 are controlled by the master tool handlers on the surgeon console 185. Surgeon 181 sees video image 1400 on display device 187, and surgeon 180 also sees video image 1400 on the stereoscopic monitor on surgeon console 185. Virtual hands 1410, 1411 are displayed in a way that clearly distinguishes them from other objects in the 1400 video image.

[00280] The opening and closing of the thumb and forefinger of a virtual hand are controlled using the ggrip handle closing parameter, which was described above. The position and orientation of the virtual hand are controlled by the position and orientation of the control point, as described above, which are mapped in the eye coordinate space, also as described above.

[00281] Thus, when surgeon 181 moves the surgeon's right hand in three dimensions, virtual hand 1411 follows the movement in video image 1400. Surgeon 181 can roll virtual hand 1411 to instruct surgeon 180 to roll the instrument teleoperated slave surgical 1421. Surgeon 181 can move virtual hand 1410 to a particular location and then use thumb and forefinger movement to instruct surgeon 180 to move teleoperated slave surgical instrument 1420 to that location and grab tissue . When surgeon 180 grabs tissue with instrument 1420, surgeon 181 can use virtual hand 1410 to instruct surgeon 180

Petition 870190091580, of 9/13/2019, p. 80/92

77/81 on how to move the fabric. This all takes place in real time, and the virtual hands 1410, 1411 are superimposed on the image of the stereoscopic endoscope. However, substitute visuals can also be used in a monoscopic system.

[00282] In another aspect, the surgeon 181 changes the view using a hand gesture pose so that the visual substitutes are a virtual ghost instrument 1510 and a virtual remote medicine device 1511, which are presented in the video image 1500 (figure 15). The remote medical device 1511 is controlled by the master finger tracking handle 170, while a second master finger tracking handle, which is not visible in figure 1, controls the virtual ghost instrument 1511.

[00283] The teleoperated slave surgical instruments 1520, 1521 are controlled by the master tool manipulators on the surgeon console 185. Surgeon 181 sees video image 1500 on display device 187, and surgeon 180 also sees video image 1500 on the stereoscopic monitor of the surgeon console 185. The device * p.48 paragraph [0262] l.4 OKvirtual remote medicine 1511 and the virtual phantom instrument 1411 are displayed in a way that clearly distinguishes them from other objects in the 1500 video image.

[00284] To operate remotely with the 1511 virtual remote medicine device, the surgeon 181 places the thumb and forefinger together as if holding an imaginary pen or pencil and then moves the right hand with the thumb and forefinger in that position to operate remotely on the displayed video image. In video image 1500, surgeon 181 thus positioned the thumb and forefinger and made the 1512 mark to illustrate where the tissue should be cut using the 1521 surgical instrument. After making the 1512 mark, the 1810 surgeon separated the thumb and the indicator and moved the virtual remote medical device 1511 to the position shown in video image 1500.

Petition 870190091580, of 9/13/2019, p. 81/92

78/81 [00285] The dialing capability of the virtual remote medicine device 1511 is controlled using the ggrip handle closure parameter described above. As indicated above, when the thumb and forefinger are just touching, the ggrip handle closing parameter is mapped to an initial value in a second range and, when the ggrip handle closing parameter is in the second range, the remote medicine to the 1511 remote medicine device. The position and orientation of the control point after being mapped in the eye coordinate system are used to control the movement of the 1511 virtual remote medicine device.

[00286] The above description and the accompanying drawings that illustrate aspects and modalities of the present invention should not be taken as limiting; claims define protected inventions. Various mechanical, compositional, structural, electrical and operational changes can be made without departing from the spirit and scope of this description and the claims. In some cases, well-known circuits, structures and techniques have not been shown or described in detail to avoid obscuring the invention.

[00287] Furthermore, the terminology of this description is not intended to limit the invention. For example, spatially relative terms, such as below, below, lower, above, upper, proximal, distal and others, can be used to describe a relationship of the element or characteristic with another element or characteristic, as illustrated in the figures. These spatially relative terms are intended to encompass different positions (that is, locations) and orientations (that is, rotational settings) of the device in use or operation, in addition to the position and orientation shown in the figures. For example, if the device in the figures is turned upside down, elements described as below or below other elements or characteristics would be,

Petition 870190091580, of 9/13/2019, p. 82/92

79/81 then, above or over the other elements or characteristics. Thus, the exemplary term below can encompass both the positions above and below. The device can be oriented in another way (rotated 90 degrees or in other orientations), and the spatially related descriptors used here are properly interpreted. Likewise, descriptions of the movement along and around the various axes include various positions and orientations of the special device.

[00288] The singular forms one, one, o and a are intended to include plural forms as well, unless the context indicates otherwise. The terms comprises, comprising, includes and others specify the presence of the declared characteristics, steps, operations, process elements and / or components, but does not exclude the presence or addition of one or more other characteristics, steps, operations, process elements, components and / or groups. Components described as coupled can be electrically or mechanically coupled directly or they can be coupled indirectly through one or more intermediate components.

[00289] Memory refers to a volatile memory, non-volatile memory or any combination of the two. A processor is coupled to a memory containing instructions executed by the processor. This could be done inside a computer system or, alternatively, through a connection to another computer using modems and analog lines or digital interfaces and a digital transport line.

[00290] Here, a computer program product includes a means configured to store computer-readable code needed for any or any combination of the processes described with respect to hand tracking or in which the computer-readable code for anyone or any combination of the processes described with

Petition 870190091580, of 9/13/2019, p. 83/92

80/81 with respect to hand tracking is stored. Some examples of computer program products are CD-ROM discs, DVD discs, flash memory, ROM cards, floppy disks, magnetic tapes, computer hard drives, servers on a network, and signals transmitted over a network representing the code for computer readable program. A non-transitory tangible computer program product includes a non-transitory tangible medium configured to store computer-readable instructions for any or any combination of the processes described with respect to the various controllers or in which the computer-readable instructions for any or any combination of the processes described in relation to the various controllers are stored. Non-transitory tangible computer program products are CDROM disks, DVD disks, flash memory, ROM cards, floppy disks, magnetic tapes, computer hard disks and other non-transitory physical storage media.

[00291] In view of this exhibition, the instructions used in any or any combination of the processes described with respect to hand tracking can be implemented in a wide variety of computer system configurations using an operating system and the computer programming language of interest to the user.

[00292] The use of different memories and processors in figure 1 is only illustrative and is not intended to be limiting. In some ways, a single hardware processor could be used, and in other ways, multiple processors could be used.

[00293] Likewise, for each illustration, the various processes were distributed between a handheld tracking controller and a system controller. This is also illustrative and is not intended to be limiting. The various processes can be distributed by

Petition 870190091580, of 9/13/2019, p. 84/92

81/81 controllers or consolidated into one controller without changing the principles of operation in the hand tracking process.

[00294] All examples and illustrative references are non-limiting and should not be used to limit claims to the specific implementations and modalities described herein and their equivalents. Headings are for formatting only and should not be used to limit material in any way, because the text under a heading can refer to or apply to the text under one or more headings. Finally, in view of this exhibition, particular features described with respect to an aspect or modality may apply to other aspects or modalities of the invention exposed, even if not specifically shown in the drawings or described in the text.

Claims (9)

1. Minimally invasive surgical system (100) characterized by the fact that it comprises: a master tool handle (621, 621R, 621L) having a first location; a hand tracking system (186), wherein the hand tracking system (186) is configured to track a second location of a sensor element (170) mounted on part of a human hand; and a controller (130) coupled to the hand tracking system (186), where the controller is configured to: determine if the distance between a position of the human hand part and a position of the master tool handle (621, 621R, 621L) is within a threshold distance, and inject a system event into the minimally invasive surgical system (100) with based on the result of a determination of whether the distance is within a threshold distance. 2. System, according to claim 1, characterized by the fact that the system event is a hand event not present when the distance is greater than the threshold distance. 3. System, according to claim 1, characterized by the fact that the system event is a hand event present when the distance is less than the threshold distance. 4. System according to claim 1, characterized by the fact that the position of the part of the human hand comprises a position of the index finger (Pindicator) in the second location. 5. System, according to claim 1, characterized by the fact that the position of the part of the human hand comprises a position of the control point (Pcp) determined from the second location. Petition 870190091580, of 9/13/2019, p. 86/92 2/3 6. System according to claim 1, characterized by the fact that the sensor element (170) mounted on the part of the human hand comprises a plurality of fiducial markers. 7. System according to claim 1, characterized by the fact that the sensor element (170) mounted on the part of the human hand comprises a passive electromagnetic sensor. 8. System, according to claim 1, characterized by the fact that it additionally comprises: a processing unit (151); a system controller (140) configured to receive the system event; a user interface generated by a module (155) that runs on the processing unit (151); and a display device (187), wherein, when the system event is a hand event not present, the system controller (140) is configured to turn on the user interface monitor on the display device (187); and wherein, when the system event is a hand held event, the system controller (140) is configured to turn off the user interface monitor on the display device (187). 9. System, according to claim 1, characterized by the fact that it additionally comprises: a teleoperated minimally invasive surgical instrument (1310, 1420, 1421) that can be coupled and detachable from the master tool handle (621, 621R, 621L); and a system controller (140) configured to receive the system event; where, when the system event is a hand held event and the teleoperated minimally invasive surgical instrument (1310, 1420, 1421) is attached to the handle of the master tool (621, Petition 870190091580, of 9/13/2019, p. 87/92 3/3 621R, 621L), the system controller (140) is configured to allow teleoperation of the teleoperated minimally invasive surgical instrument (1310, 1420, 1421); and where, when the system event is a non-present hand event, the system controller (140) is configured to interrupt the teleoperation of the teleoperated minimally invasive surgical instrument (1310, 1420, 1421).