KR101824442B1 - Method and system for hand presence detection in a minimally invasive surgical system - Google Patents

Abstract

In a minimally invasive surgical system, a hand tracking system tracks the position of a sensor element mounted on a portion of a human hand. A system control variable is generated based on the position of a portion of the human hand. The operation of the minimally invasive surgical system is controlled using system control parameters. Thus, the minimally invasive surgical system includes a hand tracking system. The hand tracking system tracks the position of a portion of a human hand. The controller connected to the hand tracking system switches this position to a system control variable and provides an instruction based on system control variables to the minimally invasive surgical system.

Description

[0001] METHOD AND SYSTEM FOR HAND PRESENCE DETECTION IN A MINIMALLY INVASIVE SURGICAL SYSTEM [0002]

Aspects of the present invention relate to the control of a minimally invasive surgical system, and more particularly to the use of a physician's hand movement in controlling a minimally invasive surgical system.

Methods and techniques for tracking hand positions and gestures are known. For example, some video game controllers use hand tracking inputs. For example, the Nintendo Wii® game platform supports wireless location and orientation detection remote control (Wii is a registered trademark of Nintendo of America Inc. of Redmond, Washington). The use of gestures and other body movements, such as swinging bats or waving magic rods, provide a basic game element for this platform. Sony Playstation Move also has features similar to the Nintendo Wii® gaming platform.

CyberGlove Systems' wireless CyberGlove® motion capture data glove includes 18 data sensors, two bend sensors on each finger, four abduction sensors, a sensor that measures the intersection of the thumb, elbow arch, wrist bend and wrist abduction (CyberGlove® is a registered trademark of CyberGlove Systems LLC, located in Senegal, California). CyberGlove® motion capture data If the three-dimensional tracking system is used on the glove, x, y, z, yaw, pitch, roll position and hand orientation information can be used. The CyberGlove® motion capture data glove's motion capture system is used in digital prototype evaluation, virtual reality biomechanics, and animation.

Another data glove with 40 sensors is Measurand Inc.'s ShapeHand data glove. Measurand Inc.'s ShapeClaw portable lightweight hand motion capture system includes a flexible ribbon system that captures forefinger and thumb motions along with the position and orientation of the hands and forearms in space.

Kim et al., "Analysis of 3D Hand Trajectory Gesture Using Stroke-Based Complex Hidden Markov Model" (Applied Intelligence, Vol.15 No.2, p.131-143, September-October 2001) We use the three-dimensional trajectory input by the user to recognize the recognition of the gesture. Kim and I propose this type of input because the three-dimensional trajectory provides a higher discrimination than the two-dimensional gesture commonly used in video-based approaches. Kim and I used a Polhemus magnetic location sensor attached to the back of the Fakespace PinchGlove in their experiments. PinchGlove provides a means for the user to signal the start and end of the gesture while the Polhemus sensor captures the three-dimensional trajectory of the user's hand.

Elena Sanchez-Nielsen et al., &Quot; Recognition of hand gestures for human-machine interaction "(Journal of WSCG, Vol. 12, No.1-3, ISSN 1213-6972, WSCG'2004, February 2-6 20003, Plzen Czech Republic) proposed a real-time screen system for use in a visual interaction environment with hand gesture recognition using multi-purpose hardware such as personal computers and webcams and inexpensive sensors. Pragati Garg et al., "Screen-Based Hand Gesture Recognition" (49 World Academy of Science, Engineering and Technology, 972-977 (2009)) reviewed screen-based hand gesture recognition. One conclusion presented was that most approaches rely on some fundamental assumptions that may be appropriate in a controlled laboratory environment, but not for any environment. The authors said, "The computer screen method for hand gesture interfaces must outperform current capabilities in terms of robustness and speed to achieve interactive participation and usability." In the medical field, gesture recognition was considered for asymmetric browsing of radiological images. See Juan P. Wachs et al., "Gesture-Based Tool for Asynchronous Browsing of Radiographic Images" (Journal of the American Medical Informatics Association (2008; 15: 321-323, DOI 10.1197 / jamia.M24).

In one aspect, a hand tracking system of a minimally invasive surgical system tracks the position of a portion of a human hand. A system control variable of the minimally invasive surgical system is created based on the position of a portion of the human hand. The operation of the minimally invasive surgical system is controlled using these system control variables.

In one aspect, sensor elements mounted on a portion of a human hand are tracked to obtain positions of a portion of a human hand. Based on this position, the position and orientation of the control point are generated. The remote control of the device in a minimally invasive surgical system is controlled based on the position and orientation of the control points. In one embodiment, the device is a remote control dependent surgical instrument. In another aspect, the device is a virtual proxy presented to a video image of a surgical site. Examples of virtual proxies include virtual dependent surgical instruments, virtual hands, and virtual telestration devices.

In another aspect, in addition to the position and orientation of the control point, a grip closure variable is created. The grip of the end actuator of the remote control dependent surgical instrument is controlled based on the grip closing parameter.

In another aspect, the system control variable is the location and orientation of the control point used for remote control of the dependent surgical instrument. In another aspect, the system control variable is determined from both hands. The system control variable is the position and orientation of the control point for one of the hands and the position and orientation of the control point for the other hand. The control point is used for remote manipulation of endoscopic camera manipulators in a minimally invasive surgical system.

In yet another aspect, a sensor element mounted on a portion of a second human hand is tracked in addition to the sensor element on a portion of the human hand. The position and orientation of the second control point is generated based on the position of the part of the second human hand. In this aspect, both the control point and the second control point are used for remote control.

In another aspect, sensor elements mounted on fingers of a human hand are tracked. The operation between the fingers is determined, and based on this operation, the orientation of the remote control dependent surgical instrument in the minimally invasive surgical system is controlled.

When this operation is the first operation, the control includes swinging the distal end of the dependent surgical wrist about the pointing direction. When the operation is a second operation different from the first operation, the control includes the yaw operation of the dependent surgical instrument wrist.

The minimally invasive surgical system includes a hand tracking system and a controller coupled to the hand tracking system. The hand tracking system tracks the position of a plurality of sensor elements mounted on a portion of a human hand. The controller transforms this position into the position and orientation of the control point. The controller sends an instruction to move the device of the minimally invasive surgical system based on the control point. Again, in one aspect, the device is a remote manipulation dependent surgical device, but in other aspects the device is a virtual proxy presented to the video image of the surgical site.

In one aspect, the system also includes a primary finger tracking device that includes a plurality of tracking sensors. The main finger tracker also includes a compression element, a first finger loop fixed to the compression body, and a second finger loop fixed to the compression body. A first tracking sensor of the plurality of tracking sensors is attached to the first finger loop. A second tracking sensor of the plurality of tracking sensors is attached to the first finger loop.

Thus, in one aspect, the minimally invasive surgical system includes a primary finger tracker. The main finger track device includes a compression body, a first finger ring fixed to the compression body, and a second finger ring fixed to the compression body. A first tracking sensor is attached to the first finger loop. A second tracking sensor is attached to the second finger loop.

The compression body includes a first end, a second end, and an outer outer surface. The outer outer surface includes a first portion extending between the first end and the second end and a second portion opposite and opposite to the second portion extending between the first end and the second end.

The compression body also has a length. This length is chosen to limit the separation between the first finger and the second finger of a human hand.

The first finger ring is secured to the compression body adjacent the first end and extends around a first portion of the outer outer surface. When the first finger ring is disposed on the first finger of the human hand, a first portion of the first portion of the outer outer surface is brought into contact with the first finger.

The second finger ring is secured to the compression body adjacent the second end and extends around a first portion of the outer outer surface. When the second finger ring is disposed on the second finger of the human hand, a second portion of the first portion of the outer outer surface is brought into contact with the second finger. When the first finger and the second finger move towards each other, the compression body is positioned between the two fingers to provide resistance to movement.

The thickness of the compression body is selected such that the compression body is not fully compressed when the tip of the first finger touches the end of the second finger lightly. The compression body is configured to provide tactile feedback corresponding to the gripping force of the end effector of the remote control dependent surgical instrument.

In one aspect, the first and second tracking sensors are passive electromagnetic sensors. In another aspect, each passive electromagnetic tracking sensor has a degree of freedom of six.

The method of using the primary finger tracker includes tracking a first position of a sensor mounted on a first finger of a human hand and a second position of another sensor mounted on a second finger. Each position has a degree of freedom of N and N is an integer greater than zero. The first position and the second position are mapped according to the control point position. The control point position has 6 degrees of freedom. The degree of freedom of 6 is less than 2 * N degrees of freedom. The first position and the second position are also mapped according to a variable having a single degree of freedom. In a minimally invasive surgical system, remote manipulation of the dependent surgical instrument is controlled based on the control point position and parameters.

In the first aspect, the variable is the grip closure distance. In a second aspect, the variable comprises orientation. In another embodiment, N is 6, but in another embodiment, N is 5.

In another aspect, a sensor element mounted on a portion of a human hand is traced to obtain a plurality of positions of a portion of a human hand. A hand gesture from a plurality of known hand gestures is selected based on the plurality of positions. The operation of the minimally invasive surgical system is controlled based on the hand gesture.

A hand gesture can be either a hand gesture pose, a hand gesture trajectory, or a combination of a hand gesture pose and a hand gesture trajectory. The user interface of the minimally invasive surgical system is controlled based on the hand gesture pose when the hand gesture is a hand gesture pose and the plural base hand gestures include multiple base hand gesture pauses.

Further, in one aspect, when the hand gesture is a hand gesture pose, the hand gesture selection includes generating the observed feature set from the plurality of tracked positions. The observed feature set is compared to feature sets of a plurality of known hand gesture pauses. One of the base hand gestures is selected as a hand gesture pose. The selected base hand gesture pose is mapped according to a system command, and the system command is triggered in a minimally invasive surgical system.

In another aspect, when the hand gesture includes a hand gesture trajectory, the user interface of the minimally invasive surgical system is controlled based on the hand gesture trajectory.

In a minimally invasive surgical system with a hand tracking system and a controller, the controller transforms this position into a hand gesture. The controller sends an instruction to modify the operating mode of the minimally invasive surgical system based on the hand gesture.

In another aspect, a sensor element mounted on a portion of a hand is traced to obtain the position of a portion of a human hand. Based on this position, the method determines whether the position of a portion of the human hand is within a threshold distance from the position of the main tool grip of the minimally invasive surgical system. Based on the determined result, the operation of the minimally invasive surgical system is controlled. In one aspect, remote control of the remote control dependent surgical instrument connected to the main tool grip is controlled based on the determination result. In another aspect, an indication of the user interface or an indication of the proxy visual is controlled based on the determination result.

In one aspect, the position of a portion of the human hand is specified by the control point position. In another aspect, the position of a portion of the human hand is a forefinger position.

The minimally invasive surgical system includes a hand tracking system. The hand tracking system tracks the position of a portion of a human hand. The controller uses this position to determine whether the physician's hand is close enough to the main tool grip to allow certain operations of the minimally invasive surgical system.

The minimally invasive surgical system also includes a controller coupled to the hand tracking system. The controller converts the position to a system control variable and provides a command based on system control variables to the minimally invasive surgical system.

1 is a high-dimensional schematic diagram of a minimally invasive remote controlled surgical system including a hand tracking system.
FIGS. 2A-2G are examples of various configurations of a hand-tracking main tool grip used to control a remote manipulation dependent surgical instrument of the minimally invasive remote manipulation surgical system of FIG.
Figures 3a-3d are examples of hand gesture pauses used to control the system mode of the minimally invasive teleoperation system of Figure 1;
Figs. 4A-4C are examples of hand gesture trajectories also used to control the system mode of the minimally invasive telescoping surgical system of Fig.
Figure 5 illustrates the placement of origin markers for hand tracking in a camera-based tracking system.
6A and 6B are more detailed views of the pseudo console of FIG. 1, including examples of coordinate systems used in hand tracking by the minimally invasive remote control surgery system of FIG.
7 illustrates the position and coordinate system used in the hand tracking by the hand-wearable main tool grip and the minimally invasive remote manipulation surgical system of FIG. 1 in more detail.
Figure 8 is a process flow diagram of the procedure used in the tracking system to track the fingers of the hand and used to generate data for remote manipulation of the dependent surgical instrument of the minimally invasive remote manipulation surgical system of Figure 1;
9 is a more detailed flowchart of the map position data according to the control position and grip parameters of FIG.
FIG. 10 is a flowchart of a process used in a tracking system capable of recognizing hand gesture pose and hand gesture locus.
11 is a process flow diagram of a process used in a tracking system for hand presence detection.
12 illustrates an example of a main finger tracking device.
Figure 13 illustrates a video image presented on a display device containing a proxy visual, which in this example includes a virtual ghost mechanism and a remote control dependent surgical instrument.
Figure 14 illustrates a video image presented on a display device including a proxy visual, which in this example includes a pair of virtual hands and a remote control dependent surgical instrument.
Figure 15 illustrates a video image presented on a display device containing a proxy visual, which in this example includes a virtual telestration device, a virtual ghost mechanism, and a remote control dependent surgical instrument.
In the drawing, the first of the three-digit reference numbers represents the drawing number of the drawing in which the element with the reference number first appears, and the first two digits of the four- The drawing number of the indicated drawing is displayed.

The positions used herein include position and orientation.

As used herein, a hand gesture, sometimes referred to as a gesture, includes a hand gesture pose, a hand gesture trajectory, and a combination of a hand gesture pose and a hand gesture trajectory.

Embodiments of the present invention provide a minimally invasive surgical system, such as the control of the da Vinci® minimally invasive remote manipulation system available from Intuitive Surgical, Inc. of Sunnyvale, California, using hand positional information to control the minimally invasive surgical system. Strengthen capacity. The measured position of one or more fingers of the hand is used to determine system control variables, which in turn are used to trigger system commands in the surgical system. The system command depends on the position of the person to which the hand position is to be tracked, i.e. whether or not the person is in the physician's console.

If the measured position is for fingers of a person's hand not in the pseudo-console, the system command changes the orientation of a portion of the remote control dependent surgical instrument based on the combination of hand orientation and relative motion of the two fingers of the hand And an instruction to move the distal end of the remote control dependent surgical instrument to follow the motion of a portion of the hand. If the measured position is for the fingers of a person's hand in the physician's console, then the system command includes an instruction to allow or inhibit the operation of the dependent surgical instrument to continue to follow the operation of the main tool grip. If the measured position is for the fingers of a person's hand not at the physician's console, or for the fingers of a person's hand at the physician's console, the system command will take action based on the hand gesture pose Commanding a portion of the system or system, and commanding a portion of the system or system to take action based on the hand gesture trajectory.

1 is a high-level schematic diagram of a minimally invasive remote controlled surgical system 100, e.g., a da Vinci® surgical system, including a hand tracking system. There are other components, cables, etc. associated with the da Vinci® surgical system, but these are not illustrated in FIG. 1 to avoid complicating the content. Additional information regarding minimally invasive surgical systems may be found, for example, in U.S. Patent Application No. 11 / 762,165 (filed June 13, 2007, entitled "Minimally Invasive Surgical System") and U.S. Patent No. 6,331,181 Issued December 18, "Surgical Robot Tool, Data Structure, and Usage"), all of which are incorporated herein by reference. In addition, U.S. Patent No. 7,155,315 (December 12, 2005, "Camera Reference Control in Minimal Invasive Surgery Devices") and 7,574,250 (filed February 4, 2003, entitled "Image Transfer Device for Remote Robotic Systems And Methods "), all of which are incorporated herein by reference.

In this example, the system 100 includes a cart 110 having a plurality of manipulators. Each manipulator and the remote control dependent surgical instrument controlled by the manipulator may be releasably connected to the main tool manipulator of the physician console 185 and additionally they may be mechanically ungrounded, sometimes referred to as the main finger tracking grip 170 Can be detachably connected to the non-powered main finger tracking grip 170 as well.

A stereoscopic endoscope 112 mounted on the manipulator 113 provides an image of the surgical site 103 in the patient 111 and this is displayed on the display 187 and the display of the pseudo console 185. [ The image includes an image of the dependent surgical devices in the field of view of the stereoscopic endoscope (112). The interaction between the main tool manipulator of the pseudo console 185 and the dependent surgical apparatus and the stereoscopic endoscope 112 is identical to that of the known system and thus those skilled in the art are aware.

In one aspect, if the physician 181 moves at least one finger of the doctor's hand, the sensor in the main finger track grip 170 accordingly causes a change in position. The hand tracking transmitter 175 provides a field through which the new position and orientation of the finger can be sensed by the main finger tracking grip 170. [ The newly detected position and orientation is provided to the hand tracking controller 130.

In one aspect, as described more fully below, the hand tracking controller 130 maps the detected position and orientation to the eye coordinate system of the physician 181 according to the control point position and the control point orientation. The hand tracking controller 130 transmits this position information to the system controller 140 and then sends the system command to the remote control dependent surgical instrument connected to the main finger track grip 170. [ As described more fully below, using the main finger track grip 170, the physician 181 can control not only the grip of the end actuator of, for example, the remote control dependent surgical instrument, but also the wrist of the wrist Roll and yaw.

In another aspect, a hand tracking of at least a portion of the hand of the doctor 181 or of the hand of the doctor 180 is used by the hand tracking controller 130 to determine whether a hand gesture pose is created by the physician, It is possible to determine whether a combination of pose and hand gesture trajectories is created. Each hand gesture pose and hand gesture pose and each combined trajectory are mapped according to different system commands. System commands control system mode changes, for example, and other aspects of the minimally invasive surgical system 100.

For example, instead of using a foot pedal and a switch as in a known minimally invasive surgical system, a hand gesture pose or a hand gesture that is a hand gesture trajectory can be used to (i) operate the remote control dependent surgical instrument associated with the main tool grip (Iii) control the endoscopic camera (which allows the main tool grip to perform endoscopic movement such as focus or electronic zoom, or (Iv) swapping the robot arm (which switches specific primary control between the two dependent mechanisms), and (v) performing TM TILEPRO swapping (TILEPRO is a trademark of Intuitive Surgical, Inc. of Sunnyvale, CA).

When the system 100 has only two main tool grips and the physician 180 wishes to control the motion of the dependent surgical instruments other than the two remote manipulation dependent surgical instruments connected to the two main tool grips, One hand gesture can be used to secure one or both of the two remote control dependent surgical instruments in place. The physician then uses a different hand gesture to associate one or both of the main tool grips with the other dependent surgical instruments fixed by the other manipulator arms, which in this embodiment is the main tool grip and another remote manipulation dependent surgical instrument Of swap relationships. The physician 181 performs the same process when there are only two main finger track grips in the system 100.

The hand tracking unit 186 mounted on the physician's console 185 tracks at least a portion of the hand of the physician 180 and transmits the sensed position information to the hand tracking controller 130. [ The hand tracking controller 130 determines when the physician's hand is close enough to the main tool grip to allow system follow-up, e.g., operation of the dependent surgical instrument to follow the operation of the main tool grip. As described more fully below, in one aspect, the hand tracking controller 130 determines the position of the physician's hand and the position of the corresponding main tool grip. If the difference between the two positions is at a fixed distance, for example below threshold separation, tracking is allowed, otherwise tracking is prohibited. Thus, the distance is used as a reference for the presence of a physician's hand in relation to the main tool grip on the pseudo console 185. In another aspect, when the position of the physician's hand with respect to the position of the main tool grip is less than the threshold separation, display of the user interface on the display is inhibited, for example the display is turned off. Conversely, when the position of the doctor's hand with respect to the position of the main tool grip exceeds the threshold value separation, the user interface is displayed on the display device, for example the display device is turned on.

Detecting the presence of a doctor 's hand was an old problem. Presence detection has been attempted several times using different contact sensing techniques such as capacitor switches, pressure sensors and mechanical switches. However, these approaches were originally problematic because physicians had different preferences for where they held their main tool grips. Using distance as a criterion of existence allows this kind of presence detection to adjust the main tool grip by breaking the physical contact momentarily after the physician touches the main tool grip lightly, It is advantageous in that it does not constrain to the finger 0 of the finger.

Surgical instrument control through hand tracking

Non mechanically grounded non-powered primary finger track grips 270, sometimes referred to as primary finger tracking grips 270, are illustrated in Figures 2a-2d in different configurations and are described more fully below. The main finger track grips 270 include finger-mounted sensors 211 and 212, sometimes referred to as finger and thumb-mounted sensors 211 and 212, which independently measure the distance between the tip of the forefinger 292B and the thumb 292A, (Position and orientation in one example) at the end of the doctor's hand, i.e., the position of the two fingers of the doctor's hand. Thus, the position of the hand itself is tracked, as opposed to tracking the position of the main tool grip in a known minimally invasive surgical system.

In one embodiment, the sensor provides tracking of six degrees of freedom (three translations, three turns) for each finger of the hand equipped with the sensor. In another aspect, the sensor provides tracking of 5 degrees of freedom (3 translations, 2 rotations) for each finger of the hand equipped with the sensor.

In another embodiment, the sensor provides tracking of three degrees of freedom for each finger of the hand equipped with the sensor (translation of 3). When two fingers are tracked at three degrees of freedom each, it is sufficient to control a slave surgical instrument that does not include a total of six translational freehand wrist mechanisms.

A pad-shaped foam connector 210 is connected between the finger and the thumb-mounted sensors 211, 212. The connector 210 restrains the thumb 292A and the forefinger 292B so that the fingers of the hand 291R are within a fixed distance and the hand 291R with the main finger tracking grip 270 There is a maximum separation distance between the fingers. As the thumb 292A and forefinger 292B move from the maximum separation (FIG. 2A) to the fully closed configuration (FIG. 2D), the pod 181 provides a clear feedback, RTI ID = 0.0 > 170, < / RTI >

The gripping force is minimal at the position illustrated in Figure 2A where the thumb 292A and the forefinger 292B are separated by a maximum distance allowed by the main finger track grip 270. [ 2D in which the thumb 292A and the index finger 292B are separated by a minimum distance that is allowable by the connector 210, i.e., by the main finger tracking grip 270, The gripping force is maximum. 2B and 2C show positions mapped according to the intermediate holding force.

As described more fully below, the position (orientation and orientation) of the thumb 292A and the forefinger 292B in FIGS. 2A-2D is dependent on the grip closure variable, such as remote control dependent And is mapped according to the normal grip closing value used to control the grip of the surgical instrument. Specifically, the sensed position of the thumb 292A and forefinger 292B is mapped by the hand tracking controller 130 according to the grip closure variable.

Thus, the position of a portion of the hand of the doctor 181 is tracked. Based on the tracked position, the system control variable, i.e., the grip closure parameter, of the minimally invasive surgical system 100 is generated by the hand tracking controller 130 and sent to the system controller 140. The system controller 140 generates a system command using the grip closure variable, which is transmitted to the remote control dependent surgical apparatus. The system command instructs the remote control surgical instrument to configure the end effector with a grip closure corresponding to the grip closure parameter. Thereby, the minimally invasive surgical system 100 can control the operation of the remote control dependent surgical instrument of the minimally invasive surgical system 100 using the grip closure parameter.

2a-2d, the position (orientation and orientation) of the thumb 292A and forefinger 292B are mapped by the hand tracking controller 130 according to the control point position and the control point orientation. The control point location and control point orientation are mapped within the eye coordinate system of the physician 181 and then provided to the system controller 140 via command signals. The control point position and control point orientation in the eye coordinate system are used by the system controller 140 for remote manipulation of the dependent surgical instrument connected to the primary finger tracking grip 170. [

Again, the position of a portion of the hand of the physician 181 is tracked. Based on the tracked position, another system control variable of the minimally invasive surgical system 100, i.e., the control point position and orientation, is generated by the hand tracking controller 130. The hand tracking controller 130 transmits a command signal to the system controller 140 together with the control point position and orientation. The system controller 140 generates a system command using the control point location and orientation, which is transmitted to the remote control dependent surgical apparatus. The system instructions instruct remote manipulation instruments to position remote manipulation instruments based on control point location and orientation. Thereby, the minimally invasive surgical system 100 can control the operation of the remote control dependent surgical instrument of the minimally invasive surgical system 100 using the control point position and orientation.

In addition to determining the grip closure based on the position of the sensors 211 and 212, other relative movements between the index finger 292B and the thumb 292A may be used to control the yaw and roll motion of the dependent surgical instrument have. Crossing together the forefinger 292B and the bite finger 292A together as rotating the spindle, as indicated by the arrows in the vicinity of the imaginary spindle 293 (Fig. 2E), causes roll motion of the distal end of the surgical instrument , Sliding the forefinger and thumb back and forth along their lengths, as indicated by the arrows along the axis (Figure 2f) in the pointing direction indicated by arrow 295, causes the yaw action along the x- . This is accomplished by mapping the vector between the position of the forefinger and the position of the thumb tip to define the x-axis of the control point orientation. The position of the control point remains relatively fixed since the finger and thumb respectively slide in a symmetrical manner along the axis 295. Even if the motion of the finger and the thumb is not a perfectly symmetrical motion, the position is still kept sufficiently fixed, and the user can easily correct any small variations that may occur.

Again, the position of a portion of the hand of the physician 181 is tracked. Based on the tracked position, the relative motion between two fingers of another system control variable, i.e., the physician's hand 291R, is generated by the hand tracking controller 130. [

The hand tracking controller 130 converts the relative motion to the orientation of the remote control dependent surgical instrument connected to the main finger track grip 170. [ The hand tracking controller 130 transmits a command signal to the system controller 140 together with this orientation. If this orientation is an absolute orientation mapping, then in one aspect the system controller 140 uses this input with the ratchet method during remote steering in the same manner as the orientation input from any other passive unipolar main tool grip. An example of a ratchet system is described in co-assigned US patent application Ser. No. 12 / 495,213, filed June 30, 2009, entitled "ratchet system for main alignment of telescoping surgical instruments" Are incorporated herein by reference.

The system controller 140 uses this orientation to generate system commands, which are transmitted to a remote control dependent surgical apparatus. The system command directs the remote control surgical instrument to rotate the remote manipulation surgical instrument based on the orientation. Thereby, the minimally invasive surgical system 100 can control the operation of the remote control dependent surgical instrument of the minimally invasive surgical system 100 using the operation between the two fingers.

When the operation is the first operation, for example when rubbing across the thumb 292A and the finger 292B, such as rotating the spindle, the orientation is in the roll direction and the system command is dependent on its pointing direction Surgical instrument brings the end curl of the wrist. When the operation is a second operation different from the first operation, for example when the forefinger and the thumb slid back and forth longitudinally along each other (Fig. 2f), the orientation is in the yaw direction, Brings the yaw action of the wrist.

In another aspect, when the physician switches the system operating mode to the gesture recognition mode, both hands are tracked, and the control points and orientations for both hands, in one embodiment, are located at the sensed positions and orientations of sensors mounted on both hands . For example, as illustrated in Figure 2G, the thumb of each hand and the tip of the forefinger abut against each other to form a circular shape. The sensed position of each hand is mapped by the hand tracking controller 130 according to the position of one phase control point. The pair of control points is transmitted to the system controller 140 together with the camera control system event.

Thus, in this embodiment, the position of a portion of each hand of the doctor 181 is tracked. Another system control variable of the minimally invasive surgical system 100 based on the tracked position, i.e., the position of the control point of one phase, is generated by the hand tracking controller 130. The hand tracking controller 130 transmits the pair of control point positions to the system controller 140 together with the camera control system event.

In response to the camera control system event, the system controller 140 generates a camera control system command based on the pair of control point locations. The camera control system command is transmitted to the remote-controlled endoscopic camera controller of the minimally invasive surgical system 100. Thereby, the minimally invasive surgical system 100 can control the operation of the remote-controlled endoscopic camera actuator of the minimally invasive surgical system 100 using a pair of control point positions.

System control through hand gesture pose and hand gesture trajectory

In this aspect, after being placed in the gesture detection mode of operation, the hand tracking controller 130 detects a hand gesture pose or a hand gesture pose and a hand gesture trajectory. Controller 130 maps hand gesture moods according to certain system mode control commands and similarly controller 130 maps hand gesture trajectories according to other system mode control commands. Note that the mapping of the pose and locus is independent, and thus this is different from, for example, manual signal language tracking. The ability to generate system commands and control system 100 using hand gesture pose and hand gesture trajectories instead of manipulating switches, multiple foot pedals, etc., as in the known minimally invasive surgical system, Provide a use to the doctor.

When the doctor is standing, the use of the hand gesture pose and hand gesture trajectory in the control system (100) requires the physician to detach his / her eye from the patient and / or watch screen to find the foot pedal or switch . Finally, the elimination of the various switches and foot pedals reduces the floor space required for a minimally invasive telescoping surgical system.

The hand gesture pose and the specific set of hand gesture trajectories used to control the minimally invasive surgical system 100 are not significant as long as each hand gesture pose and each hand gesture trajectory is clear. Specifically, a hand gesture pose should not be interpreted by the hand tracking controller 130 as one or more other hand gesture pauses in one pose, and one hand gesture trajectory may be interpreted as one or more hand gesture trajectories in one trajectory It should not be interpreted. Thus, the hand gesture pose and hand gesture trajectory discussed below are exemplary only, and are not intended to be limiting.

Figures 3a-3d are examples of hand gesture pose 300A-300D, respectively. Figures 4A-4C are examples of hand gesture trajectories. For example, although the configuration of FIG. 2A seems similar to FIG. 3A, it is noted that the operation of the minimally invasive surgical system 100 is different when the two configurations are used.

In Figure 2a, the remote manipulation minimally invasive suture instrument is connected to the main finger track grip 170 and the system 100 is in follow-up mode, whereby the movement of the remote manipulation minimally invasive suture instrument is tracked Follow the movement. In Figures 3A-3D and 4A-4C, the physician places the system 100 in a gesture recognition mode and produces one of the illustrated hand gesture pose or hand gesture trajectories. The hand gesture pose and hand gesture trajectories are used for control of the system mode and not for the follow-up operation mode. For example, the system mode controlled by the hand gesture pose is to enable, disable, circulate, visualize, clutch, visualize, and / or clear the visual display.

In the hand gesture pose 300A (Fig. 3A), the thumb 292A and the forefinger 292 are separated beyond the main clutch threshold, for example, the distance between two fingers of the hand 291R is 115 mm or more. The hand gesture pose 300B (FIG. 3B) with the forefinger 292B stretched and the thumb 292A curved can be used to signal the hand tracking controller 130 to allow the doctor to track the hand gesture trajectory 4a and 4b). The user interface can be turned on using the hand gesture pose 300C (Fig. 3C) with the thumb 292A pointed up and the forefinger 292B bent, and the user interface can cycle between modes. The user interface can be turned off using the hand gesture pose 300D (FIG. 3D) in which the thumb 292A faces down and the forefinger 292B is bent. Other hand gesture pose may include "A-okay" hand gesture pose, L-shaped hand gesture pose, and the like.

In one aspect, the hand tracking controller 130 recognizes and identifies a hand gesture pose using a multi-directional feature vector. Initially, multiple hand gesture poses are specified. Next, a feature set including a plurality of features is specified. The feature set is designed to uniquely identify each hand gesture pose in a plurality of poses.

The hand gesture pose recognition process is trained using the training database. The training database includes a plurality of examples for each of the hand gesture poses. Multiple examples include feature vectors for poses made by several different people. A feature set is generated for each instance in the training database. These feature sets are used to train the multi-directional Bayesian classifier described more fully below.

When the doctor 180 is about to enter the hand gesture operating mode, the doctor activates the switch, for example by pressing the foot pedal and then making at least one hand a hand gesture pose. Note that one foot pedal is required in this example, which allows removal of the remaining foot pedal from the foot tray of a known minimally invasive surgical system, and thus has the advantages described above. The hand tracking unit 186 sends a signal to the hand tracking controller 130 indicating the detected position and orientation of the thumb and forefinger of the doctor's hand or hands.

The hand tracking controller 130 generates the observed feature set using the tracking data for the fingers of the doctor's hand. Next, the hand tracking controller 130 determines the likelihood, that is, the probability that the feature set observed using the trained multi-directional Bayesian classifier and Mahalanovis distance is a feature set of the hand gesture pose in the plurality of poses. This is done for each hand gesture pose in multiple poses.

The hand gesture pose among the plurality of poses selected by the hand tracking controller 130 as the observed hand gesture pose is the minimum Mahalanobis distance when the Mahalanobis distance is less than the maximum Mahalanobis distance of the training database for the corresponding hand gesture pose. Distance. The selected hand gesture pose is mapped according to the system event. The hand tracking controller 130 provides system events to the system controller 140.

The system controller 140 processes system events and issues system commands. For example, if a hand gesture pose 300C (FIG. 3C) is detected, the system controller 140 sends a system command to the display controller 150 to turn on the user interface. In response, the display controller 150 executes at least a portion of the user interface module 155 on the processor 151 to create a user interface on the display of the pseudo console 185.

Thus, in this embodiment, the minimally invasive surgical system 100 tracks the position of a portion of a human hand. Based on the tracked position, a system control variable is created, for example a hand gesture pose is selected. The hand gesture pose may be used to control the user interface of the minimally invasive surgery system 100, for example, to display a user interface on the display of the pseudo console 185. [

The user interface control is not intended to be limiting, merely by way of example. Using a hand gesture, changes in the mode of the known minimally invasive surgical system, such as main clutch, camera control, camera focus, manipulator arm swapping, etc., can be performed.

If the hand gesture pose recognition process determines that the observed hand gesture pose is a hand gesture pose to the hand gesture trajectory, no system event is provided by the hand tracking controller 130 based on the pose recognition. Instead, a hand gesture locus recognition process is started.

In this example, the hand gesture pose 300B (Fig. 3B) is the pose used to create the hand gesture trajectory. Figures 4A and 4B are two-dimensional examples of hand gesture traces 400a and 400b made using the hand gesture pose 300B. Figure 4c presents other two-dimensional examples of hand gesture trajectories that may be used.

In one embodiment, the hand gesture trajectory recognition process uses a hidden Markov model Λ. A training database is needed to generate the probability distribution of Hidden Markov Model Λ. A set of hand gesture trajectories is specified prior to obtaining the training database. In one aspect, the sixteen hand gesture traces of Figure 4c are selected.

In one embodiment, multiple test objects are selected to create each hand gesture trajectory. In one example, each test subject performed each trajectory a predetermined number of times. For each trajectory performed, the position and orientation data for each subject was stored in the training database. In one aspect, a training database was used to train separate left and right hiccup Markov models using the iterative Baum-Welch method, as described more fully below.

When the physician makes a trajectory, the data is switched to the observation sequence O by the hand tracking controller 130. Using the observation sequence O and the hidden Markov model Λ, the hand tracking controller 130 determines if the hand gesture trajectory corresponds to the observed symbol order. In one aspect, the hand tracking controller 130 generates an overall probability of the observed symbol sequence using an iterative recursive algorithm with the Hidden Markov model A. If the probability exceeds the threshold probability, the hand gesture trajectory with the highest probability is selected. If the highest probability is less than the threshold probability, the hand gesture trajectory is not selected and the processing ends.

Selected hand gesture trajectories are mapped according to system events. The hand tracking controller 130 provides a system event to the system controller 140.

The system controller 140 processes system events and issues system commands. For example, if the selected hand gesture trajectory is mapped according to an event to change the illumination level for the surgical site, the system controller 140 sends a system event to the controller of the lighting device to change the illumination level.

Presence detection through hand tracking

In one aspect, as indicated above, the position of the physician's hands 291R, 291L (Figure 6a) is tracked to determine whether remote manipulation of the minimally invasive surgical system 100 is allowed, and in some embodiments, It can be determined whether or not it can be displayed. The hand tracking controller 130 tracks at least a portion of the hands of the physician 180B (Fig. 6A). The hand tracking controller 130 includes a main tool grip, e.g., main tool grips 621L and 621R 6b) indicating the positions of the main tool grips 62a, 6a, and the position of a part of the hand. The hand tracking controller 130 maps these two locations into a common coordinate frame and then determines the distance between the two locations within the common coordinate frame. This distance is a system control variable of a minimally invasive surgical system based on the tracked position of a portion of a physician's hand.

If this distance is less than the safety threshold, that is, less than the maximum allowable separation between a portion of the hand and the main tool grip, remote manipulation of the minimally invasive surgical system 100 is allowed, otherwise remote manipulation is prohibited. Similarly, in embodiments where presence detection is used to control the display of the user interface, if this distance is less than the safety threshold, i. E. Less than the maximum allowable separation between a portion of the hand and the main tool grip, Display of the user interface on the display is prohibited, and otherwise, display of the user interface is permitted.

Thus, this distance is used to control the remote manipulation of the minimally invasive surgical system 100. Specifically, the hand tracking controller 130 sends the system event to the system controller 140, which indicates whether remote control is allowed. In response to a system event, the system controller 140 configures the system 100 to allow or prohibit remote control.

Hand Positioning Technology

Before considering the various aspects of the hand tracking described in more detail above, an example of tracking technology is described. This example is merely an example, and any tracking technique that provides the necessary hand or finger location information in light of the following description may be used.

In one aspect, pulsed DC electromagnetic tracking is used with two fingers of the hand, such as the thumb and forefinger, mounted on a forefinger, as illustrated in Figures 2a-2d and 7. Each sensor measures 6 degrees of freedom and, in one embodiment, has a size of 8 mm x 2 mm x 1.5 mm. The tracking system has a 0.8m hemispherical manual workspace and 0.5mm and 0.1 degree position sensing resolution. The update rate is 160 Hertz and has a 4 ms detection delay. When integrated into the system, additional delays can occur due to communication and additional filtering. A valid command delay of less than 30 ms has been found to be acceptable.

In this aspect, the tracking system includes an electromagnetic hand tracking controller, a sensor used in the main finger track grip, and a hand tracking transmitter. A tracking system suitable for use in one embodiment of the present invention is a 3D guidance trakSTAR system with a mid-range transmitter available from Ascension Technology Corporation of Burlington, Va. (TrakSTAR (TM) is a trade name of Ascension Technology Corporation) . This transmitter produces a pulsed DC magnetic field for high accuracy tracking over medium ranges, which is specified at 78 cm (31 inches). The system provides a dynamic tracking of 240 to 420 updates per second to each sensor. The output of the miniaturized passive sensor is not affected by the power line noise source. Clearly connected transmit and receive lines between transmitter and sensor are not required. All postures are tracked and there is no movement or optical interference due to inertia. It has high metal immunity and no distortion due to non-magnetic metal.

An electromagnetic tracking system with a finger cover is used herein, but this is merely an example, and is not intended to be limiting. For example, a pen-shaped device may be retained by a physician. A pen-shaped device is a piece of fingers having three or more non-collinear fiducial markers on the outer surface of the device. Typically, more starting point markers are used due to self-blocking to make at least three starting point markers visible at any point in time. A fiducial marker is sufficient to determine the degree of freedom of 6 degrees of freedom (3 translation and 3 rotation) of the finger piece, and is therefore sufficient for the action of a hand with a pen-shaped device. In addition, the pen-shaped device senses the grip in one embodiment.

The pen-shaped device is viewed by two or more cameras of the known parameters, whereby the origin markers can be localized in three dimensions to infer the three-dimensional pose of the finger piece. The origin markers may include, for example: 1) a retro-reflective sphere that is illuminated close to the camera; 2) a concave or convex hemisphere that is illuminated close to the camera; Or 3) an active marker such as a (blinking) LED. In one aspect, near infrared illumination of a finger piece is used, and the filter can be used to block visible spectrum to the camera to minimize scattering from the background clutter.

5) or a bare hand 502 is used and the origin markers 511 are attached to the thumb and forefinger of the glove 501 being worn by the doctor (and / or the glove < RTI ID = 0.0 > (E.g., to other fingers), and / or is attached directly to the skin of the hand 502. Again, multiple markers can be used to accommodate self-blocking. The origin markers can also be located in other fingers, thereby enabling more user interface features through a specifically defined hand gesture.

The three-dimensional position of the origin marker is estimated by the triangular space of a plurality of cameras having a common view. The three-dimensional pose (displacement and orientation) and grip size of the hand can be inferred using the three-dimensional position of the origin marker.

Marker position should be checked before use. For example, a doctor can show a camera a hand with a marker of a different pose. Next, different poses are used for the test.

In another embodiment, hand tracking with fewer markers is used. Articulated hand movements can be tracked using images viewed from one or more cameras, and computerized software can be executed to process these images. Running computer software does not have to track all degrees of freedom of a useful hand. The running software only needs to track the parts associated with the two fingers of the hand, which are useful for controlling the surgical tool as evidenced herein.

In camera-based tracking, the accuracy of the measurements depends on the local accuracy of the marker in the image; 3 - Dimensional Reconstruction Accuracy Based on Camera Geometry; And the minimum number of origin markers, e.g., three or more, the minimum number of cameras (1 or 2) or more, and time averaging and filtering.

Three-dimensional reconstruction accuracy is highly dependent on the accuracy of the camera test. Some of the origin markers attached to the known location on the pseudo console can be used to determine the external parameters (rotation and displacement) of multiple cameras in relation to the pseudo console. This process can be done automatically. An active origin marker can be used as a black origin marker since it is only illuminated during and during the assay procedure. During the procedure, the black origin markers are turned off to avoid confusion with the origin markers used to localize the physician's hand. Relative external variables can also be estimated by moving the marker within the camera's common field of view.

Other tracking techniques suitable for use include, but are not limited to, inertial tracking, in-depth camera tracking, and fiber warping detection.

As used herein, a sensor element, sometimes referred to as a tracking sensor, may be a sensor for any of the hand tracking techniques described above and may be, for example, a passive electromagnetic sensor, a origin marker, or a sensor for other technologies.

Coordinate system

Before considering the various processes described in more detail above, an example of the pseudo console 185B (Fig. 6A) is considered, and various coordinate systems are defined for use in the following examples. The pseudo console 185B is an example of the pseudo console 185. [ The pseudo console 185B includes a three-dimensional viewer 610, sometimes referred to as a viewer 610, main tool manipulators 620L and 620R with main tool grips 621L and 621R, and a base 630. [ The main tool grip 621 (Fig. 6B) is a more detailed illustration of the main tool grips 621L and 621R.

The main tool grips 621L and 621R of the main tool manipulators 620L and 620R are held by the doctor 180B using the index finger and the thumb so that targeting and gripping include intuitive pointing and pinching operations . A remote manipulation endoscope or the like in the same manner as the known main tool manipulator of the known minimally invasive remote manipulation surgical system using the main tool manipulators 620L and 620R in combination with the main tool grips 621L and 621R Can be controlled. The positional coordinates of the main tool manipulators 620L and 620R and the main tool grips 621L and 621R can also be known from the kinematics used to control the subsidiary surgical instrument.

In the normal screen operation mode, the viewer 610 displays a three-dimensional image of the surgical site 103 from the stereoscopic endoscope 112. The viewer 610 is positioned near the physician's hand on the console 185B (Fig. 6B), whereby the image of the surgical site seen on the viewer 610 is viewed directly on the surgical site 103, As shown in FIG. In the image, the surgical instrument appears to be positioned substantially where the physician's hand is located and is oriented substantially as expected by physician 180B based on the position of his or her hand. However, the doctor 180B can not see the position or orientation of his or her hand or the main tool grips 621L, 621R while viewing the displayed image of the surgical site in the viewer 610. [

The main tool manipulators 620L and 620R are moved directly to the front of the doctor 180B and below the viewer 610 and are positioned above the base 630 and are no longer located below the viewer 610, That is, the main tool manipulator is out of the way of the hand gesture. This provides an unobstructed space below the viewer 610 where the physician 180B can make a hand gesture that is one or both of a hand gesture pose or hand gesture trajectory.

6A, three coordinate systems, i.e., screen coordinate system 660, global coordinate system 670, and tracer coordinate system 650 are defined with respect to pseudo console 185B. An equivalent coordinate system is also limited to the physician 181 (Fig. 1), so that the more fully described mapping below can be done for tracking data from the main finger track grip 170 or the main tool grips 621L, 621R I know. See, for example, U.S. Patent Application No. 12 / 617,937, filed November 13, 2009, entitled " Patient Surgical Interface for Minimally Invasive Remote Surgical Instruments ", which is incorporated herein by reference.

In the screen coordinate system 660, the physician 180B looks down at the Z-axis Zview. The Y-axis Yview faces up on the display. The X-axis Xview faces left in the display. In the global coordinate system 670, the Z-axis Zworld is a vertical axis. The global X-axis Xworld and the global Y-axis Yworld are in a plane perpendicular to the Z-axis Zworld.

6B illustrates the main tool grip 621 and the main tool manipulator 620 in more detail. The coordinate systems 680, 690 are discussed more fully below in connection with the method 1100 of FIG.

The process of controlling the surgical instrument through hand tracking

Figure 7 shows the sensor 212 mounted on the forefinger 292B at location 713 in the tracking coordinate system 750 and the sensor 212 mounted on the thumb 292A at location 711 in the tracking coordinate system 750. [ Sensor 211 is illustrated. Sensors 211 and 212 are part of the electromagnetic tracking system described above. The thumb 292A and the forefinger 292B are examples of the fingers of the right hand 291R. As previously noted, a portion of a human hand includes at least one finger of the hand. As known to those skilled in the art, the fingers of the hand, sometimes referred to as the phalanges, are the thumb (first finger), the forefinger (second finger), the stop (third finger), the asymptomatic (fourth finger), and the young finger (fifth finger).

Here, the thumb and forefinger are used as examples of two fingers of a human hand. This is for illustrative purposes only and is not intended to be limiting. For example, the thumb and the stop may be used instead of the thumb and forefinger. The description herein can also be applied directly to the use of pauses. The use of the right hand is also only an example. If a similar sensor is worn on the left hand, the description herein can also be applied directly on the left hand.

The cables 741 and 742 connect the hand-tracking controller 130 to the sensors 211 and 212 of the main finger track grip 270. In one aspect, cables 741 and 742 send position and orientation information from sensors 211 and 212 to hand tracking controller 130.

The use of a cable to transmit sensed position and orientation data to the hand tracking controller 130 is merely exemplary and is not limited to this particular embodiment. In view of the teachings herein, those skilled in the art can select a mechanism for transmitting sensed position and orientation data from the main finger track grip or main finger track grips to the hand tracking controller 130 (e.g., ).

The cables 741 and 742 do not limit the operation of the main finger track grip 270. [ Since the main finger track grips 270 are not mechanically grounded, each main finger track grip does not effectively constrain both position and orientation motion within the workspace of the hand-tracking transmitter and the working space that the physician can reach (e.g., For example, in the Cartesian coordinate system, left, right, up and down, inside and out, roll, pitch, and yaw).

In one aspect, as described above, each sensor 211, 212 of the main finger track grip 270 senses a degree of translation of 3 and a degree of rotation of 3, i.e., 6 degrees of freedom. Thus, the data sensed from the two sensors represents 12 degrees of freedom. In another aspect, each sensor 211, 212 of the main finger track grip 270 senses a translational degree of 3 and a rotational degree of 2 (yaw and pitch), i.e., a degree of freedom of 5. Thus, the data sensed from the two sensors indicates a degree of freedom of ten.

Control of the remote control dependent surgical instrument using control point position and control point orientation based on the tracked position requires 6 degrees of freedom (displacement of 3 and orientation of 3), which is described more fully below. Thus, in embodiments in which each sensor has a degree of freedom of 5 or 6, the sensors 211, 212 provide an extra degree of freedom. As described above and more fully below, the extra degrees of freedom are mapped according to the parameters used to control other remote control dependent surgical instrument embodiments other than position and orientation.

In yet another additional aspect, each sensor 211, 212 senses only the translational degrees of freedom of 3, and thus displays a total of 6 degrees of freedom. This is sufficient to control three degrees of freedom of translation, roll and grip closure of the dependent surgical instrument not including the wrist mechanism. By using the following description, the control point position can be generated by using the degree of freedom of 6. The control point orientation is taken as the orientation of the dependent surgical instrument. The grip closure variable is determined as described below using the control point position and the control point orientation. The roll is determined as described above using the relative motion of the thumb and forefinger.

In an aspect in which the sensor senses a degree of freedom of 6 or the sensor senses a degree of freedom of 5, the forefinger sensor 212 generates a signal indicative of the index finger position Pindex and the index finger Rindex in the tracking coordinate frame 750 do. The thumb sensor 211 generates a signal indicative of the thumb position Pthumb and the thumb orientation Rthumb in the tracking coordinate frame 750. In one aspect, the positions Pindex and Pthumb are taken with the center of the user's fingernail of the forefinger 292B and the center of the user's fingernail, respectively, of the thumb 292A.

In this example, the positions Pindex and Pthumb are represented as 3x1 vectors in the tracking coordinate frame 750, respectively. The positions Pindex and Pthumb are in the tracer coordinates.

The orientations Rindex and Rthumb are displayed as a 3x3 matrix in the tracking coordinate frame 750, respectively,

The control point position Pcp is centered between the forefinger 292B and the thumb 292A. The control point position Pcp is within the control point frame 790, but is specified within the tracer coordinates. Which is the Z-axis of the control point frame 790, which is described more fully below.

Also, as described below, the forefinger 292B and the thumb 292A are mapped according to the grip portion of the dependent surgical instrument, but the two fingers are more delicate than the grip portion of the instrument. The Y-axis of the control point frame 790 corresponds to the pin used for closing the tool clamp. Thus, as described below, the Y-axis of control point frame 790 is perpendicular to the vector between forefinger 292B and thumb 292A.

The control point position Pcp is represented by a 3x1 vector within the tracer coordinates of the tracking coordinate frame 750. [ The control point orientation Rcp is represented by a 3x3 matrix within the tracer coordinates, i.e.:

8 is a process flow chart for mapping the position of a portion of a hand according to the grip closing parameters used to control the grip of one of the dependent surgical instruments, e.g., one of the remote control dependent surgical instruments of FIG. This mapping process also maps the temporal position changes according to the new grip closure variable and the corresponding position of the slave tip end and the speed at which it moves according to its position.

Initially, when the process 800 is entered, the hand position data reception procedure 810 receives the index finger position and orientation (Pindex, Rindex) and the thumb position and orientation (Pthumb, Rthumb) (811). The index finger position and orientation (Pindex, Rindex) and the thumb position and orientation (Pthumb, Rthumb) are based on data from the tracking system. The process 810 proceeds to the position data mapping process 820 according to the control point and the grip parameters .

The position data mapping process 820 according to the control point and the grip parameters includes the control point position Pcp, the control point orientation Rcp, and the control point position Pc using the index finger position and orientation (Pindex, Rindex) and the thumb position and orientation (Pthumb, Rthumb) Creates a grip closure variable ggrip. The control point position Pcp, the control point orientation Rcp, and the grip close variable ggrip are stored in the data 821.

In one aspect, the control point mapping performed in step 820 is defined to emulate important characteristics of the known main tool actuator control point placement. Thus, the response to thumb and forefinger actions will be familiar and intuitive to a user of a known remote controlled minimally invasive surgical system with a pseudo console similar to the pseudo console 180B (Fig. 6A).

9 is a more detailed process flow diagram for one aspect of the position data mapping process 820 according to the control point and grip parameters . First, in step 820 , a hand position data mapping process 910 according to the control point generates a position of the control point position Pcp from the index finger position Pindex and the thumb position Pthumb, i.e., the following:

The control point position Pcp is an average of the finger position Pindex and the thumb position Pthumb, and the hand position data mapping process 910 according to the control point proceeds to the control point orientation generating process 920.

와 엄지손가락 포인팅 방향 벡터

사이의 반 회전으로서 제어 포인트 배향에 대한 Z-축 포인팅 방향 벡터

를 정의할 수 있다. 엄지손가락 배향 Rthumb로부터 엄지손가락 포인팅 방향 벡터

는 다음과 같다:As indicated above, the Z-axis of the control point orientation is aligned in the pointing direction. In the control point orientation generation process (920) of this embodiment, a Rodriguez axis /

And thumb pointing direction vector

Axis as a half rotation between the Z-axis pointing direction vector

Can be defined. Thumb orientation Rthumb to thumb pointing direction vector

Is as follows:

는 다음과 같다:Similarly, from the forefinger orientation Rindex to the forefinger pointing direction vector

Is as follows:

와 엄지손가락 포인팅 방향 벡터

에 수직인 벡터이다. 벡터 ω는 집게손가락 포인팅 방향 벡터

와 엄지손가락 포인팅 방향 벡터

의 곱셈 값으로서, 다음과 같다:Vector ω is the forefinger pointing direction vector

And thumb pointing direction vector

. ≪ / RTI > Vector ω is the forefinger pointing direction vector

And thumb pointing direction vector

As follows: < RTI ID = 0.0 >

와 엄지손가락 포인팅 방향 벡터

사이의 각도 크기이다. 각도 θ는 다음과 같이 정의된다:The angle < RTI ID = 0.0 >#<

And thumb pointing direction vector

Lt; / RTI > The angle θ is defined as:

는 다음과 같다:With axis ω and angle θ, the Z-axis pointing direction vector

Is as follows:

및

를 생성한 다음, 이들을 다시 직교시켜서 제어 포인트 배향 행렬을 생성할 수 있다.Thus, process 910 has generated control point position Pcp, and the first part of process 920 has generated an approximate pointing direction of the Z-axis within control point frame 790. [ And then proceeds to interpolate the forefinger and thumb-orientation vectors to obtain the control point unit vector axis

And

And then orthogonalize them to generate a control point orientation matrix.

이 생성된다:However, by using the tracking mapping, greater remote control steering capability can be achieved from the tracked position of the fingers. This mapping can achieve the effective roll and yaw action of the control point, such as by manipulating the small ball piece between the fingers using the relative position of the forefinger and the thumb. The remainder of the process 920 is performed as follows, and a complete set of orthogonal normal control point unit vector axes

Is generated:

Using these vectors, the control point orientation Rcp is:

Now, by processes 910 and 920, the process 820 has mapped the index finger and thumb positions and orientations Pindex, Rindex, (Pthumb, Rthumb) according to the control point position and orientation (Pcp, Rcp) . Then process 820 should still generate the grip closure variable ggrip. Accordingly, the control point orientation generation process 920 proceeds to the grip close parameter generation process 930.

에 의해 한정된 중심선 축으로부터 집게손가락 위치 Pindex와 엄지손가락 위치 Pthumb의 거리에 따라 그립 닫힘이 결정된다. 이것은 엄지손가락과 집게손가락이 맞닿았을 때 미끄러져 움직이는 것에 대해서 그립 닫힘이 불변인 채로 있을 수 있도록 한다.In step 930, the control point position Pcp and the Z-axis direction

The grip closeness is determined by the distance between the index finger position Pindex and the thumb position Pthumb from the center line axis defined by the grip axis. This allows the grip closure to remain unchanged as the thumb and forefinger slip when in contact.

Thus, the index finger position Pindex and the thumb position Pthumb are mapped on the Z-axis in frame 790. [ Position Pindex_proj is the projection of the index finger position Pindex on the Z-axis of frame 790 and position Pthumb_proj is the projection of the thumb position Pthumb on the Z-axis of frame 790:

The position Pindex_proj and the position Pthumb_proj can be used to evaluate the evaluation grip closing distance dval, i.e.:

Here, the double parallel line is a known expression of the 2-normal Euclidean distance. The evaluation grip closure distance dval is combined by the maximum distance threshold dmax and the minimum distance threshold dmin. As illustrated in Figure 7, the pad shaped foam connector 210 between the sensors 211, 212 is constrained so that the fingers are within a fixed separation, i. E. Between the maximum distance threshold dmax and the minimum distance threshold dmin. In addition, the separation distance when two fingers touch lightly corresponds to the neutral distance.

For a particular set of sensors and connectors, the maximum distance threshold dmax, the minimum distance threshold dmin, and the neutral distance are determined empirically. In one aspect, three different combinations of sensor and connector are provided for the small hand, the average hand, and the large hand. Each combination has its own maximum distance threshold dmax, minimum distance threshold dmin, and neutral distance, and the length of connector 210 in each combination is different.

The process 930 compares the distance dval with the minimum distance threshold dmin. In this comparison, if the distance dval is found to be less than the minimum distance threshold dmin, then the grip closure distance d is set to the minimum threshold distance dmin. Otherwise, process 930 compares the distance dval with the maximum distance threshold dmax. In this comparison, if it is found that the distance dval exceeds the maximum distance threshold value dmax, the grip closing distance d is set to the maximum threshold distance dmax. In other cases, the grip closing distance d is set to the distance dval.

The test performed on the distance dval to determine the grip closure distance d is summarized as follows:

Next, in process 930, a grip close variable ggrip is generated:

Therefore, the grip closing distance d between the maximum distance threshold value dmax and the distance d0 is mapped according to a value between 0 and 1. The grip closure distance d between the minimum distance threshold value dmin and the distance d0 is mapped according to a value between -1 and 0.

A value of 1 is obtained for the grip closing parameter ggrip when the forefinger 292B and the thumb 292A are separated by the maximum extent allowed by the connector 210 (Fig. 2A). A value of zero is obtained for the grip closure variable ggrip when the tip of the forefinger 292B and the tip of the thumb 292A lightly touch (Figure 2c). A value in the range of 0 to 1 controls the opening / closing of the grip portion of the end effector's end effector. A value of -1 is obtained for the grip closing parameter ggrip when the forefinger 292B abuts the thumb 292A and the connector 210 is sufficiently compressed between the index finger 292B and the index finger 292A (Fig. 2d). Values in the range of 0 to -1 control the clamping force of the closed clamping portion of the end actuator. The connector 210 provides a passive tactile clue to the clasp closure.

This example of mapping the grip closure distance d according to a value in one of the two ranges is only an example, and is not intended to be limiting. This example maps the grip closure distance d according to the value in the first range of the grip closure parameter ggrip to control the opening / closing of the gripper portion of the end effector's distal actuator when the grip closure distance d is greater than the neutral distance d0 For example. Here, "open / closed" means opening and closing of the gripper. The grip closure distance d is mapped according to a value in the second range of the grip closure parameter ggrip and the grip force of the closed gripper portion of the end actuator can be controlled when the grip closure distance d is less than the neutral distance d0.

Thus, the process 820 may include the following operations: the index finger position and orientation (Pindex, Rindex) and the thumb position and orientation (Pthumb, Rthumb) according to the control point position and orientation (Pcp, Rcp) and grip close parameter ggrip stored in the data ). The process 820 proceeds to the mapping process 830 (FIG. 8) according to the global coordinates .

을 이용하여 전역 좌표 제어 포인트 위치 및 배향(Pcp_wc, Rcp_wc)에 따라 맵핑되며, 이것은 전역 좌표 시스템(670)의 좌표에 따라 추적자 좌표 시스템 (700)(도 7b)의 좌표를 맵핑하는데, 예를 들어 다음과 같다: The mapping process 830 according to the global coordinates receives the data 821 and maps the data 821 according to the global coordinate system (see the global coordinate system 670 (Fig. 6A)). Specifically, the control point position and orientation (Pcp, Rcp) and the grip closure variable ggrip are converted to 4x4 equalization

(Pcp_wc, Rcp_wc), which maps the coordinates of the tracker coordinate system 700 (Figure 7b) according to the coordinates of the global coordinate system 670, for example, As follows:

here,

는 전역 좌표 wc에서의 배향에 따라 추적자 좌표 tc에서 배향을 맵핑하고,

Maps the orientation at the tracer coordinate tc according to the orientation at the global coordinate wc,

는 전역 좌표 wc에서의 위치에 따라 추적자 좌표 tc에서 위치를 옮긴다. 그립 닫힘 변수 ggrip는 이 맵핑에 의해 변화되지 않는다. 전역 좌표 wc의 데이터는 데이터(831)로서 저장된다. 과정(830)은 눈 좌표에 따른 맵핑 과정(840)으로 진행한다.

Moves the position from the tracer coordinate tc according to the position in the global coordinate wc. The grip closure variable ggrip is not changed by this mapping. The data of the global coordinate wc is stored as the data 831. The process 830 proceeds to a mapping process 840 according to the eye coordinates .

을 이용하여 눈 좌표 제어 포인트 위치 및 배향(Pcp_ec, Rcp_ec)에 따라 맵핑되며, 이것은 눈 좌표 시스템(660)의 좌표에 따라 전역 좌표 시스템(670)(도 6a)의 좌표를 맵핑하는데, 예를 들어 다음과 같다: The mapping process 840 according to the eye coordinates receives the data 831 of the global coordinate wc and maps the data 831 according to the eye coordinate system (see the eye coordinate system 660 (Fig. 6A)). Specifically, the global coordinate control point position and orientation (Pcp_wc, Rcp_wc) and the grip closing variable ggrip are converted to 4x4 equalization

(Pcp_ec, Rcp_ec), which maps the coordinates of the global coordinate system 670 (FIG. 6A) according to the coordinates of the eye coordinate system 660, for example, As follows:

here,

는 눈 좌표 ec에서의 배향에 따라 전역 좌표 wc에서 배향을 맵핑하고,

Maps the orientation in the global coordinate wc according to the orientation at the eye coordinate ec,

는 눈 좌표 ec에서의 위치에 따라 전역 좌표 wc에서 위치를 옮긴다.

Moves the position in the global coordinate wc according to the position at the eye coordinate ec.

Again, the grip closure variable ggrip is not changed by this mapping. The data of the eye coordinates is stored as the data 841. The process 840 proceeds to a rate generation process 850.

을 이용하여 눈 좌표 ec의 데이터에 따라 직접 맵핑되고, 이것은 눈 좌표 시스템(660)의 좌표에 따라 추적자 좌표 시스템(650)(도 6a)의 좌표를 맵핑하는데, 예를 들어 다음과 같다:In process 800, the mapping processes 830 and 840 are described as two different processes to simplify illustration only. In one aspect, the mapping processes 830 and 840 are combined, whereby the control point data of the tracer coordinate tc is transformed to a 4x4 < RTI ID = 0.0 >

, Which maps the coordinates of the tracer coordinate system 650 (Figure 6A) according to the coordinates of the eye coordinate system 660, for example as follows: < RTI ID = 0.0 >

In this aspect, the position Pcp_ec of the control point in eye coordinates is

, And the orientation Rcp-ec of the control point in the eye coordinates is

to be.

In some aspects, the global coordinate mapping can be eliminated. In this case, the control point data is directly mapped from the tracking coordinate system to the eye coordinate system without using the global coordinate system.

Position, orientation and speed are required for remote control. Thus, the rate generation process 850 generates the required rate. Speed can be generated in many ways. Some embodiments, such as inertial and gyroscope sensors, can directly measure the differential signal to create a linear velocity and angular velocity of the control point. If the velocity can not be measured directly, then in one embodiment, process 850 estimates velocity from the position measurements of the eye coordinate system.

The velocity can be estimated using the finite higher points of the eye coordinate system over the sampling interval. For example, the linear velocity Vcp_ec is estimated as follows,

The angular velocity ωcp_ec is estimated as follows:

을 사용해서 추적자 좌표 시스템(750)으로부터 눈 좌표 시스템(660)으로 회전된다. 구체적으로, 상기 정의된 회전 맵핑을 사용하면 다음과 같이 된다: In another aspect of the velocity generation process 850, the control point linear velocity Vcp_tc and the control point angular velocity cp_tc are sensed in the tracer coordinates of the tracer coordinate system 750 (FIG. 7). In this embodiment, the directly sensed control point linear velocity Vcp_tc and the directly sensed control point angular velocity cp_tc are rotated

To the eye coordinate system 660 from the tracer coordinate system 750. [ Specifically, using the rotation mapping defined above:

The speed generation process 850 proceeds to the control command transmission process 860. [ The process 860 sends appropriate system control commands to the dependent surgical instrument based on the position, orientation, velocity, and grip closure variables stored as data 851.

In one aspect, steps 810 through 850 are performed by the hand tracking controller 130 (FIG. 1). The controller 130 executes the finger tracking module 135 on the processor 131 and performs the process 850 in the step 810. [ In this aspect, the finger tracking module 135 is stored in the memory 132. [ The process 850 sends the system event to the system controller 140, which then performs the process 860. [

It should be appreciated that the hand tracking controller 130 and the system controller 140 may actually be implemented by some combination of hardware, software that can be executed on the processor, and firmware. Further, the functions of these controllers may be performed by a single unit, or may be divided into different components, as described herein, and each component may be implemented in hardware, software that may be executed on the processor, Can be carried out by combination. When divided into different components, the components can be concentrated in one place, or distributed throughout the system 100 for distributed processing purposes.

The process of gesture hand pose and gesture trajectory control

10 is a process flow diagram for one embodiment of a hand gesture pose of system 100 and a process 1000 of hand gesture trajectory control. In the above-described embodiment, the hand gesture pose recognition process 1050 uses a multi-directional Bayesian classifier, and the hand gesture locus recognition process 1060 uses a separate hidden Markov model Λ.

As described above, Figures 3a-3d are examples of hand gesture pose. In order to train the hand gesture pose recognition process (1050), several hand gesture pauses are specified. The number of hand gesture pauses used is dependent on the ability to define a unique pose that can be clearly identified by the recognition process 1050 and the ability of the physician to remember and faithfully reproduce each of the different hand gesture pauses .

In addition to defining a hand gesture pose, a feature set comprising a plurality of features fi is defined, where i ranges from 1 to n. The number n is the number of features used. The number and types of features are selected so that each of the hand gesture pose in the allowable pose set can be accurately identified. In one embodiment, the number n is six.

The following is an example of a feature set with n features.

과 엄지손가락(292A)의 포인팅 방향

의 내적이다. 특징 f2는 집게손가락(292B)과 엄지손가락(292A) 사이의 거리이다. 특징 f3은 집게손가락(292B)의 포인팅 방향

상에 돌출된 엄지손가락(292A)의 거리이다. 특징 f4는 집게손가락(292B)의 포인팅 방향

을 따른 축으로부터 엄지손가락(292A)의 거리이다. 특징 f5는 엄지손가락(292A)의 포인팅 방향

의 Z-성분이다. 특징 fn은 엄지손가락(292A)의 엄지손가락 정규 벡터

와 집게손가락(292B)의 포인팅 방향

의 내적이다.The feature f1 is the pointing direction of the forefinger 292B

And the pointing direction of the thumb 292A

The inner product of. Feature f2 is the distance between forefinger 292B and thumb 292A. The feature f3 is the pointing direction of the forefinger 292B

Lt; RTI ID = 0.0 > 292A < / RTI > Feature f4 is the pointing direction of the forefinger 292B

0.0 > 292A < / RTI > Feature f5 indicates the pointing direction of the thumb 292A

≪ / RTI > The feature fn is the thumb normal vector of the thumb (292A)

And the pointing direction of the forefinger 292B

The inner product of.

Before using method 1000, it is necessary to develop a training database of hand gesture pose. Several different users create at least one hand gesture pose and the position and orientation data for each user's hand gesture pose is measured using a tracking system. For example, each person in the group creates an acceptable hand gesture pose. The forefinger and thumb position and orientation (Pindex, Rindex), (Pthumb, Rthumb) are stored in the training database for each person's hand gesture pose in the group.

및 공분산

을 계산할 수 있다.Using the training database, a feature set {fi} is generated for each hand gesture pose for each user. Next, using a set of training feature vectors for each hand gesture pose,

And covariance

Can be calculated.

Thus, the training database can be used to obtain feature vector averages and covariances for each trained gesture. In addition, for each hand gesture pose, a Mahalanobis distance d (fi) (see discussion below) is generated for each trainer and a maximum Mahalanobis distance d (fi) for each hand gesture pose is generated for each hand gesture pose Lt; / RTI >

및 모든 다른 허용가능한 제스처 포즈의 특징 벡터 평균의 마할라노비스 거리를 시험함으로써 달성될 수 있다. 이 시험 거리는 해당 주어진 제스처에 사용된 최대 훈련 거리 역치보다 훨씬 더 커야 한다.In addition, Mahalanobis distance measurements can be used to verify that all trained gestures are sufficiently different and distinct for a given set of features used. This means that the feature vector averages of a given gesture

And by testing the Mahalanobis distance of the feature vector mean of all other acceptable gesture poses. This test distance should be much larger than the maximum training distance threshold used for the given gesture.

As is known to those skilled in the art, the content of the hidden Markov model requires two model variables N, M and three probability measures A, B, [pi]. The hidden Markov model Λ is as follows:

Λ = (A, B, π)

The model variable N is the number of states in the model, and the model variable M is the number of observation symbols per state. The three probability measures are the state transition probability distribution A, the observation gesture probability distribution B, and the initial state distribution pi.

In one aspect of the separate hidden Markov model, the transition probability distribution A is an N X N matrix. The observation gesture probability distribution B is an N X M matrix, and the initial state distribution π is an N X 1 matrix.

Given the observation order O and the Hidden Markov Model Λ, the probability of the observation order O in a given Hidden Markov Model Λ, P (O ||), is evaluated in process 1000, which is explained more fully below.

A training database is required to generate the probability distribution for the hidden Markov model Λ. A set of hand gesture trajectories is specified prior to obtaining the training database.

Several test objects j are selected to make each hand gesture trajectory. In FIG. 4c, 16 hand gesture trajectories are presented in a two-dimensional projected form, but when performing various hand gesture trajectories, the test subjects are not constrained, which may result in some three-dimensional changes. In one embodiment, each target has performed each hand gesture trajectory k times, which generates a training sequence of j * k per hand gesture trajectory.

In one embodiment, separate left and right hysteresis Markov models were used. The hidden Markov model Λ was chosen to maximize locally using the Baum-Welch method, in which probability P (O | Λ) is repeated. For example, Lawrence R. Rabiner, "Selected Applications in Mapping and Velocity Recognition for Hidden Markov Models ", Proceedings of the IEEE, Vol. 77, No. 2, pp. 257-286 (Feb. 1989), which is incorporated herein by reference in its entirety for the purpose of presenting an understanding of the hidden Markov model to those skilled in the art. In one embodiment, the iterative method ceased when the model converged within 0.1% in three consecutive iterations.

The initial state probability π is set so that the model always starts in state 1. Transition probability matrix A was initiated as a random entry, which was stored in descending order by row. To reinforce the lateral structure, all entries in the lower diagonal of the transition probability matrix A are set to zero. Also, by setting the entry to 0, no transition state exceeding 2 is allowed, in this case (i-j)> 2 for all columns i and j. Transition probability matrix A is normalized at the end of a row.

The initialization of the observation probability matrix B distributed the observation orders evenly based on the desired number of states. Thus, each state can initially observe one or more symbols as a probability based on the number of local frequencies. This matrix was also normalized in series. For example, N. Liu, R.I.A. Davis, B.C. Lovell, P.J. See Kootsookos, " The Effect of Initial HMM Selection in Multi-Order Training for Gesture Recognition ", International Conference on Information Technology, 5-7, Las Vegas, pgs 608-613 (April 2004), which teaches a hidden Markov model As a reference for the present invention. Hidden Markov models were developed for each hand gesture trajectory.

Returning to method 1000, the gesture mode enable check procedure 1001 determines whether the physician has activated the gesture recognition mode of operation of the system 100. In one aspect, to activate the gesture recognition mode, the physician presses the foot pedal of the pseudo console 185 (Fig. 1A). When the gesture recognition mode, the operation check process 1001 proceeds to hand position data receiving process 1010, and if not processing returns back through the sense (1002).

The hand position data reception procedure 1010 receives the index finger position and orientation (Pindex, Rindex) and the thumb position and orientation (Pthumb, Rthumb) for the gesture made by the doctor. As noted above, the index finger position and orientation (Pindex, Rindex) and thumb position and orientation (Pthumb, Rthumb) are based on data from the tracking system. The process 1010 proceeds to a feature generation process 1011. [

In the feature generation process 1011, each feature f1_o to fn_o is generated from the observed feature vector fi_o using the index finger position and orientation (Pindex, Rindex) and the thumb position and orientation (Pthumb, Rthumb). The feature generation process 1011 advances the feature to a known pose and a comparing process 1012.

A feature comparison process 1012 compares the observed feature vector fi_o with the trained feature set {fi} for each pose. This process determines the likelihood that the observed feature vector will be included in the training data set feature set {fi} in the case of a particular hand gesture pose, i.e., the likelihood that it corresponds to the training data set. This can be expressed as:

Here, the training data set feature set {fi} is based on the object class?.

은 다음과 같다:In this example,

Is as follows:

Here, N is the number of dimensions of the feature vector, for example n in the above example.

The statistic used to specify this probability is the Mahalanobis distance d (fi_o), which is defined as:

이다. 마할라노비스 거리는 당업자에게 알려져 있다. 예를 들어, Moghadam, Baback and Pentland, Alex, "확률적 비주얼 러닝 또는 오브젝트 표상", IEEE Transactions On Pattern Analysis and Machine Intelligence, Vol. 19, No. 7, pp. 696 to 710 (July 1997)을 참조하며, 이것은 본원에 참고자료로 포함된다.here

to be. Mahalanovis Street is known to those skilled in the art. For example, Moghadam, Baback and Pentland, Alex, "Stochastic Visual Running or Object Representation ", IEEE Transactions On Pattern Analysis and Machine Intelligence, Vol. 19, No. 7, pp. 696 to 710 (July 1997), which is incorporated herein by reference.

의 고유벡터 Φ와 고유값 Λ을 사용하면,

이 대각선화된 형태에 사용되고, 이때 마할라노비스 거리 d(fi_o)는 다음과 같다:Covariance

≪ / RTI > and the eigenvalue < RTI ID = 0.0 &

Is used for this diagonalized form, where Mahalanobis distance d (fi_o) is:

이다. 대각선화된 형태는 마할라노비스 거리 d(fi_o)가 합계의 항으로 표시될 수 있도록 한다:here

to be. The diagonalized form allows the Mahalanobis distance d (fi_o) to be expressed in terms of the sum:

In this example, this is an expression that is evaluated to determine Mahalanobis distance d (fi_o). Thus, the process 1011 generates a Mahalanobis distance d (fi_o). The completion process 1012 proceeds to the pose selection process 1013. [

In the pose selection process 1013, if the Mahalanobis distance d (fi_o) is less than the maximum Mahalanobis distance in the training database for the hand gesture pose, a hand gesture pose having a minimum Mahalanobis distance d (fi_o) is selected do. If the Mahalanobis distance d (fi_o) exceeds the maximum Mahalanobis distance in the training database for that hand gesture pose, the hand gesture pose is not selected. The pose selection process 1012 proceeds to a time filter process 1014.

The temporal filter process 1014 determines whether the result of the process 1013 has provided the same result as the determined number of times. If the process 1013 has provided the same result for a predetermined number of times, the temporal filter process 1014 proceeds to the gesture pose check process 1015, and if not, returns. The predetermined number of times is selected to prevent vibration or temporal detection when the time filter process 1014 switches hand gesture pose.

The gesture pose check procedure 1015 determines whether the selected hand gesture pose is a hand gesture pose used in the hand gesture trajectory. If the selected hand gesture pose is a hand gesture pose used in the hand gesture trajectory, the gesture pose check process 1015 proceeds to the speed sequence generation process 1020, and if not, the process proceeds to the pose change checking process 1016.

The pose change check process 1016 determines whether the hand gesture pose has last and passed through the method 1000. If the selected hand gesture pose is the same as the immediately preceding time filtered hand gesture pose result, the pose change checking process 1016 returns via the return 1003, otherwise proceeds to the mapping process 1030 according to the system event do.

Depending on the system event, the mapping process 1030 maps the selected hand gesture pose according to the system event, for example, the system event assigned to the hand gesture pose is searched. When the system event is found, the mapping process 1030 proceeds to the system event providing process 1031 according to the system event .

In one aspect, the system event providing process 1031 sends a system event to the event handling device of the system controller 140 (FIG. 1). In response to a system event, the system controller 140 sends appropriate system commands to the controller and / or other devices of the system 100. For example, if a hand gesture pose is assigned to a user interface turn-on event, the system controller 140 sends an instruction to the display controller 150 to turn on the user interface. The display controller 150 executes the portion of the user interface module 155 required to turn on the user interface on the processor 151. [

When the hand gesture pose is a hand gesture pose used to create the trajectory, the method 1000 proceeds to a speed sequence generation process 1020 in a gesture pose check procedure 1015. In one aspect, the main feature used for hand gesture trajectory recognition is a unit velocity vector. The unit velocity vector is invariant depending on the starting position of the gesture. In addition, the normal velocity vector describes the variation of the magnitude and velocity of the gesture. Thus, in step 1020, the control point samples are converted in order of the normal control point velocity, i.e., in the order of the unit velocity vectors:

Upon completion of the speed sequence generation process 1020, the process 1020 advances to a process 1021 for converting the speed sequence to the symbol sequence . As noted above, the separate hidden Markov model A requires an ordered sequence of symbols as input. In step 1021, the separated symbols are generated from the normal control point rate sequence via vector quantization.

In one aspect, vector quantization was performed using modified K-means clustering, where the condition was to stop the process when the clustering assigned a pause change. Although K-means clustering is used, it is important that this process is a unit vector. In this case, vectors with similar directions are recruited. This is done using an inner product between each unit feature vector and a regular cluster center vector as a similarity metric.

Clustering is initiated with random assignment of the vector up to 32 clusters, the entire process is repeated several times, and the best clustering results are selected based on the maximum overall "internal" cluster cost metric. It is noted that in this case, the "internal" cluster cost is based on the criteria of similarity. Each cluster of results is assigned a unique index, which is used as a symbol of the hidden Markov model. Next, the input vector is mapped according to its closest cluster mean, and the corresponding exponent of the cluster is used as a symbol. In this way, the order of the unit velocity vectors can be translated in the order of exponentiation or sign.

In one version, the clustered vectors were assigned symbols based on a fixed 8-way two-dimensional vector quantization codebook. Thus, the process 1020 generates the observed symbol sequence and proceeds to the gesture probability generation process 1023.

In one aspect, to determine if a gesture corresponds to an observed symbol order, the gesture probability generation process 1023 uses a forward regression algorithm with a hidden Markov model to find the probability that each gesture matches the observed symbol order. The forward regression algorithm is described in Rainer, "Mapping for Hidden Markov Models and Selected Uses in Velocity Recognition ", which is incorporated herein by reference. Upon completion of the gesture probability generation process 1023, the process proceeds to a trajectory selection process 1024.

In the trajectory selection process 1024, a hand gesture trajectory with a maximum probability is selected from the allowable hysteresis Markov model trajectory gesture models. In addition, this probability should be greater than a given threshold that can be accommodated. If the maximum probability is below the threshold, the hand gesture trajectory is not selected. This threshold should be tuned to maximize recognition accuracy while avoiding false perceptions.

Upon completion, the trajectory selection process 1024 proceeds to the trajectory detection check process 1025. [ If the trajectory selection process 1024 selects the hand gesture trajectory, the trajectory detection check process 1025 proceeds to the mapping process 1030 according to the system event, and if not, the trajectory selection process 1024 returns through the return process 1004.

Depending on the system event, the mapping process 1030 maps the hand gesture trajectory selected according to the system event, e.g., the system event assigned to the hand gesture trajectory is searched. When the system event is found, the mapping process 1030 proceeds to the system event providing process 1031 according to the system event .

In one aspect, the system event providing process 1031 sends a system event to the event handling device of the system controller 140 (FIG. 1). In response to a system event, the system controller 140 sends appropriate system commands to the appropriate controller (s) or devices. For example, when a system event is assigned to an action in the user interface, the system controller 140 sends an instruction to the display controller 150 to take the action in the user interface, for example, Change.

Presence detection process

In another aspect, as described above, at least part of the traced position of the hand of the physician 180B is used to determine whether the hand is present in the main manipulator tool grip 621 (sometimes referred to as the main tool grip 621) You can decide. 11 is a process flow diagram for one aspect of a presence detection process 1100 performed by the hand tracking controller 130 of the system 100 in one aspect. The process 1100 is performed separately for each of the physician's hands in one embodiment.

In the joint angle obtaining process 1110, the joint angle of the main tool manipulator 620 (FIG. 6B) is measured. The joint angle obtaining process 1110 proceeds to the forward kinematic generating process 1111. [

Knowing the lengths of the various links in the main tool manipulator 620 and knowing the position of the base 629 of the main tool manipulator 620 allows the geometric relationship to be used in the main workspace coordinate system 680, It is possible to generate the position of the light source 621. Thus, the forward kinematic generation process 1111 uses the angle from the process 1110 to generate the position Pmtm of the main tool grip 621 in the main workspace coordinate system 680. [ The forward kinematic generation process 1111 proceeds to the mapping process 1112 according to the global coordinates .

According to the global coordinates, the mapping process 1112 maps the position Pmtm of the main workspace coordinate system 680 according to the position Pmtm_wc of the global coordinate system 670 (Fig. 6A). Specifically:

here,

는 4x4 균등 불변 변환이며, 이것은 전역 좌표 시스템(670)의 좌표에 따라 작업공간 좌표 시스템(680)의 좌표를 맵핑한다. 완료시, 전역 좌표에 따라 맵핑 과정(1112)은 주 도구 그립 분리에 따라 손 생성 과정(1130)으로 진행한다.

Is a 4x4 uniform invariant transformation which maps the coordinates of the workspace coordinate system 680 according to the coordinates of the global coordinate system 670. Upon completion, according to the global coordinates, the mapping process 1112 proceeds to the hand generation process 1130 according to the main tool grip separation .

Referring back to hand position data receiving process 1120, a hand position data receiving process 1120 receives the index finger position and orientation (Pindex, Rindex) and thumb position and orientation (Pthumb, Rthumb) (search). The index finger position and orientation (Pindex, Rindex) and the thumb position and orientation (Pthumb, Rthumb) are based on data from the tracking system. The hand position data reception process 1120 proceeds to the hand position generation process 1121. [

The hand position generating process 1121 maps the index finger position and orientation (Pindex, Rindex) and the thumb position and orientation (Pthumb, Rthumb) according to the control point position and orientation of the tracking coordinate system as described above, Are incorporated herein by reference. The position Phand is the position of the control point in the tracking coordinate. The hand position generating process 1121 proceeds to the mapping process 1122 according to the global coordinates .

The use of control point positions in presence detection is merely exemplary and is not intended to be limiting. In view of the present disclosure, presence detection can be done, for example, using the position of the forefinger and using the position of the thumb tip, or using only one of these positions. The process described below is equivalent for each of these various positions associated with a part of a human hand.

According to the global coordinates, the mapping process 1122 maps the position Phand of the tracking coordinates according to the position Phand_wc of the global coordinate system 670 (Fig. 6A). Specifically:

here,

는 4x4 균등 불변 변환이며, 이것은 전역 좌표 시스템(670)의 좌표에 따라 추적 좌표 시스템(650)의 좌표를 맵핑한다. 완료시, 전역 좌표에 따라 맵핑 과정(1112)은 주 도구 그립 분리에 따라 손 생성 과정(1130)으로 진행한다.

Is a 4x4 uniform invariant transformation that maps the coordinates of the tracking coordinate system 650 according to the coordinates of the global coordinate system 670. [ Upon completion, according to the global coordinates, the mapping process 1112 proceeds to the hand generation process 1130 according to the main tool grip separation .

According to the main tool grip separation, the hand generation process 1130 generates a separation distance dsep between the position Pmtm_wc of the global coordinate system 670 and the position Phand_wc of the global coordinate system 670. In one embodiment, the separation distance dsep is:

Upon completion , the hand generation process 1130 proceeds to the distance safety check procedure 1131 according to the main tool grip separation .

The distance safety check procedure 1131 compares the separation distance dsep with the safety distance threshold. This threshold should be small enough to be conservative, allowing the physician to change the grip or manipulate the extreme end of the end effector. If the separation distance dsep is less than the safe distance threshold, the distance safety check procedure 1131 proceeds to the hand presence on procedure 1140. [ Conversely, if the separation distance dsep exceeds the safe distance threshold, the distance safety check procedure 1131 proceeds to the hand presence off procedure 1150. [

The hand presence process 1140 determines whether the system 100 is remote-controlled. If the system 100 is being remotely controlled, no action is required and the remote control is still allowed, so the process 1140 skips to the beginning of the process 1100. If the system 100 is not being remotely controlled, the hand presence process 1140 sends a hand presence event to the system event providing process 1160, which in turn sends the hand presence event to the system controller 140. [

The hand presence off procedure 1150 determines whether the system 100 is in a remote steered state. If the system 100 is not in a remote steered state, no action is required, so the process 1150 skips to the start process 1100. If the system 100 is in the remote steered state, the hand presence off process 1150 sends a hand incident event to the system event providing process 1160, which in turn sends the hand member incident to the system controller 140.

The system controller 140 determines whether a hand presence event or a handoff event requires a change in the system operation mode and issues appropriate commands. In one aspect, the system controller 140 enables remote manipulation in response to a hand present event, e.g., allowing remote manipulation, and when a remote manipulation minimally invasive surgical instrument is connected to the main tool grip, I can not control. As is known to those skilled in the art, the remote manipulation minimally invasive surgical instrument may be detachably connected to the main tool grip.

In another aspect, it is determined whether the hand presence event and handoff event are used by the system controller 140 in combination with other events to allow remote control. For example, the presence detection of a physician's head may be combined with the detection of the presence of a physician's hand or hands in determining whether to allow remote control.

Similarly, as described above, the hand presence event and handoff event can be used by the system controller 140 to control the display of the user interface on the display of the minimally invasive surgical system. When the system controller 140 receives a handshake event and the user interface is not turned on, the system controller 140 sends an instruction to the display controller 150 to turn on the user interface. The display controller 150 executes the portion of the user interface module 155 required to turn on the user interface on the processor 151. [ When the system controller 140 receives the hand existence event, if the user interface is turned on, the system controller 140 sends an instruction to the display controller 150 to turn off the user interface. The display controller 150 executes the portion of the user interface module 155 required to turn off the user interface on the processor 151. [

The hand presence event and the handoff event may be used by the system controller 140 in combination with other events to determine whether to display the user interface. Thus, user interface display control and remote control control are examples of system mode control using presence detection and are not intended to limit system control to these two specific modes.

For example, presence detection can also be used to control proxy visuals such as those described more fully below. In addition, various modes, e.g., a combination of remote control and proxy visual display, can be controlled by the system controller 140 based on hand presence events and handoff events.

The hand presence detection also removes the dual use of the main tool grips 621L and 621R, for example by engaging the foot pedal and then using the main tool grips 621L and 621R to position the user It is useful to avoid controlling the interface. When the main tool grip is dual-use, for example, when used both to control the surgical instrument and the user interface, the physician typically has to press the foot pedal to switch the user interface operating mode. If, for some reason, the physician did not press the foot pedal, but the system is considered switched to the user interface operating mode, movement of the main tool grip may result in undesirable operation of the surgical instrument. The presence detection process 1100 can be used to prevent this problem and eliminate the dual use of the main tool grip.

Using the presence detection process 1100, in one example, when a hand presence off event is received by the system controller 140, the system controller 140 moves the main tool manipulators 620L, 620R (Figure 6A) And sends an instruction to the display controller 150 to display the user interface on the display of the pseudo console 185B. The physician's hand movements are tracked and can be used to control elements of the user interface, for example by moving the slider switch or changing the display. As noted above, the control point is mapped in the eye coordinate frame, and thus can be associated with the position of one element in the user interface. The operation of the control point can be used to manipulate the element. This is accomplished without the need for the physician to activate the foot pedal, and is done so that the physician can not inadvertently move the surgical instrument. This eliminates the problem associated with using the main tool grip to control both the surgical instrument and the user interface.

In this example, the global coordinate frame is an example of a common coordinate frame. The use of a global coordinate frame as a common coordinate frame is only an example, and is not intended to be limiting.

Main finger track grip

12 illustrates an example of a main finger track grip 1270. [ The main finger track grip 1270 is an example of a main finger track grip 170,270.

The main finger track grip 1270 includes a compression element 1210 and two finger rings 1220 and 1230. The compression body 1210 has a first end portion 1213 and a second end portion 1214. A body section 1215 extends between the first end 1213 and the second end 1214.

The compression body 1210 has an outer outer surface. The outer outer surface includes a first portion 1216 and a second portion 1217. A first portion 1216, e.g., an upper portion, extends between the first end 1213 and the second end 1214. A second portion 1217, for example a lower portion, extends between the first end 1213 and the second end 1214. [ The second portion 1217 is removed from the first portion 1216 in the opposite direction.

In one embodiment, the outer outer surface is the surface of the fabric case. The fabric is suitable for use in the operating room. The fabric case surrounds the compressible foam. The foam is selected to provide resistance to compression and expansion as the compression is released. In one embodiment, several foam strips are included in the fabric case. The foam should also be able to flex so that the first portion 1216 can be positioned between the first finger and the second finger of the human hand as the end of the first finger moves toward the end of the second finger.

The body compartment 1215 has a length L between the finger ring 1220 and the finger ring 1230. As described above, the length L is selected to limit the separation between the first finger in the ring 1220 and the second finger in the ring 1230 (see FIG. 2A).

In one embodiment, the body compartment 1215 has a thickness T. As illustrated in Figure 2C, the thickness T is defined by the area 1236 on the second portion 1217 of the outer outer surface adjacent to the end 1214 and the second portion 1236 adjacent the end 1213, The second portion 1217 is selected so that it does not completely contact itself along the length L when the region 1226 on the first portion 1217 is configured to lightly touch.

The first finger ring 1220 is secured to the compression body 1210 adjacent the first end 1213. The ring 1220 extends around the area 1225 of the first portion 1216 of the outer outer surface of the compression body 1210. When the first finger ring 1220 is placed on the first finger of the human hand, the region 1225 contacts the first finger, for example, a first portion of the first portion 1216 of the outer outer surface, / RTI >

In this example, the finger ring 1220 has two ends, a first fabric end 1221A and a second fabric end 1221B. Ends 1221A and 1221B are the ends of the fabric strip secured to the compression body 1210. A portion of the loop fabric 1222B is attached to the inner surface of the end portion 1221B and a portion of the hook fabric 1222A is attached to the outer surface of the end portion 1221A. An example of a hook fabric and a loom fabric is a nylon anchoring tape consisting of two strips of nylon fabric, one with very small hook-shaped seals and the other with a rough surface. The two strips form a strong bond when pressed together. An example of a commercially available fixed tape is a VELCRO (R) fixed tape (VELCRO (R) is a registered trademark of Velcro Industries B.V.).

The second finger ring 1230 is secured to the compression body 1210 adjacent the second end 1214. [ The loop 1230 extends around the region 1235 of the first portion 1216 of the outer outer surface of the compression body 1210. When the second finger ring 1230 is placed on the second finger of the human hand, the region 1235 contacts the second finger, for example, a second portion of the first portion 1216 of the outer outer surface, / RTI > The second portion 1235 of the first portion is removed in reverse from the first portion 1225 of the first portion.

In this example, the finger ring 1230 also has two ends, a first fabric end 1231A and a second fabric end 1231B. End portion 1231A and end portion 1231B are the ends of the fabric strip secured to compression body 1210. A portion of the loop fabric 1232B is attached to the inner surface of the end portion 1231B and a portion of the hook fabric 1232A is attached to the outer surface of the end portion 1231A.

The first position tracking sensor 1211 is fixed to the first finger ring 1220. The second position tracking sensor 1212 is fixed to the second finger ring 1230. The location tracking sensor may be any of the sensor elements described above. In one example, the location tracking sensors 1211 and 1212 are passive electromagnetic sensors.

Proxy Visual System

In one aspect, a hand tracking control system may be used to control any of a plurality of proxy visuals that may be used by a physician to supervise another physician. For example, when the doctor 180 (FIG. 1A) is being supervised by the doctor 181 using the primary finger track grip 170, the doctor 181 may use the primary finger tracking grip 170, And the physician 180 may use the main tool grip to control the remote control dependent surgical instrument.

Or, the physician 181 may teletreat or control a virtual hand on the display. The physician 181 may also show how to manipulate the main tool grip in the pseudo console by manipulating the virtual image of the main tool grip 621 presented on the display. These examples of proxy visuals are merely illustrative, and are not intended to be limiting.

Also, the use of a primary finger track grip 170 that is not on the pseudo console is exemplary and is not intended to be limiting. For example, in the presence detection system described above, the physician's doctor may move his / her hand from the main tool grip and then use that hand to supervise another physician as the hand is tracked by the hand tracking system .

To facilitate supervision, a proxy visual module (not shown) is processed as part of the vision processing subsystem in one aspect. In this aspect, a running module receives the position and orientation of the control point of the supervisor's hand to create a stereoscopic image, which is combined with the endoscopic camera image in real time to create a pseudo-console 185, an assistant display, 187).

When the doctor 181 initiates supervision by taking a predetermined action, for example by taking a hand gesture pose, the proxy visual system loop is activated, e.g., a proxy visual module is executed in the processor module. A particular action, e.g., a hand gesture pose, used as a predetermined action is not necessarily required as long as the system controller 140 (FIG. 1) is configured to recognize the action.

In one aspect, the proxy visual is a virtual ghosting mechanism 1311 (FIG. 13) controlled by the main finger track grip 170 and is controlled by one of the main tool manipulators of the pseudo console 185, 1310 are controlled. Physician 181 sees both instruments 1310 and 1311 on display 187 and physician 180 sees both instruments 1310 and 1311 on the stereoscopic display of pseudo console 185. [ The use of the virtual ghost mechanism 1311 as a proxy visual is only an example, and is not limited to this particular image. In view of the present disclosure, other images facilitating the distinction between the image representing the proxy visual and the image of the actual end actuator of the remote control dependent surgical instrument may also be used as a proxy visual.

The virtual ghost mechanism 1311 appears to be similar to the actual mechanism 1310 except that the virtual ghost mechanism 1311 is displayed in a manner that clearly distinguishes the actual mechanism 1310 from the virtual ghost mechanism 1311 , Transparent or translucent ghost images, images with distinct colors, etc.). The control and operation of the virtual ghost mechanism 1311 is the same as described above for the actual remote manipulation instrument. Thus, the physician 181 can manipulate the virtual ghost mechanism 1311 using the primary finger track grip 170 to demonstrate proper use of the remote control dependent surgical instrument 1310. [ Physician 180 may mimic the operation of virtual ghost mechanism 1311 using instrument 1310. [

The virtual ghost mechanism is disclosed in commonly assigned U. S. Patent Application Publication No. < RTI ID = 0.0 > This is further described in US 2009/0192523 A1 (filed March 31, 2009, entitled "Combinations of Surgical Instruments"), which is incorporated herein by reference in its entirety. Also, U. S. Patent Application No. 12 / 485,503 (filed June 16, 2009, entitled "Virtual Measurement Tool for Minimally Invasive Surgery"); U.S. Patent Application No. 12 / 485,545, filed June 16, 2009, entitled "Starting a Virtual Measurement Tool for Minimally Invasive Surgery"); U.S. Patent Application Publication No. US 2009/0036902 A1 (Launched on August 11, 2008, "Interactive User Interface for Robotic Minimally Invasive Surgical System"); U.S. Patent Application Publication No. US 2007/0167702 A1 (filed December 30, 2005, entitled "Medical Robotic System Providing Three-dimensional Telestations"); U.S. Patent Application Publication No. US 2007/0156017 A1 (filed Dec. 30, 2005, entitled "Stereoscopic Telestations for Robotic Surgery") and U.S. Patent Application Publication Nos. US 2010/0164950 A1 (filed May 13, 2009, entitled "Effective 3-D Telesation for Local Robotics Supervision"), all of which are incorporated herein by reference.

In another aspect, the proxy visual is a pair of virtual hands 1410, 1411 (FIG. 14) controlled by a main finger track grip 170 and a second main finger track grip which is not visible in FIG. The remote control dependent surgical instruments 1420 and 1421 are controlled by the main tool manipulator of the pseudo console 185. The physician 181 sees the video image 1400 on the display device 187 and the doctor 180 sees the video image 1400 on the stereoscopic display of the pseudo console 185. [ The virtual hands 1410 and 1411 are displayed in a manner such that they are clearly distinguished from other objects in the video image 1400.

The opening and closing of the thumb and forefinger of the virtual hand is controlled using the grip closure variable ggrip described above. The position and orientation of the virtual hand is controlled by the control point position and orientation as described above, and they are also mapped in the eye coordinate space as described above.

Thus, as the physician 181 moves the right hand three-dimensionally, the virtual hand 1411 follows the movement in the video image 1400. [ The physician 181 may roll the virtual hand 1411 to indicate that the physician 180 is not holding the remote control dependent surgical instrument 1421. The physician 181 moves the virtual hand 1410 to a specific position and then uses the thumb and forefinger movement to move the remote manipulation dependent surgical instrument 1420 to that position to hold the tissue 180 You can tell. When physician 180 grabs tissue with instrument 1420, physician 181 may use virtual hand 1410 to instruct physician 180 to move tissue. All of these occur in real time, and the virtual hands 1410 and 1411 are superimposed on the stereoscopic endoscopic image. However, proxy visuals can also be used in monocular stereoscopic systems.

In another aspect, physician 181 may use the hand gesture pose to set the display mode such that the proxy visual is a virtual ghosting mechanism 1510 and a virtual telestration device 1511 presented in video image 1500 (Fig. 15) Change. The telestration device 1511 is controlled by the main finger tracking grip 170, and the second main finger tracking grip which is not visible in FIG. 1 controls the virtual ghost mechanism 1511.

The remote control dependent surgical instruments 1520 and 1521 are controlled by the main tool manipulator of the pseudo console 185. The physician 181 sees the video image 1500 on the display device 187 and the doctor 180 sees the video image 1500 on the stereoscopic display of the pseudo console 185. [ The virtual telestration device 1511 and the virtual ghost mechanism 1411 are displayed in a manner that clearly distinguishes them from other objects in the video image 1500. [

For telstraing with the virtual telestration device 1511, the physician 181 places the thumb and forefinger as if holding a virtual pen or pencil, then places the thumb and forefinger at this location Moves the right hand while holding the telescope in the displayed video image. In the video image 1500, the physician 181 positions the thumb and forefinger in this manner to create a mark 1512 to illustrate where the surgical instrument 1521 should cut tissue. After making the mark 1512, the doctor 1810 separates the thumb and forefinger and moves the virtual telestration device 1511 to the position shown in the video image 1500.

The display capability of the virtual telestration device 1511 is controlled using the grip closure variable ggrip described above. When the thumb and forefinger touch lightly, as is well known, the grip closure parameter ggrip is mapped according to the initial value of the second range, such that when the grip closure parameter ggrip is in the second range, 1511 can be telesired. After mapping according to the eye coordinate system, control point location and orientation can be used to control the operation of the virtual telestration device 1511.

The above description and the accompanying drawings illustrating aspects and embodiments of the present invention should not be construed as limitations, but rather define the invention in which the claims are protected. Various mechanical, structural, structural, electrical, and operational changes may be made without departing from the spirit and scope of the present disclosure and claims. In some instances, well-known circuits, structures and techniques have not been shown or described in detail in order to avoid complicating the present invention.

Furthermore, the terminology of the present specification is not intended to limit the present invention. For example, spatial relative terms such as "under", "under", "under", "up", "top", "near", "distal", etc. may refer to one element or feature Can be used to describe the relationship with the feature. These spatial relative terms are intended to encompass different locations (i.e., locations) and orientations (i.e., rotatable arrangements) of the device in use or in operation, in addition to the locations and orientations shown in the figures. For example, if a device in the drawing is reversed, the elements described as "under" or "under" another element or feature will be "above" or "directly above" another element or feature. Thus, the exemplary term "below" can encompass both upper and lower positions and orientations. The device can be oriented differently (rotated 90 degrees, or in other orientations) and the spatially relative description used herein is interpreted accordingly. Likewise, the description of the movement along the various axes and about the movement around the axis also encompasses various special device positions and orientations.

Also, the singular forms "a" and "it" are intended to include plural forms unless the context clearly dictates otherwise. It should be understood that the terms " comprises, "" including," " comprising, "or the like specify the presence of stated features, steps, operations, elements and / Elements, elements, and / or groups. The components described in pairs may be electrically or mechanically connected directly, or they may be indirectly connected through one or more intermediate components.

Memory refers to volatile memory, non-volatile memory, or some combination of the two. The processor is coupled to a memory containing instructions executed by the processor. This may be accomplished within a computer system, or through a connection with a modem and an analog line, or with another computer via a digital interface and a digital transmission line.

Herein, the computer program product may be embodied as a medium configured to store computer readable code necessary for any one or in some combination of the processes described with respect to hand tracking, or any one or any combination of the processes described in connection with hand tracking Readable < / RTI > code. Some examples of computer program products are signals transmitted over a network representing a CD-ROM disk, a DVD disk, a flash memory, a ROM card, a floppy disk, a magnetic tape, a computer hard drive, a server on a network, and computer readable program code . The non-transitory type computer program product may be configured to store computer readable instructions for any or any combination of the processes described in connection with the various controllers, or any one or any combination of the processes described in connection with the various controllers Non-transitory type medium in which computer-readable instructions are stored. Non-transitory type computer program products are CD-ROM disks, DVD disks, flash memory, ROM cards, floppy disks, magnetic tape, computer hard drives, and other non-transitory physical storage media.

In view of the present disclosure, the instructions used in any or all of the processes described in connection with the hand tracking can be implemented in a wide variety of computer system configurations using the operating system of interest and the computer programming language of interest to the user.

The use of different memories and processors in Fig. 1 is merely exemplary, and is not intended to be limiting. In some aspects, a single hardware processor may be used, and in other aspects, multiple processors may be used.

Also, in each example, several processes were distributed between the hand tracking controller and the system controller. This is also an example, and is not intended to be limiting. The various processes can be distributed among the controllers or integrated into the controller without changing the operating principle of the hand tracking process.

All examples and illustrative references are non-limiting and should not be used to limit the claims to the specific embodiments and embodiments described herein and their equivalents. A title is merely a formality, and should not be used to limit the subject in any way, and the content under one title may refer to or apply to the content under one or more titles. Finally, in light of the present disclosure, certain features described in connection with one aspect or embodiment may be applied to other disclosed aspects or embodiments of the present invention even if not specifically shown in the drawings or described in the context.

Claims (14)

In a remote control system,
Controller,
The controller
Receive human hand position data from a hand tracking unit using a main tool grip;
Creating a hand position using the hand position data, the hand position indicating a position of at least a portion of the hand of the person using the main tool grip;
Determining whether the hand position is within a threshold distance from the position of the main tool grip and determining is based on the hand position and the position of the main tool grip;
Wherein movement of the remote control dependent mechanism is permitted when the hand position is within a threshold distance from the position of the main tool grip and movement of the remote control dependent mechanism is based on movement of the main tool grip;
And inhibit movement of the remote control dependent mechanism if the hand position is not within a threshold distance from the position of the main tool grip. The remote control system of claim 1, wherein the hand position includes a forefinger position. 2. The remote control system of claim 1, wherein the controller is further configured to transmit a hand presence event within the remote control system to allow movement of the remote control dependent mechanism. 2. The remote control system of claim 1, wherein the controller is further configured to transmit a handshake event in the remote control system to inhibit movement of the remote control dependent mechanism. The method according to claim 1,
The main tool grip comprising: a main tool grip configured to be connected to the controller;
The hand tracking unit comprising: a hand tracking unit configured to track a position of at least a portion of the human hand using the main tool grip; And
And an actuator configured to be connected to the remote control dependent mechanism. 6. The system of claim 1 or 5, wherein the controller
Convert the hand position data to a hand gesture pose;
Determine whether the hand gesture pose is a hand gesture pose used in a plurality of known hand gesture trajectories;
The method of claim 1, wherein if the hand gesture pose is not a hand gesture pose used in a plurality of known hand gesture trajectories, the first system is changed from a first one of the plurality of system operating modes to a second one of the plurality of system operating modes And a second mode of the plurality of system operating modes is selected based on the hand gesture pose;
If the hand gesture pose is a hand gesture pose used in a plurality of known hand gesture trajectories, converting the hand position data to a hand gesture trajectory and selecting a plurality of system operating modes from a first one of the plurality of system operating modes Wherein the third mode of the plurality of system operating modes is selected based on the hand gesture trajectory. ≪ Desc / Clms Page number 16 > 6. The apparatus according to any one of claims 1, 3, and 5, wherein the controller
Converting the hand position data into a control point position and a control point position orientation of a single control point;
Transmit a command to move the remote control dependent mechanism, and wherein the command is further configured to be based on the single control point. In a remote control system,
Controller,
The controller
Receive human hand position data from a hand tracking unit using a main tool grip;
Creating a hand position using the hand position data, the hand position indicating a position of at least a portion of the hand of the person using the main tool grip;
Determine whether the position of the main tool grip is within a threshold distance from the hand position by determining a distance between the position of the main tool grip and the hand position;
Wherein the system event is configured to be based on whether the position of the main tool grip is within a threshold distance from the hand position. 9. The method of claim 8,
If the main tool grip is not within a threshold distance from a person's hand;
And wherein the main tool grip is a hand presence event if the main tool grip is within a threshold distance from a person's hand. 9. The method of claim 8,
A processing unit;
Another controller configured to receive the system event;
A user interface generated by a module executing on the processing unit;
A display device configured to output a display of a user interface, wherein the display of the user interface further includes a display device different from the display of the image of the surgical site,
And if the system event is a hand presence event, the other controller is configured to turn off the display of the user interface. 9. The method of claim 8,
Further comprising another controller configured to receive the system event,
If the system event is a hand presence event, the other controller is configured to allow remote manipulation of a remote control dependent mechanism coupled to the main tool grip,
And if the system event is a handshake event, the other controller is configured not to allow remote control of the remote control dependent device. 9. The method of claim 8,
The main tool grip comprising: a main tool grip configured to be connected to the controller;
The hand tracking unit comprising: a hand tracking unit configured to track a position of at least a portion of the human hand using the main tool grip; And
Further comprising an actuator configured to be coupled to a remote control dependent mechanism. 13. The system according to claim 8 or 12, wherein the controller
Convert the hand position data to a hand gesture pose;
Determine whether the hand gesture pose is a hand gesture pose used in a plurality of known hand gesture trajectories;
The method of claim 1, wherein if the hand gesture pose is not a hand gesture pose used in a plurality of known hand gesture trajectories, the first system is changed from a first one of the plurality of system operating modes to a second one of the plurality of system operating modes And a second mode of the plurality of system operating modes is selected based on the hand gesture pose;
If the hand gesture pose is a hand gesture pose used in a plurality of known hand gesture trajectories, converting the hand position data to a hand gesture trajectory and selecting a plurality of system operating modes from a first one of the plurality of system operating modes Wherein the third mode of the plurality of system operating modes is selected based on the hand gesture trajectory. ≪ Desc / Clms Page number 16 > 13. The system according to claim 11 or 12, wherein the controller
Converting the hand position data into a control point position and a control point position orientation of a single control point;
Transmit a command to move the remote control dependent mechanism, and wherein the command is further configured to be based on the single control point.

Images