KR20090034813A - Surgical tool position and identification indicator displayed in a boundary area of a computer display screen - Google Patents

Abstract

An endoscope captures images of a surgical site for display in a viewing area of a monitor. When a tool is outside the viewing area, a GUI indicates the position of the tool by positioning a symbol in a boundary area around the viewing area so as to indicate the tool position. The distance of the out-of-view tool from the viewing area may be indicated by the size, color, brightness, or blinking or oscillation frequency of the symbol. A distance number may also be displayed on the symbol. The orientation of the shaft or end effector of the tool may be indicated by an orientation indicator superimposed over the symbol, or by the orientation of the symbol itself. When the tool is inside the viewing area, but occluded by an object, the GUI superimposes a ghost tool at its current position and orientation over the occluding object.

Description

SURGICAL TOOL POSITION AND IDENTIFICATION INDICATOR DISPLAYED IN A BOUNDARY AREA OF A COMPUTER DISPLAY SCREEN}

The present invention relates generally to robotic surgical systems and, more particularly, to surgical instrument position and confirmation indicators displayed in the border zone of a computer display screen.

Robotic surgical systems, such as those used to perform minimally invasive surgical procedures, reduce pain, short hospital stays, rapid return to daily life, minimal scars, and short recovery times compared to conventional open surgery techniques. It offers many advantages, including alleviation of tissue damage. Thus, the need for minimally invasive surgery using a robotic surgical system is increasing significantly.

One example of a robotic surgical system is the da Vinci ^® surgical system of Intuitive Surgical, Inc. of Sunnyvale, California. The da Vinci ^® surgical system includes a surgeon's console, a patient-side cart, a high performance 3-D imaging system, and an EndoWrist ^TM joint instrument owned by Intuitive Surgical Inc. (articulating instrument), and the endorist articulation instrument, when added to the motion of the robotic arm holding the surgical instrument, allows for a full 6 degrees of freedom to match the actual movement of open surgery. It is made of a circular wrist.

The surgeon's console of the da Vinci ^® surgical system has a high resolution stereoscopic image display with two progressive scan cathode ray tubes. The da Vinci ^® surgical system provides higher fidelity than polarization technology, shutter eyeglass technology, or other technology. Each eye looks at each cathode ray tube showing the left or right view through an objective lens and a series of mirrors. During the entire procedure, the surgeon sits comfortably in the appropriate place to display and manipulate the 3-D surgical image and observes the display.

A stereoscopic endoscope is positioned near the surgical site to capture the left and right screens for display on a stereoscopic image display. However, if the surgical tool is outside the viewing area of the stereoscopic image display, the surgeon may not know in time how far and in what direction the surgical tool is placed. This fact makes it difficult for surgeons to bring surgical instruments to the surgical site. The surgeon may also be embarrassed if the surgical tool suddenly appears in sight. Even when the surgical instrument is within the viewing zone of the stereoscopic image display, the surgeon may determine which surgical instrument is the surgical instrument (eg, the robotic arm on the patient side cart) in combination with the surgical instrument. You may not know. This makes it difficult, for example, for the surgeon to instruct the patient-side surgical assistant to replace the surgical instrument with another while the surgery is in progress.

In order to locate the surgical instrument outside of the viewing zone of the stereoscopic image display, it may be necessary to move the endoscope until the surgical tool appears in the viewing zone. In this case, when the surgical tool is delivered to the surgical site, the camera's zooming and focusing adjustments may need to be adjusted frequently, which can make the surgeon boring and waste time. If the surgical tool is within the field of view of the camera, but the surgical tool is outside of the viewing zone due to zoom-in adjustment, a zoom-out adjustment may be performed to return the surgical tool to the viewing zone. However, such a zoom-out adjustment may not be desirable if sophisticated surgery is performed that requires close observation by the surgeon.

Thus, one of the various embodiments of the present invention is a method of indicating the position of a surgical instrument relative to an image displayed on the computer display screen when the surgical instrument is outside the viewing zone of the computer display screen.

Another one of the various embodiments of the present invention is a method of indicating the distance of a surgical instrument from an image displayed on a computer display screen when the surgical instrument is outside the viewing zone of the computer display screen.

Yet another of the various embodiments of the present invention is a method of indicating the orientation of a surgical instrument relative to an image displayed on the computer display screen when the surgical instrument is outside the viewing area of the computer display screen.

Yet another of the various embodiments of the present invention is a method of indicating the position or orientation of a surgical instrument relative to an image displayed on the computer display screen when the surgical instrument is blocked within the viewing zone of the computer display screen.

Yet another of the various embodiments of the present invention is a method of indicating the position or orientation of a surgical instrument relative to an image displayed on the computer display screen when the surgical instrument is blocked within the viewing zone of the computer display screen.

Another one of the various embodiments of the present invention resides on a computer display screen that clearly identifies which patient side manipulator is connected to which surgical tool to enhance the surgeon's ability to perform surgery and communication with the surgical assistant. It is a method of indicating a surgical instrument identification mark.

These and additional objects are accomplished by various embodiments of the present invention, in brief, one embodiment comprises the steps of determining the position of a surgical instrument; Determining a position of a symbol within a boundary of the computer display screen to indicate the position of the surgical instrument; And displaying the symbol at the determined location of the border zone.

Another embodiment includes a processor, the processor being coupled to the computer display screen to determine a current position of the surgical instrument; Determine the position of the symbol within the boundary of the computer display screen to indicate the position of the surgical instrument; And a device for indicating the position of the surgical instrument on a computer display screen configured to cause the symbol to be displayed at a determined location within the border zone.

Another embodiment includes a surgical instrument; A first robot arm mechanically coupled to the surgical instrument to adjust the position and direction of the surgical instrument; camera; A second robot arm mechanically coupled to the camera to adjust the position and direction of the camera; Computer display screens; And a processor coupled to the robotic arm, a camera, and a computer display screen, the processor being configured to cause a symbol to be displayed at a boundary of the computer display screen to indicate the position of the surgical instrument. to be.

In a preferred embodiment of the method, apparatus and medical robotic system, a symbol may be associated with the surgical instrument and / or the surgical instrument by other means, such as color or text or numerical information displayed on or adjacent to the symbol. Provide information identifying the coupled patient side manipulator. In the latter case, the text information is displayed continuously on the computer display screen. Alternatively, the symbol may only be displayed when a cursor is placed over the symbol or when the symbol is clicked on using the pointing device.

Additional objects, features and advantages of the various embodiments of the present invention can be seen from the following description of the preferred embodiments described in connection with the accompanying drawings.

1 is a plan view of an operating room using a robotic surgical system used in various embodiments of the present invention.

2 shows two surgical instruments positioned in the field of view of an endoscope camera.

3 shows one surgical instrument positioned within the field of view of the endoscope camera and one surgical instrument positioned out of the field of view of the endoscope camera.

4 illustrates a first computer display screen in a method of indicating the position of a surgical instrument when the surgical instrument is out of view of an endoscope camera used in various embodiments of the present invention.

5 shows a second computer display screen in a method of indicating the position of a surgical instrument when the surgical instrument is out of view of an endoscope camera used in various embodiments of the present invention.

FIG. 6 shows a third computer display screen in a method of indicating the position of a surgical instrument when the surgical instrument is out of view of an endoscope camera used in various embodiments of the present invention.

7 shows a fourth computer display screen in a method of indicating the position of a surgical instrument when the surgical instrument is blocked within the field of view of an endoscope camera used in various embodiments of the present invention.

FIG. 8 shows a flowchart in a method for indicating the position of a surgical instrument when the surgical instrument is out of view or blocked within the field of view of the endoscope camera used in various embodiments of the present invention.

9 shows a left screen and a right screen at a point in an endoscope camera reference frame used in a robotic surgical system configured to perform the method described in connection with FIG. 8 for use in various embodiments of the present invention.

Fig. 10 shows the entire left camera screen displayed in the left viewing zone of the computer monitor.

11 shows some left camera screens displayed in the left viewing zone of the computer monitor.

FIG. 12 shows a flowchart of a method for identifying a surgical instrument in a camera screen that may be used in the method described in connection with FIG. 8 used in various embodiments of the present invention.

1 shows, as one example, a plan view of an operating room using a robotic surgical system. In this case, the robotic surgical system is a minimally invasive robotic surgical system (MIRS) system 100, which is generally located on the patient P side lying on the surgical table (0). It includes a console (C) that is used by the surgeon (S) during the minimally invasive diagnosis or minimally invasive surgery with assistance from one or more assistants (A).

The console includes a 3-D monitor 104, one or more operable master manipulators 108, 109 (also referred to herein as "control devices" and "input devices") that show the surgeon an image of the surgical site. And a processor 102. Control devices 108 and 109 may include one or more of a variety of input devices, such as joysticks, glove, trigger-gun, manual controllers, and the like. The processor 102 is a personal computer that is integrated into the console or located next to the console.

The surgeon captures the surgical site by stereoscopic endoscope 140 (having a left camera and a right camera to capture the left stereoscopic image and the right stereoscopic image) and displays the three displayed on the 3-D monitor 104 of the console. Observing in D-image, the processor 102 is associated with the associated slave manipulators 128, 129 (also referred to herein as "robot arms" and "patient side manipulators") detachably coupled surgical tools 138. 139) (also referred to herein as "surgical instruments") to implement the minimally invasive surgery method by manipulating the control devices 108, 109 respectively.

Each of the surgical instruments 138, 139, as well as the endoscope 140, cannula or other surgical instrument guide (not shown) to extend through the corresponding minimally invasive incision, such as the incision 166, to the surgical site. Is inserted into the patient. Each robot arm is conventionally formed with a linkage, such as linkage 162, which is coupled to each other and operated through a motor controlled joint, such as joint 163. FIG.

The number of surgical instruments used at one time and the number of robotic arms used in the minimally invasive robotic surgical system 100 are largely dependent on diagnostic or surgical methods and spatial constraints within the operating room, among other factors. If one or more of the surgical instruments used during the procedure need to be replaced, the surgeon removes the surgical instrument from the robotic arm so that the surgical instrument is no longer in use and the other surgical instruments 131 in the tray T of the operating room. Can be instructed to replace. To assist the assistant in identifying the surgical instrument to be replaced, each of the robotic arms 122, 128, 129 may have an identification number or color indicator printed on such as a setup joint.

Preferably, the monitor 104 is positioned near the surgeon's hand so that the surgeon displays an image directed towards the feeling of actually looking directly down the surgical site. For this purpose, it is desirable that the image of the surgical instruments 138, 139 appear to be actually placed where the surgeon's hand is located. To this end, the processor 102 preferably changes the orientation of the control devices 108, 109 to orient the surgical instrument 138, 139 as shown by the endoscope 140.

The processor 102 performs various functions in the minimally invasive robot surgery system 100. One important function that the processor 102 performs is that the control device 108, 109 via the control signal bus 110 allows the surgeon to effectively move and / or manipulate each surgical instrument 138, 139. It translates the mechanical movement of each robot arm (128, 129) to transmit (translate). Another important function is as described herein, when the surgical instrument is outside the camera capture screen displayed on the monitor 104 or blocked within the camera capture screen displayed on the monitor 104. The way to mark locations is. Another important function is to perform a method of easily identifying the surgical instrument and / or each patient-side manipulator on the monitor 104 to facilitate the surgeon / surgical assistant's communication.

Although the processor 102 has been described as a personal computer, in practice the processor 102 may be implemented in any combination of hardware, software, and firmware. In addition, the functionality of the processor may be executed by one unit as described herein, or may be executed by being divided into different components, which may in turn be implemented in any combination of hardware, software and firmware. .

While performing the minimally invasive surgery, the surgical instrument 138, 139 is viewed by the monitor 104 so that the surgeon can see and use the surgical instruments 138, 139 through the monitor 104 while performing the minimally invasive surgery. It is desirable to remain within zone 200 (as shown in FIG. 2). If one surgical instrument 138 is outside of the viewing zone 200 of the monitor 104 (as shown in FIG. 3), the surgeon cannot see the surgical instrument on the monitor 104 and proceeds to surgery. The surgical instrument cannot be used properly during the process. In addition, the surgeon may have difficulty moving the surgical instrument out of sight to the observation zone 200 of the monitor 104 without knowing where the surgical instrument is out of view relative to the current observation zone 200. Can suffer.

In order to inform the surgeon about the position of the surgical instrument with respect to the surgical instrument that is out of view or blocked, the processor 102 displays the position of the surgical instrument on the monitor 104, as described in connection with FIG. It is configured with a Graphical User Interface (GUI) computer program that performs the method. However, before describing the above embodiment of the GUI, examples of the output generated by the GUI will be described with reference to FIGS. 4 to 7.

In each of FIGS. 4-7, the viewing zone 300 of the monitor 104 corresponds to the field of view of the endoscope 140 (with the proper scale of the entire field of view) as shown in FIG. 10, or in FIG. 11. It may correspond only to a portion of the field of view of the endoscope 140 as shown, (with the proper scale corresponding to the zoom-in image corresponding to the portion of the field of view displayed on monitor 104). Surgical instruments within the observation zone 300 are indicated by bold lines within the observation zone 300. The part surrounding the observation area 300 is the boundary area 400, in which a non-clickable symbol or a clickable icon (hereinafter collectively referred to as "symbol") is defined. It is positioned to indicate the position of the corresponding surgical instrument.

The symbol also preferably provides information identifying each surgical instrument and / or corresponding patient side manipulator. One way to do this is to provide the information in a color of a symbol that can match a color indication printed on a patient side manipulator, such as a setup joint. For example, the patient side manipulators 122, 128, and 129 may be displayed in colors coded red, green, and yellow, respectively, and symbols corresponding to surgical instruments attached to the patient side manipulator may also be coded in the same way. Is displayed. In an alternative embodiment, a numeric display and / or other identifying information that can match a number printed on a patient side manipulator, such as a setup joint, may be displayed on or near the symbol. For example, the patient side manipulators 122, 128, and 129 may be represented by numbers 1, 2, and 3, respectively, and symbols corresponding to surgical instruments attached to the patient side manipulators are also represented by numbers in the same manner. When text information is provided to a symbol, the text information may be displayed on or adjacent to the symbol. The text information may be displayed continuously on the computer display screen or only when the cursor is placed over the symbol or when a symbol is clicked using a pointing device (mouse, light pen, etc.).

Surgical instruments outside of the viewing zone 300 are indicated by dashed lines for the purpose of describing the method of a particular embodiment executed by the GUI. Surgical instruments indicated by such dashed lines (or extensions of surgical instruments indicated by dashed lines) are not visible to the surgeon on the monitor 104. However, the relative position of the surgical instrument with respect to the viewing zone 300 in FIGS. 4-7 is relative to the field of view of the endoscope 140 or to the field of view of the endoscope 140 within the reference endoscope camera frame. Corresponds to.

Although the surgical instruments shown in FIGS. 4-7 are represented in 2-D images, this is for simplicity purposes only and is not meant to be limited to 2-D images. Preferably, the 3-D image is displayed in the observation area 300. The indication of the direction of the symbol and in particular the end effector or surgical instrument shaft superimposed on the symbol can be seen in 2-D or 3-D in the boundary zone 400. Further, while the examples described herein relate to images captured by the endoscope 140, various embodiments of the present invention may include MRI, ultrasound, or other that may be displayed in the viewing zone 300 of the monitor 104. The present invention can also be applied to an image captured by another type of imaging device such as using an imaging method.

FIG. 4 illustrates, as a first embodiment, a GUI generation screen displayed on the monitor 104, where a first symbol 410 is used to indicate the location of the surgical instrument 138 out of view. And a direction indicator 411 is superimposed on the first symbol 410 to indicate the current direction of the end effector 215 of the surgical instrument 138 out of view. Surgical instrument 139 in the field of view is shown to partially protrude from observation zone 300 to second symbol 420 in boundary zone 400.

In this embodiment, the position of the first symbol 410 is determined by the intersection of the line 402 and the border zone 400, the line 402 of the monitor 104 from one reference point on the surgical instrument 138 out of view. It extends to the center point 401 of the observation zone 300. The position of the second symbol 420 is determined by the intersection of the shaft 222 of the surgical instrument 139 and the boundary zone 400 in the field of view.

The distance away from the viewing area 300 of the surgical instrument 138 that is out of view may be indicated in various ways, such as the size of the symbol, color, brightness / intensity, blinking frequency, or frequency. . Alternatively, the distance away from the viewing zone 300 by the surgical instrument 138 out of view can be indicated simply by indicating a distance value (eg, a distance in centimeters) above the symbol. For example, if a surgical instrument, such as indicated by reference numeral 139, is in view, its symbol may be of maximum size, such as symbol 420 of surgical instrument 139 in view. However, if a surgical instrument, such as indicated by reference numeral 138, is out of view, the size of its symbol is observed by the surgical instrument out of view so that the surgical instrument approaches toward the viewing zone 300 and becomes larger as it moves. The distance from zone 300 may be indicated. Alternatively, the color spectrum may be used to indicate the distance in the color of the symbol, or the brightness / concentration of the symbol or the frequency of blinking of the symbol as the surgical instrument approaches toward the viewing zone 300 and moves. By indicating the distance in terms of the brightness / concentration of the symbol or the frequency of blinking of the symbol, or by decreasing the frequency of the symbol relative to the nominal position of the symbol as the surgical instrument approaches towards the observation zone 300 and moves. The distance can be indicated by the frequency of the symbol.

FIG. 5 illustrates, as a second embodiment, a GUI generation screen displayed on the monitor 104, where a first symbol 510 is used to indicate the location of the surgical instrument 138 out of view. And a direction indicator 511 is superimposed on the first symbol 510 to indicate the current direction of the shaft 217 of the surgical instrument 138 out of view.

In this embodiment, the position of the first symbol 510 is determined by the intersection of the line 502 and the border zone 400, which line 502 extends along the axis of the shaft 217. The distance away from the viewing zone 300 by the surgical instrument 138 that is out of view may be indicated in the same manner as described above in connection with the drawings.

FIG. 6 illustrates, as a third embodiment, a GUI generation screen displayed on the monitor 104, where a first symbol 610 is used to indicate the location of the surgical instrument 138 out of view. And a direction indicator 611 is superimposed on the first symbol 610 to indicate the current direction of the shaft 217 of the surgical instrument 138 out of view.

In this embodiment, the position of the first symbol 610 is determined by the intersection of the trajectory 602 and the boundary area 400, where the trajectory 602 is one reference point on the surgical instrument 138 that is out of view. It is formed by the path of the reference point that occurs when moving within. In this manner, the first symbol 610 is the position where the surgical instrument first appears in the observation zone 300 when the surgical instrument continues to move along the current trajectory (or the surgical instrument is the observation zone 300). When moving away from, the trajectory is disposed within the boundary zone 400 of the position at which the surgical instrument appears when the trajectory is formed in the reverse direction. For example, if only two points are used to determine the trajectory, as the surgical instrument 138 moves from the first position of time point tl to the second position of time point t2, the path of the reference point is It is indicated by a line extending over the point. When the viewpoint t2 is the current viewpoint and the viewpoint tl is the earlier viewpoint, the present direction of the shaft 217 is displayed by the direction display part 611. By using more than two points to define the trajectory of the surgical instrument 138 that is out of view, the trajectory may take a more elaborate curve shape. The distance away from the viewing zone 300 by the surgical instrument 138 out of view may be indicated in the same manner as described above with respect to FIG. 4.

FIG. 7 illustrates, as a fourth embodiment, a GUI generation screen displayed on the monitor 104 where both surgical instruments 138 and 139 are positioned so that they are within the viewing zone 300. The end effector of 138 is blocked by some object 700. In this case, because each surgical instrument is in the viewing zone 300, each symbol 710, 420 is marked at its maximum size. Although the end effector of the surgical instrument 138 is blocked by an object 700, the object 700 such that a virtual image 711 (eg, computer model) of the end effector indicates the actual position and orientation. ) Is shown above. If the virtual image 711 is overly confused, the outline of the end effector of the surgical instrument 138 may be used instead, depending on the option programmed or selected by the surgeon.

As noted above, the symbols 420, 410, 510, 610, 710 can be non-clickable symbols or clickable icons. In the former case, when the surgeon passes the cursor of a pointing device such as a mouse over a non-clickable symbol, additional information about the surgical instrument may be provided. In the latter case, when the surgeon clicks on the clickable icon using the pointing device, additional information about the surgical instrument may be provided. In both cases, the additional information is information other than identifying the patient-side manipulator associated with the corresponding surgical instrument, and may be represented by a color or a number that is always displayed on or adjacent to the symbol. Examples of such additional information may include the type of surgical instrument and identification information of the master manipulator associated with the surgical instrument. The additional information may be provided in a separate window, such as a picture-in-picture, or as text on or adjacent to the symbol. If a separate window is provided, the additional information further includes a computer-generated screen of the surgical site that includes a view of the endoscope 140 and a computer-generated model of the surgical site that includes a computer-generated model of all surgical instruments outside of the endoscope view. can do.

Although the symbols 420, 410, 510, 610, 710 are shown in a circle, the symbols 420, 410, 510, 610, 710 may be displayed in one or more different forms. For example, if the surgical instrument is positioned to be visible inside the viewing zone 300, the symbol may take the form of a computer model of the surgical instrument shaft such that the virtual shaft is marked at the boundary zone 400. On the other hand, if the surgical instrument is positioned to be outside of the viewing zone 300, the symbol may take the form of a computer model of the distal end of the surgical instrument such that the virtual end effector is indicated at the boundary zone 400. Can be. As the surgical instrument moves from the outside of the observation zone 300 to the interior of the observation zone 300, the symbol may change constantly from the virtual end effector to the virtual shaft, and the surgical instrument may be moved from the observation zone 300. In the case of moving from the inside outward of the viewing zone 300, the symbol, on the contrary, may change constantly from the virtual shaft to the virtual end effector. The direction of the virtual shaft or the virtual end effector, if desired, preferably coincides with the direction of the actual surgical instrument. As described above for the symbol, when the surgical instrument is outside of the viewing zone 300, the size of the virtual end effector may represent a distance away from the viewing zone 300. Likewise, in order to identify the surgical instrument and / or the patient side manipulator of the surgical instrument, a virtual shaft or virtual end effector, as described for the symbol above, may optionally be indicated by a colored or numbered color.

8 illustrates, as one example, a flow chart of a method of presenting position and confirmation information of a surgical instrument on a monitor 104. This method is preferably executed for each surgical instrument by a GUI running into the processing unit 102. In step 801, the position and direction of the surgical instrument is determined within the reference frame of the imaging device, and the captured image of the imaging device is displayed on the monitor 104. Although the image is described as being captured by the stereoscopic camera of the endoscope 140 for the above example, it will be appreciated that an image captured by another imaging device using another imaging method may be used in the method. In addition, for the above example, it is assumed that the entire field of view of the camera is displayed in the observation area 300, as shown in FIG. In this case, therefore, it is not possible to determine the position and orientation of the surgical instrument using conventional imaging techniques when the surgical instrument is outside the field of view of the camera.

Thus, the position and orientation of the surgical instrument (also referred to herein as "state of the surgical instrument") receives information from a joint sensor on the robotic arm of the surgical instrument, and applies this information to kinematics of the robotic arm. It is estimated first within the frame of reference of the surgical instrument. In this case, since the state of the surgical instrument is mainly determined from the kinematics of the robotic arm, the state of the surgical instrument can be easily determined even if the surgical instrument is outside the field of view of the endoscope 140 or blocked within the field of view of the endoscope 140. have.

The state of the estimated surgical instrument is transferred to a reference frame of the camera and then corrected using a predetermined error transform. The error transform can be determined from the difference between the state of the surgical instrument determined using the kinematics of the robotic arm and the state of the surgical instrument determined using video image processing. The error transform may be determined first in a pre-operation adjustment step and then periodically updated when the surgical instrument is within the field of view of the endoscope 140 during minimally invasive surgery.

However, if only a portion of the field of view of the camera is displayed in the viewing zone 300 of the monitor 104 as shown by zone 1101 in FIG. 11, the monitor 104 as shown in zone 1102 in FIG. 11. If there is a surgical instrument in a portion of the field of view of the camera that is not indicated in the viewing zone 300, the conventional imaging technique can still be used to determine the position of the surgical instrument. In both FIG. 10 and FIG. 11, only the left camera screen Il is shown. However, for the 3-D display, the corresponding right camera screen I2 is also required when described in connection with, for example, FIG. 9, but for the sake of simplicity, the right camera screen I2 is shown in FIGS. 10 and 11. ) Is not shown.

Additional details for determining the position and orientation of the surgical instrument, particularly details for performing surgical instrument positioning, are described, for example, “Sensor Derivation Data and / or Camera Derivation During Minimally Invasive Robotic Surgery. Method and Systems for Performing 3-D Tool Tracking by Fusion of Sensor and / or Camera derived Data during Minimally Invasive Robotic Surgery, " Is described in US patent application Ser. No. 11 / 130,471, filed May 16, 2005.

In step 802, a determination is made as to whether the position of the surgical instrument is within the viewing zone 300 of the monitor 104, where the determination determines whether the surgical instrument is within the field of view of the endoscope 140. Same as This latter decision can be made using epipolar geometry. For example, referring to FIG. 9, the endoscope 140 is separated by a baseline distance “b” and has two cameras Cl and C2 having image planes Il and I2 formed at the focal length “f” of the camera. ) Is included. The image planes Il and I2 are adjusted using a conventional stereo rectification algorithm to remove the different effects of the internal camera structure and the external camera structure.

One point P in the camera reference frame is projected onto the image points Pl and P2 of the image planes Il and I2, and the image planes Il and I2 are the one point P, the camera Cl, It is an epipolar plane comprising two optical centers of C2) and the image points P1 and P2. The position of the point P is the reference point distance "b", the focal length "f", and the image point Pl, P2 from each image plane center point (ie, the intersection of the yl axis and the y2 axis and the x-axis). Can be determined within the camera frame of reference using a known value for disparity " d "

Thus, in order for the surgical instrument to be within the field of view of the endoscope 140, at least one point on the surgical instrument must be projected onto at least one of the two image planes Il and I2. Although disparity information calculated for nearby points can be used to estimate the position of a point on a surgical instrument that is projected on only one of the two imaging planes Il, I2, for example, The disparity value can be calculated for the one point and it is preferred that the one point on the surgical instrument is projected on both sides of the two image planes Il, I2 so that the degree of disparity can be determined directly. Further, although only one point of the surgical instrument may be within the field of view of the endoscope 140, the surgical instrument may be technically within the field of view of the endoscope 140, but for practical reasons, the surgeon may visually display the surgical instrument on the monitor 104. A sufficient number of points need to be within the field of view of the endoscope 140 so as to be identified.

If it is determined in step 802 that the position of the surgical instrument is outside of the viewing zone 300 of the monitor 104, then in step 803 the position of the symbol indicates the relative position of the surgical instrument relative to the viewing zone 300. The location for the symbol within the boundary area 400 surrounding 300 is determined. Examples of such determinations are described above in connection with FIGS. 4 to 6. After determining the position of the symbol within the boundary area 400, in step 804, the symbol is displayed in the boundary area 400 at the determined position. In addition, the direction indicator may be superimposed on the symbol and other surgical instruments and / or robotic arm identification information thereof described with reference to FIGS. 4-6. The method then repeats for another processing time by returning to step 801.

On the other hand, if it is determined in step 802 that the position of the surgical instrument is within the viewing zone 300 of the monitor 104, in step 805, a check is made to see if the surgical instrument is within the field of view of the endoscope 140. Referring to FIG. 12 as one example for performing this operation, in step 1201, a 3-D computer model of a surgical instrument is generated. This is usually a single proactive process. In step 1202, the 3-D computer model of the surgical instrument is positioned and oriented according to the condition of the surgical instrument determined in step 801. In operation 1203, the outline of the 3-D computer model of the surgical instrument is operated by projecting the outlines of the left and right image surfaces Il and I2 of the left and right cameras Cl and C2 of the endoscope 140. The right 2-D contour and the left 2-D contour of the computer model of the instrument are generated. In step 1204, the 2-D contour of the computer model of the surgical instrument generated in the left image plane Il in step 1203 is compared with the left camera screen captured by the left camera Cl, or in step 1203 the right image plane The 2-D contour of the computer model of the surgical instrument generated in (I2) is compared with the right camera screen captured by the right camera C2, or both are compared.

In step 806, for example, the surgical instrument determines the endoscope 140 by determining whether the mutual comparison value calculated in step 1204 matches or exceeds a threshold value for one or both of the left camera screen and the right camera screen. A determination is made as to whether it is identified in the field of view. If the result of step 806 is YES, the surgical instrument is confirmed on the right camera screen and / or left camera screen. The process then proceeds to step 803 to determine the symbol location within the boundary area 400, which can be determined simply by the intersection of the shaft of the surgical instrument and the boundary area 400. The process then proceeds to step 804 to mark the symbol at the determined location in the boundary area 400, and then the process moves to step 801 to repeat the method for another processing time.

However, if the result of step 806 is NO, then the surgical instrument is probably blocked by another object. In this case, in step 807, the left camera in which the 2-D contour of the computer model of the surgical instrument generated by projecting the 3-D computer model of the surgical instrument on the left image surface Il is captured by the left camera Cl. Overlapping on the screen, the 2-D contour of the computer model of the surgical instrument generated by projecting the 3-D computer model of the surgical instrument onto the right image plane I2 is superimposed on the right camera screen captured by the right camera C2. Lose. As a result, a 3-D outline of the computer model of the surgical instrument is displayed in the viewing zone 300 of the monitor 104 overlaid on the obstruction object. As an alternative embodiment, the surgery is performed by appropriately superimposing the left and right images of the 3-D computer model on the left and right camera screens captured by the left and right cameras Cl and C2 of the endoscope 140. The 3-D computer model of the instrument may be displayed as a virtual surgical instrument rather than just its appearance on the obstruction.

The process then proceeds to step 803 to determine the symbol location within the boundary area 400, which can be determined simply by the intersection of the shaft of the surgical instrument and the boundary area 400. The process then proceeds to step 804 to mark the symbol at the determined location in the boundary area 400, and then the process moves to step 801 to repeat the method for another processing time.

While various embodiments of the invention have been described with respect to preferred embodiments, the invention is fully protected within the scope of the appended claims.

Claims (76)

A computer-implemented method for indicating the position of a surgical instrument on a computer display screen, Determining the position of the surgical instrument; Determining the position of the symbol within the border zone around the image displayed on the computer display screen to indicate the position of the surgical instrument; And Displaying the symbol at the determined location of the border zone; And a computer-implemented method for indicating a position of a surgical instrument on a computer display screen. The computer-implemented method of claim 1, wherein the symbol provides information identifying the surgical instrument. The computer-implemented method of claim 1, wherein the symbol provides information identifying a patient side manipulator associated with the surgical instrument. 4. The computer-implemented method of claim 3, wherein the symbol is a color displayed on the patient side manipulator in relation to the patient side manipulator. 4. The computer-implemented method of claim 3, wherein the symbol is represented by a number displayed on the patient side manipulator in relation to the patient side manipulator. The method of claim 1, wherein the position of the surgical instrument is determined within a frame of reference of an imaging device providing an image displayed on the computer display screen. . 7. The computer-implemented method of claim 6, wherein the position of the surgical instrument is determined using kinematics for a patient-side manipulator that moves the surgical instrument. 7. The computer-implemented method of claim 6, wherein the position of the surgical instrument is determined using image identification techniques. 2. The surgical instrument of claim 1, wherein the surgical instrument comprises a shaft having an axis extending along the longitudinal direction, wherein determining the position of the symbol within the boundary region comprises crossing a line through the axis of the shaft with the boundary region. Determining a location, comprising: determining a location; and executing the location of the surgical instrument on the computer display screen. The method of claim 1, Determining whether the position of the surgical instrument is within a viewing zone that includes an image displayed on the computer display screen, and executing the computer to indicate the position of the surgical instrument on the computer display screen. Way. 11. The method of claim 10, wherein if the position of the surgical instrument is outside of the viewing zone, determining the position of the symbol within the boundary zone comprises a line extending from the position of the surgical instrument to a reference point in the viewing zone. Determining a location to intersect the zone, the computer-implemented method for indicating the position of the surgical instrument on a computer display screen. 11. The method of claim 10, wherein if the position of the surgical instrument is outside of the viewing zone, determining the position of the symbol within the boundary zone determines the trajectory of the surgical instrument from the current position and the past position of the surgical instrument. And determining a location at which the estimated portion by the extrapolation of the trajectory and the boundary area intersect with each other. 11. The method of claim 10, wherein if the position of the surgical instrument is outside of the viewing zone, displaying the symbol within the boundary zone determines the distance from the position of the reference point within the viewing zone to the position of the reference point on the surgical instrument. And adjusting the size of the symbol to indicate the distance. 11. The method of claim 10, wherein if the position of the surgical instrument is outside of the viewing zone, displaying the symbol within the boundary zone determines the distance from the position of the reference point within the viewing zone to the position of the reference point on the surgical instrument. And displaying the symbol such that the color of the symbol indicates the distance. 15. The computer-implemented method of claim 14, wherein the density of the collar of the symbol represents the distance. 15. The method of claim 14, wherein the position of the symbol collar within the color spectrum is indicative of the distance. 11. The method of claim 10, wherein if the position of the surgical instrument is outside of the viewing zone, displaying the symbol within the boundary zone determines the distance from the position of the reference point within the viewing zone to the position of the reference point on the surgical instrument. And displaying the symbol such that the frequency of blink of the symbol indicates the distance. 11. The method of claim 10, wherein if the position of the surgical instrument is outside of the viewing zone, displaying the symbol within the boundary zone determines the distance from the position of the reference point within the viewing zone to the position of the reference point on the surgical instrument. And displaying the symbol such that the frequency of the symbol around the determined location within the boundary zone indicates the distance. 11. The method of claim 10, wherein if the position of the surgical instrument is outside of the viewing zone, displaying the symbol within the boundary zone determines the distance from the position of the reference point within the viewing zone to the position of the reference point on the surgical instrument. And displaying the symbol to indicate the distance by superimposing a distance value over the symbol. 11. The method of claim 10, wherein if the position of the surgical instrument is outside of the viewing zone, determining the position of the surgical instrument comprises determining a direction of an end effector of the surgical instrument, wherein the symbol within the boundary zone. Displaying the direction comprises displaying a direction indicator over the symbol such that a direction indicator is oriented so as to indicate the direction of the end effector. Way. 11. The method of claim 10, wherein the surgical instrument includes a shaft having an axis extending along the longitudinal direction, and if the position of the surgical instrument is outside of the viewing zone, determining the position of the surgical instrument is in the direction of the axis. Determining the direction of the symbol on the computer display screen, wherein the displaying of the symbol within the boundary zone comprises displaying a direction indicator over the symbol such that the direction indicator is oriented to indicate the direction of the axis. Computer-implemented method for indicating the position of a surgical instrument. 11. The method of claim 10, wherein if the position of the surgical instrument is outside of the viewing zone, determining the position of the surgical instrument comprises determining a direction of an end effector of the surgical instrument, wherein the symbol within the boundary zone. And displaying the symbol so that the symbol is directed such that the symbol can indicate the direction of the end effector. 11. The method of claim 10, further comprising the step of identifying an image of the surgical instrument within the observation zone if the surgical instrument is in the observation zone. How to run a computer. 24. The method of claim 23, wherein if the image of the surgical instrument cannot be identified within the observation zone, the virtual surgical instrument is displayed on the computer display screen in the observation zone of the place where the surgical instrument is to be located. A computer-implemented method for indicating the position of the instrument. The method of claim 23, wherein the observation zone includes a right two-dimensional screen and a left two-dimensional screen, and the checking of the image of the surgical instrument comprises: Generating a three-dimensional computer model of a surgical instrument; Positioning and orientation of the computer model to match the current position and orientation of the surgical instrument within the frame of reference of the computer display screen; Generating a two-dimensional outline of the computer model projected onto a selected one of the right two-dimensional screen and the left two-dimensional screen; And Comparing the two-dimensional contour of the computer model with one selected from the right two-dimensional screen and the left two-dimensional screen; And a computer-implemented method for indicating a position of a surgical instrument on a computer display screen. The apparatus of claim 24, wherein the image of the surgical instrument is displayed on the computer display screen when the image of the surgical instrument is not identified on the selected one of the right two-dimensional screen and the left two-dimensional screen. Indicating the position of the surgical instrument on the computer display screen, further comprising displaying at least a two-dimensional outline of the computer model of the surgical instrument to be displayed as an overlay on a selected one of the left two-dimensional screens. How to run a computer. 25. The method according to claim 24, wherein if the image of the surgical instrument is not identified on the selected one of the right two-dimensional screen and the left two-dimensional screen, the surgical instrument may be moved in the direction and position of the surgical instrument within the reference frame of the computer display screen. And overlapping the right 2D virtual image and the left 2D virtual image of the surgical instrument with a corresponding one of the right 2D screen and the left 2D screen to display a 3D virtual image. Computer-implemented method for indicating a position of a surgical instrument on a computer display screen. An apparatus for indicating the position of a surgical instrument on a computer display screen, A processor, the processor being coupled to the computer display screen, Determine a current position of the surgical instrument; Determine the position of the symbol within the border zone around the image displayed on the computer display screen to indicate the position of the surgical instrument; And And wherein the symbol is configured to be displayed at a determined location within the border zone. 29. The apparatus of claim 28, wherein the symbol provides information identifying the surgical instrument. 29. The apparatus of claim 28, wherein the symbol provides information identifying the patient side manipulator associated with the surgical instrument. 31. The apparatus of claim 30, wherein the symbol is a color marked on the patient side manipulator in relation to the patient side manipulator. 31. The apparatus of claim 30, wherein the symbol is represented by a number displayed on the patient side manipulator in relation to the patient side manipulator. 29. The apparatus of claim 28, wherein the position of the surgical instrument is determined within a frame of reference of the imaging device providing an image displayed on the computer display screen. 34. The apparatus of claim 33, wherein the position of the surgical instrument is determined using kinematics for a patient-side manipulator that moves the surgical instrument. 34. The apparatus of claim 33, wherein the position of the surgical instrument is determined using image identification techniques. 29. The apparatus of claim 28, wherein the surgical instrument includes a shaft having an axis extending along the longitudinal direction and wherein the processor determines the location where a line passing through the axis of the shaft intersects the boundary zone. A device for indicating the position of a surgical instrument on a computer display screen, the position of the symbol being determined. 29. The surgical instrument of claim 28, wherein the processor is configured to determine whether the position of the surgical instrument is within a viewing zone comprising an image displayed on the computer display screen. Device for indicating location. 38. The system of claim 37, wherein if the position of the surgical instrument is outside of the viewing zone, the processor determines the location where a line extending from the position of the surgical instrument to a reference point within the viewing zone crosses the boundary zone. And determine the location of the symbol within the zone. 38. The apparatus of claim 37, wherein if the position of the surgical instrument is outside of the viewing zone, the processor determines the trajectory of the surgical instrument from the current position and the past position of the surgical instrument, and estimates by extrapolation of the trajectory. And determine the location of the symbol within the border zone by determining where the portion and the border zone intersect. 38. The apparatus of claim 37, wherein if the position of the surgical instrument is outside of the viewing zone, the processor determines a distance from the position of the reference point within the viewing zone to the position of the reference point on the surgical instrument, the symbol to indicate the distance. And display the symbol within the boundary area by adjusting the size of the device. 38. The apparatus of claim 37, wherein if the position of the surgical instrument is outside of the viewing zone, the processor determines the distance from the position of the reference point in the viewing zone to the position of the reference point on the surgical instrument, wherein the collar of the symbol is the distance. And display the symbol within the boundary area by displaying the symbol to indicate the position of the surgical instrument on the computer display screen. 42. The apparatus of claim 41, wherein the density of the collar of the symbol represents the distance. 42. The apparatus of claim 41 wherein the position of the symbol collar within the color spectrum is indicative of the distance. 38. The apparatus of claim 37, wherein if the position of the surgical instrument is outside of the viewing zone, the processor displays the symbol within the boundary zone by determining the distance from the position of the reference point within the viewing zone to the position of the reference point on the surgical instrument. And the frequency of blinking of the symbol is indicative of the distance. 38. The apparatus of claim 37, wherein if the position of the surgical instrument is outside of the viewing zone, the processor displays the symbol within the boundary zone by determining the distance from the position of the reference point within the viewing zone to the position of the reference point on the surgical instrument. And the frequency of the symbol around the location determined within the boundary zone indicates the distance. 38. The apparatus of claim 37, wherein if the position of the surgical instrument is outside of the viewing zone, the processor displays the symbol within the boundary zone by determining the distance from the position of the reference point within the viewing zone to the position of the reference point on the surgical instrument. And display the distance by superimposing a distance value on a symbol. 38. The apparatus of claim 37, wherein if the position of the surgical instrument is outside of the viewing zone, the processor determines the direction of the end effector of the surgical instrument and directs a direction indicator to indicate the direction of the end effector. And display a direction indicator on the computer display screen. 38. The apparatus of claim 37, wherein the surgical instrument includes a shaft having an axis extending along the longitudinal direction, and if the position of the surgical instrument is outside of the viewing zone, the processor determines the direction of the axis, and the direction indicator is And display a direction indicator over the symbol to direct the direction of the axis. 38. The apparatus of claim 37, wherein if the position of the surgical instrument is outside of the viewing zone, the processor determines the position and orientation of the end effector of the surgical instrument and directs a symbol to indicate the direction of the end effector. A device for indicating the position of a surgical instrument on a computer display screen, the apparatus being configured to display a symbol. 38. The computer display of claim 37, wherein if the surgical instrument is in an observation zone displayed on a computer display screen, the processor is configured to view an image of the surgical instrument within the observation zone. Device for indicating the position of the surgical instrument. 51. The method of claim 50, wherein if the image of the surgical instrument cannot be identified within the observation zone, the processor is configured to display the virtual surgical instrument in the observation zone of the location where the surgical instrument is to be located. A device for indicating the position of a surgical instrument on a computer display screen. 51. The system of claim 50, wherein the viewing zone comprises a right two-dimensional screen and a left two-dimensional screen, wherein the processor includes: Generating a three-dimensional computer model of a surgical instrument; Positioning and orientation of the computer model to match the current position and orientation of the surgical instrument within the frame of reference of the computer display screen; Generating a two-dimensional outline of the computer model projected onto a selected one of the right two-dimensional screen and the left two-dimensional screen; And Comparing the two-dimensional contour of the computer model with one selected from the right two-dimensional screen and the left two-dimensional screen; And to identify an image of the surgical instrument by means of an image of the surgical instrument on a computer display screen. 53. The apparatus of claim 52, wherein if the image of the surgical instrument is not identified on the selected one of the right two-dimensional screen and the left two-dimensional screen, the processor is configured to display at least the surgery when the image of the surgical instrument is displayed on a computer display screen. And to display the two-dimensional contour of the computer model of the instrument as an overlay on a selected one of the right two-dimensional screen and the left two-dimensional screen. 53. The method of claim 52, wherein if the image of the surgical instrument is not identified in the selected one of the right two-dimensional screen and the left two-dimensional screen, the surgical instrument in the direction and position of the surgical instrument in the reference frame of the computer display screen In order to present a three-dimensional virtual image, the processor is configured to overlap the right two-dimensional virtual image and the left two-dimensional virtual image of the surgical instrument with a corresponding one of the right two-dimensional screen and the left two-dimensional screen. A device for indicating the position of a surgical instrument on a computer display screen. As a medical robotic system, Surgical instruments; A robot arm mechanically coupled to the surgical instrument to adjust the position and direction of the surgical instrument; An imaging device; Computer display screens; And A processor coupled to the robotic arm, an imaging device, and a computer display screen, The processor is configured to display an image captured by the imaging device in a viewing area of a computer display screen and to display a symbol in a boundary area around the viewing area to indicate the position of the surgical instrument. Medical robotic system. 56. The medical robotic system of claim 55, wherein the symbol provides information identifying the surgical instrument. 56. The medical robotic system of claim 55, wherein the symbol provides information identifying a patient side manipulator associated with the surgical instrument. 59. The medical robotic system of claim 57, wherein the symbol is a color marked on the patient side manipulator in relation to the patient side manipulator. 58. The medical robotic system of claim 57, wherein the symbol is represented by a number displayed on the patient side manipulator in relation to the patient side manipulator. 56. The medical robotic system of claim 55, wherein the position of the surgical instrument is determined within a frame of reference of the imaging device. 61. The medical robotic system of claim 60, wherein the position of the surgical instrument is determined using kinematics for a patient side manipulator that moves the surgical instrument. 61. The medical robotic system of claim 60, wherein the position of the surgical instrument is determined using image identification technology. 56. The apparatus of claim 55, wherein the surgical instrument includes a shaft having an axis extending along the longitudinal direction, and wherein the processor determines the location where a line passing through the axis of the shaft intersects the boundary zone. A medical robotic system, configured to determine the position of a symbol. 56. The medical robotic system of claim 55, wherein the processor is configured to determine whether the position of the surgical instrument is within a viewing zone displayed on a computer display screen. 56. The computer readable medium of claim 55, wherein the processor is configured to cause the symbol to be displayed on a boundary area of a computer display screen to indicate the direction of the end effector of the surgical instrument when the surgical instrument is outside of the viewing zone. Medical robot system. 56. The boundary zone of claim 55, wherein the processor is configured to indicate a distance from the position of the reference point on the surgical instrument to the position of the reference point of the viewing zone when the surgical instrument is outside of the viewing zone. And the symbol is displayed on the medical robot system. 67. The medical robotic system of claim 66, wherein the distance is represented by the size of the indicated symbol. 67. The medical robotic system of claim 66, wherein the distance is represented by a collar of the indicated symbol. 67. The medical robotic system of claim 66, wherein the distance is represented by the brightness of the displayed symbol. 67. The medical robotic system of claim 66, wherein the distance is indicated by the frequency of blinking of the indicated symbol. 70. The medical robotic system of claim 70, wherein the frequency of blinking of the symbol increases as the distance decreases. 67. The medical robotic system according to claim 66, wherein the frequency of the symbol around the determined position in the boundary zone indicates the distance. 67. The medical robotic system of claim 66, wherein the distance is indicated by superimposing a distance value over a symbol. 56. The medical robotic system of claim 55, wherein the imaging device comprises at least one camera. 56. The medical robotic system of claim 55, wherein the imaging device captures MRI images. 56. The medical robotic system of claim 55, wherein the imaging device captures ultrasound images.

Images