KR20010027541A - Apparatus and Method for viewing optimal three dimentional image of surgical instrument for automatic controlling the position of the endoscope in Minimally Invasive Surgery - Google Patents

Abstract

PURPOSE: An apparatus and method for viewing optimal three-dimensional image information of a surgical instrument for automatically controlling the position of an endoscope are provided to accurately perform confirmation of a surgical instrument and recognition of the position of the instrument with only one image. CONSTITUTION: An apparatus for viewing optimal three-dimensional image information of a surgical instrument for automatically controlling the position of an endoscope finds out the kind of a surgical instrument from an endoscope image using cross-ratio which does not vary even under projective transformation that is the characteristic of a camera, and calculates the angle and distance of the surgical instrument from the projective image of the conic curve of a ring mark shown on the cylindrical surface of the surgical instrument to detect the position of the instrument.

Description

Apparatus and Method for viewing optimal three dimentional image of surgical instrument for automatic controlling the position of the endoscope in Minimally Invasive Surgery}

In the present invention, in performing a minimally invasive surgery, an endoscope position for viewing a surgical instrument inserted into the abdomen in an optimal stereoscopic image using an endoscope. Apparatus and Method for viewing optimal three dimentional image of surgical instrument for automatic controlling the position of the endoscope.

1 is a view showing a schematic configuration of a microsurgical system using a general robot. As shown, a microsurgical system using a robot is basically a surgical tool (1) inserted into the patient's body, an endoscope (2) for the doctor to see the surgical tool, a type of surgical tool (1) And robot arm (3), robot arm controller (4), computer (5), image processor (6), force / torque processor (7), and voice processor to automatically adjust the position of endoscope (2) to view position. (8), the monitor 9, and the like.

In the microsurgical system configured as described above, in order to view the image at the desired position from the image at the current position of the surgical tool 1 with the monitor 9, the posture of the robot arm 3 must be adjusted as shown in FIG. do.

In order to obtain an image of an optimal surgical tool in such a microscopic surgery using a conventional endoscope, Wang et al. Of Computer Motion Inc., as shown in Figure 3a, color square mark (Color) mark on the surgical instrument (Instrument) -Coded Mark) was attached. Each color was classified into red (R), green (G), and blue (B), and the LUT was built in advance, and used for tool recognition, and as shown in FIG. Three-dimensional image information was extracted from the deformation that occurs. However, due to the characteristics of the camera, three-dimensional image information cannot be accurately extracted from the deformation of the rectangular mark image as shown in FIG. 3B. That is, the location of the distorted image of the color mark is checked to confirm the position, but the accuracy of the position exists because the position is inaccurate.

In addition, Aerospace of Germany is similar to Computer Motion Inc., but proposed a method to reduce the recognition time by transforming the color to HVS.

As shown in FIG. 4A, IBM proposes a method of installing a light-emitting circular mark such as an optical fiber in a surgical instrument and extracting three-dimensional image information from the circular deformation form. It was. It is possible to check the surgical instrument with an injected optical display mark such as an optical fiber, but since a circular is installed in a cylindrical surgical instrument, an error as shown in FIG. 4B may occur.

As such, it is impossible to distinguish between the tools using a single endoscope image of the surgical tools as described above, and at least two images are required for the identification and location recognition of the surgical tools.

The present invention has been made to improve the above problems, and an optimal 3D image information recognition apparatus and method of the surgical tool for automatic adjustment of the endoscope position that can accurately confirm the position and recognition of the surgical tool even with a single image The purpose is to provide.

1 is a view showing a schematic configuration of a microsurgical system using a general robot,

FIG. 2 is a view illustrating a process of automatically obtaining an image (optimal image) of a desired posture and position in the microsurgical system of FIG. 1;

3A and 3B are views showing a color code mark and an image thereof displayed on a surgical tool for recognition of a surgical tool in a conventional microsurgical system, respectively;

4A and 4B are views showing original marks and images of the surgical tools for recognizing surgical tools in another conventional microsurgical system, respectively;

5 is a view showing a ring mark displayed on a surgical tool for the recognition of the surgical instrument in the optimal three-dimensional image information recognition apparatus of the surgical tool for automatic endoscope position adjustment according to the present invention,

6 is a view showing the arrangement of the points for obtaining the constant intersection ratio in the ring mark of FIG.

7 is a view showing a conical curve image of the ring mark of FIG. 5 with an endoscope;

FIG. 8 is a view for explaining a process of extracting three-dimensional information of a circle from a cone curve extracted in FIG. 5;

FIG. 9 is a view illustrating an arrangement relationship between an surgical tool and an endoscope and an image thereof for determining a three-dimensional posture of a surgical tool in the apparatus for recognizing 3D image information of FIG. 5;

FIG. 10 is a view showing a schematic configuration of a microsurgical system using a robot equipped with an optimal 3D image information recognition device of a surgical tool for automatically adjusting the endoscope position according to the present invention.

<Description of the code | symbol about the principal part of drawing>

1. surgical instruments 2. endoscope

3. robot arm 4. robot arm controller

5. Computer 6. Image Processor

7. Power / Torque Processor 8. Voice Processor

9. Monitor

In order to achieve the above object, the optimum three-dimensional image information recognition apparatus of the surgical tool for automatic endoscope position adjustment according to the present invention, the robot arm; An endoscopic surgical tool held by the robotic arm; An endoscope robotic surgical device comprising: a controller in electrical communication with the robotic arm, the movement of which is represented by a proportional movement of the robotic arm and a surgical instrument, comprising: four parallel borders to the cylindrical surface of the surgical instrument; Two ring marks marked to have; Intersections with the four borders of the ring mark and any straight line that intersects the borders at right angles are called X ₁ , X ₂ , X ₃ , X ₄ , respectively, and the points in the image of the surgical tool corresponding to these points. Where x ₁ , x ₂ , x ₃ , and x ₄ , respectively, end The above equation using the point that satisfies the relationship of A surgical tool type reading unit for calculating a ratio by which a surgical instrument is read and a specific surgical tool; And a position reading unit for extracting 3D image information from a cone curve of the surgical tool image to read a distance and an angle of the surgical tool.

In the present invention, k is any non-zero constant, and the radius of the circle projected on the cone curve extracted from the cone curve of the surgical tool image is r, and U is [U ₁ U ₂ U ₃ ]. A matrix consisting of eigenvectors for the eigenvalues λ ₁ , λ ₂ , λ ₃ of the displayed cone, R is called the rotation transformation matrix about the x 'axis, and n is the normal vector of the plane on which the circle is placed. d is called the normal distance, ct is the vector to the center of the circle, and When, the normal vector, the normal distance and the vector to the center of the circle are respectively represented by the equation n = URn ', , ct = URct ', and the position reading unit preferably reads the position of the surgical tool by calculating a distance and an angle of the surgical tool using the equations.

In addition, in order to achieve the above object, an optimal method for recognizing 3D image information of a surgical tool for automatic endoscope position adjustment according to the present invention includes: a robot arm; An endoscopic surgical tool held by the robotic arm; An endoscope robotic surgical device comprising: a controller in electrical communication with the robotic arm, the movement of which is represented by a proportional movement of the robotic arm and a surgical instrument, comprising: four parallel borders to the cylindrical surface of the surgical instrument; Mark two ring marks, and the intersections with the four boundary lines of the ring mark and any straight line that intersects the boundary lines at right angles are X ₁ , X ₂ , X ₃ , and X ₄ , respectively. When the points in the image of the surgical tool corresponding to x ₁ , x ₂ , x ₃ and x ₄ , respectively, the ratio of the points end The above equation using the point that satisfies the relationship of Calculating a ratio by means of reading the specific surgical tool; And extracting three-dimensional image information from a cone curve of the surgical tool image, calculating a distance and an angle of the surgical tool, and reading the position of the surgical tool.

In the present invention, k is any non-zero constant, and the radius of the circle projected on the cone curve extracted from the cone curve of the surgical tool image is r, and U is [U ₁ U ₂ U ₃ ]. A matrix consisting of eigenvectors for the eigenvalues λ ₁ , λ ₂ , λ ₃ of the displayed cone, R is called the rotation transformation matrix about the x 'axis, and n is the normal vector of the plane on which the circle is placed. d is called the normal distance, ct is the vector to the center of the circle, and When, the normal vector, the normal distance and the vector to the center of the circle are respectively represented by the equation n = URn ', , ct = URct ', and the step of reading the surgical tool position is preferably performed by calculating the distance and angle of the surgical tool using the equations.

Hereinafter, an apparatus and method for recognizing an optimal 3D image information of a surgical tool for automatically adjusting an endoscope position according to the present invention will be described in detail with reference to the accompanying drawings.

In the surgical tool optimum three-dimensional image information recognition device of the present invention, to mark the surgical instrument recognition and three-dimensional information to the cylindrical surgical instrument (Instrument), as shown in Fig. have. Recognition of the surgical instrument is made by a cross-ratio defined by the distance between these four ring patterns, as shown in FIG. 6, and a three-dimensional image of the recognized instrument. The information is calculated from the shape of the ellipse formed by the ring pattern, as shown in FIG.

In other words, the present invention reads the type of surgical instrument from an endoscope image by using a constant cross-ratio even under projection deformation, which is a characteristic of a camera, and projects the cone curve of a ring mark displayed on a cylindrical surface of the surgical instrument. The position is read by calculating the angle and distance of the surgical instrument from the image.

Looking at the operating principle of the optimum three-dimensional image information recognition device of the surgical instrument for automatic adjustment of the endoscope position having the above characteristics as follows.

The method proposed in the present invention is based on the basic theory of cross-ratio and conical geometry as shown in FIG. Cross-Ratio is a quantity in which the ratio of the projective transformation, that is, the object-to-image transformation, is always invariant and is defined as follows.

As shown in Figs. 5 and 6, when (X1, X2, X3, X4) and (x1, x2, x3, x4) are four points on a straight line defined in the object plane and the image plane, respectively, (cross-ratio) is defined as in Equation 1 below.

In other words, under projection deformation, the distance between two points and the like is variable, but this ratio is an invariant value and is used as a discriminator for recognition.

In addition, a general conic image (Conic) including a garden and an ellipse may be represented as shown in the following equation (2).

Ax ² + 2Bxy + Cy ² + 2 (Dx + Ey) + F = 0

The equation projected on the image can be written as Equation 3 below.

Where k is an arbitrary non-zero constant. If the cone curve rotates about an axis, the rotated cone curve is expressed by the following equation.

Q '= k'R ^T QR

FIG. 8 shows a process of coordinate transformation for obtaining three-dimensional information about a circle having a radius r projected on an image as shown in FIG. 9 with the representation of a conic curve (Conic). That is, a process of extracting 3D image information of a circle shape from the cone shape extracted from the image is shown.

In FIG. 8 and FIG. 9, the equation for the conic curve in each process is shown in Equation 5 below.

In addition, the relationship between each conic curve (Conic) is the following equations (6) and (7).

Q '= k'U ^T QU

Q "= k" U ^T Q'U

Where U = [U ₁ U ₂ U ₃ ] is a metric consisting of eigenvectors for the conic eigenvalue, λ ₁ , λ ₂ , λ ₃ , where R is the rotation transformation matrix for the x 'axis ( Rotation Transformation matrix.

From the above relations, the normal vector of the plane on which the circle is placed, the vector to the center of the circle, and the normal distance are given by Equations 8, 9, 10 and same.

n = URn '

ct = URct '

here, to be.

Using this equation, the position of the surgical tool is read by calculating the distance and angle of the surgical tool.

The microsurgical system having an optimal 3D image information recognition apparatus of a surgical tool for automatically adjusting the endoscope position according to the present invention configured as described above has a microphone, a voice processor, and a scaffold control signal in a conventional system as shown in FIG. 10. It has a structure without back.

As described above, the optimal three-dimensional image information recognition apparatus of the surgical tool for automatic endoscope position adjustment according to the present invention is a type of surgical tool by using an invariant cross-ratio even under the projection deformation that is a characteristic of the camera Is read from the endoscope image and its position is calculated by calculating the angle and distance of the surgical tool from the projection image of the cone curve of the ring mark displayed on the cylindrical surface of the surgical tool. Therefore, even in the case of projection deformation, the invariable cross- ratio and the circle's projective geometry are used, so it is possible to determine the location of the surgical tool only by comparing two or more pictures. On the other hand, it is possible not only to accurately locate the surgical tool with a single cut of image, but also to accurately identify the type of surgical tool that was impossible to read by the conventional method. It has the advantage of proceeding the surgery without limiting the location.

Claims (4)

Robotic arm; An endoscopic surgical tool held by the robotic arm; In the robotic surgical instrument using an endoscope comprising: a controller in electrical communication with the robot arm, the movement is represented by a proportional movement of the robot arm and surgical instruments, Two ring marks marked to have four parallel borders on the cylindrical surface of the surgical instrument; Intersections with the four borders of the ring mark and any straight line that intersects the borders at right angles are called X ₁ , X ₂ , X ₃ , X ₄ , respectively, and the points in the image of the surgical tool corresponding to these points. Let's say x ₁ , x ₂ , x ₃ and x ₄ respectively, The intersection ratio of the points end Equation using the point that satisfies the relationship of A surgical tool type reading unit for calculating a ratio by which a surgical instrument is read and a specific surgical tool; And A position reading unit for extracting 3D image information from a cone curve of the surgical tool image to read a distance and an angle of the surgical tool; Optimal three-dimensional image information recognition device of the endoscope surgical instrument, characterized in that provided. The method of claim 1, Let k be any random non-zero constant, the radius of the circle projected on the cone curve extracted from the cone curve of the surgical tool image, r, and U be the eigen of the cone represented by [U ₁ U ₂ U ₃ ]. A matrix consisting of eigenvectors for the values λ ₁ , λ ₂ , λ ₃ , where R is the rotation transformation matrix about the x 'axis, n is the normal vector of the plane on which the circle lies, and d is the normal distance. Ct is the vector to the center of the circle, and when, The normal vector, the normal distance, and the vectors to the center of the circle are each n = URn ', ct = URct ' Displayed as And the position reading unit reads the position of the surgical tool by calculating a distance and an angle of the surgical tool by using the equations. Robotic arm; An endoscopic surgical tool held by the robotic arm; In the robotic surgical instrument using an endoscope comprising: a controller in electrical communication with the robot arm, the movement is represented by a proportional movement of the robot arm and surgical instruments, Mark two ring marks to have four parallel boundaries on the cylindrical surface of the surgical instrument, Intersections with the four borders of the ring mark and any straight line that intersects the borders at right angles are called X ₁ , X ₂ , X ₃ , X ₄ , respectively, and the points in the image of the surgical tool corresponding to these points. Let's say x ₁ , x ₂ , x ₃ and x ₄ respectively, The intersection ratio of the points end Equation using the point that satisfies the relationship of Calculating a ratio by means of reading the specific surgical tool; And Extracting the 3D image information from the cone curve of the surgical tool image and calculating the distance and angle of the surgical tool to read the position of the surgical tool; Optimal three-dimensional image information recognition method of the endoscope surgical instrument comprising a. The method of claim 3, Let k be any random non-zero constant, the radius of the circle projected on the cone curve extracted from the cone curve of the surgical tool image, r, and U be the eigen of the cone represented by [U ₁ U ₂ U ₃ ]. A matrix consisting of eigenvectors for the values λ ₁ , λ ₂ , λ ₃ , where R is the rotation transformation matrix about the x 'axis, n is the normal vector of the plane on which the circle lies, and d is the normal distance. Ct is the vector to the center of the circle, and when, The normal vector, the normal distance, and the vectors to the center of the circle are each n = URn ', ct = URct ' Displayed as The step of reading the surgical instrument position is the optimal three-dimensional image information recognition method of the endoscope surgical tool, characterized in that made by calculating the distance and angle of the surgical instrument using the equations.