TW201818347A - The method of complete endoscopic MIS instrument 3D position estimation using a single 2D image - Google Patents

Abstract

This invention is about the method of endoscopic MIS instrument tracking based 3D complete eight quadrants position estimation and using 2D image. It can extends the original formular for one particular pose to formulars for any pose. It can select the colors of the two rings as well as the RGB thresholds for fast and accurate ring shape extraction from the 2D image by analyzing a large amount of laparoscopic images. And, it can propose an algorithm with a 2D laparoscopic image as the input and the corresponding six 3D pose parameters as the output., and the six 3D pose parameters of a real rod-shaped body is estimated by the proposed system and transmitted to drive its 3D model in the remote. Thus, compared to other existing MIS pose estimation methods, the proposed Double-Ring marker based algorithm of this invention is accurate and computationally very efficient.

Description

Endoscopic surgical instrument tracking method based on two-dimensional image and three-dimensional complete eight-image limited position

The present invention relates to an endoscopic surgical instrument tracking method based on two-dimensional images and three-dimensional complete eight-image limited position, which can be used in actual surgery, and combined with 3D augmented reality (AR) in a remote surgical environment. Application.

At present, minimally invasive surgery (MIS) has become an increasingly popular operation in recent decades, and minimally invasive surgery has greatly reduced the pain of patients and shortened the recovery time, making minimally invasive surgery widely accepted by patients. . Traditionally, MIS, at least one assistant controls the laparoscopic camera, and a surgeon, controlling the laparoscopic instrument for surgery is necessary. In order to help the surgeon focus on the surgery, the assistant needs to ensure that the tip of the surgical instrument is located in the center of the camera view. With the help of the assistant, the surgeon can directly observe the 2D camera view to operate the tool, but for an assistant It is difficult to maintain a stable camera for a period of time, and because the 2D camera view lacks depth information, a surgeon should be trained enough to perform surgery, so a system can automatically track the instrument and reconstruct its orientation and position in minimally invasive surgery. The process of (MIS) has become more important.

According to the current method of computer vision system to obtain depth information in two-dimensional images, the same object is used to estimate the position of two or more cameras. In the patent application No. 104143279 of the National Patent Application No. 104143279, the method for three-dimensional positioning of a two-dimensional image of a marked round rod object and the method for positioning and tracking the endoscope device, the uniform round rod shape has been proposed. Objects (such as endoscopic surgical instruments) can be photographed by a single lens camera by applying two identical shapes on the two positions of the rod, but the circular marks of different colors, using the relevant technology of modern digital image processing, Effectively detect all the 2D information of the marker and the round object in the image plane, and quickly find the 3D positioning information of the round object through the geometric relationship of the lens mapping at the imaging position of the camera. There are seven 3D positioning parameters for this round rod object. The first three parameters are the 3D coordinate values, and finally the three angles of the In-plane angle and the Out-plane angle. The patent application also proposes marking the ring on the rod-shaped object, and estimates the angle of rotation of the shaft in a binary code. These seven parameters determine the 3D pose of the unique round bar object.

However, the aforementioned patent application uses a uniform round rod-like object (such as an endoscopic surgical instrument) in a single lens environment, by which two identical shapes are respectively attached to the two positions on the rod, but the rings of different colors are used. Mark to quickly calculate the 3D parameters of the object. However, its derivation formula for deducing a single pose is not enough to estimate the arbitrary posture of the round rod. It is not performed in the context of real MIS minimally invasive surgery. There is room for improvement in accuracy and no measurement error is analyzed.

In view of the technical problem of the above-mentioned patent application No. 104143279, the present inventors have exhaustedly developed an endoscopic surgical instrument tracking method based on two-dimensional images and three-dimensional complete eight-image limited position, so the main purpose of the present invention is to provide A fast and accurate endoscopic surgical instrument tracking method; a secondary objective of the present invention is to provide a complete contour that can be identified in a complex organ background.

In order to achieve the above object, the present invention utilizes the following technical means: an endoscopic surgical instrument tracking method using a three-dimensional complete eight-image limited position, comprising: a double-loop contour removal step, which is to input the projected surgical instrument onto the plane The image captures a planar image of the first ring body (A) and the second ring body (B); an image coordinate conversion step sets the center of the first ring body and the second ring body plane image as an origin, and Performing coordinate conversion; a step of finding the center of gravity of the double ring, respectively calculating the coordinates of the center of gravity of the first ring body and the second ring body; and obtaining the two-dimensional two-dimensional information step, the center of gravity of the first ring body and the second ring The coordinate line forms a straight line equation (L) of the axis segment and passes through the edge point to calculate the two center endpoint coordinates A1, A2 of the first ring body and the two center endpoint coordinates B1, B2 of the second ring body; The pixel conversion millimeter unit step converts the aforementioned four endpoint coordinates A1, A2, B1, and B2 into millimeter units; and performs a three-dimensional attitude estimation step using the endpoint coordinates A1, A2, and B1 of the last step. B2 and known camera coke (Λ), the ring shaft length (L), the center of gravity bicyclic axial length (L _AB), bicyclic ends projection points _{(x a1, x a2, x} b1, x b2) and _{(y a1, y a2, y} b1, y The parameters of _b2 ), together with the three-dimensional eight-image limit system, determine the parameters of the three-dimensional positioning parameters {X _A1 , Y _A1 , Z _A1 , α, β, γ} of the surgical instrument, and then quickly calculate Surgical instruments in any position in space.

In the double loop profile extraction step, the first ring body is defined as (Ra, Ga, Ba) and the second ring body is (Rb, Gb, Bb), and the first ring body is used by using the concept of channel difference. Each pixel in the second ring image is subtracted from the absolute value of the three channels, and the fixed threshold is set in the three channels after subtraction, and the following mathematical formula is defined: [Math 1] Where the threshold value is set primarily for the tolerance of the light source for each color, assuming that the above conditions are true, the pixels belonging to the ring body.

The invention can achieve the following effects by the above technical means: 1. The invention derives the two-dimensional double-loop profile from the three-dimensional instrument attitude six-parameter relationship from the original three-dimensional first quadrant to the complete eight-quadrant, so For other 3D instrument attitude positioning parameter estimation algorithms, the tagged algorithm proposed by the present invention is quite fast, accurate and simple. 2. The invention is applied to the double-loop color selection of the MIS minimally invasive surgery environment, and performs color model analysis on a large number of complex organ background images, determines the optimal color of the double loop, and analyzes the colors that can be easily recognized in the laparoscopic environment. And for the illumination of the surgical instrument caused by light illumination, the ideal RGB threshold value is selected, and the double loop contour on the surgical instrument can be completely recognized in the actual operation.

Eight-quadrant extension derivation of the three-dimensional positioning system of the two-dimensional image of the marked round rod-shaped object: The present invention proposes the "marked round rod-like object" proposed by the laboratory in the laboratory of Yu Mingjun (the patent application No. 104143279) Based on the three-dimensional image positioning system and the endoscopic device tracking application, the patent application can be attached to the rod-shaped object by two ring marks of the same shape and different colors at two positions. The 3D attitude of the round rod object is derived from the geometric relationship between the two markers through the imaging position of the single camera sensor, but currently only a single attitude derivation formula of the algorithm is not sufficient to estimate the arbitrary posture of the round rod, and There is still room for improvement in accuracy. Therefore, the object of the present invention is to promote the 3D arbitrary posture formula and accuracy of the round rod-like object.

The derivation principle of the 3D six parameters of the rod object:

As shown in Figure 1, it is assumed that the 3D coordinate system of the origin is (X, Y, Z) overlapping with the 2D coordinate system (x, y) of the camera film, X and x are vertical axes, and Y and y are horizontal axes. The Z axis is the vertical axis. At the front end of both ends of the rod-shaped object, a first ring body (hereinafter referred to as an A ring) and a second ring body (hereinafter referred to as a B ring) having the same width L are coated or marked, and the predetermined distance between the two rings is L _AB . In order to accurately and conveniently identify the ring A and the ring B in the 2D image, it is recommended to use two different colors that can be easily distinguished from the use situation, and respectively assign the ring A and the ring B.

The 3D attitude of the round rod is defined as six parameters (X _AA1 , Y _AA1 , Z _{AA1 and} α, β, γ), wherein (X _AA1 , Y _AA1 , Z _AA1 ) 3D coordinates are set as reference point AA1, As shown in Fig. 2, point AA1 (BB1) is the plane focus of the axis of the rod and the upper edge of ring A (ring B), so the axis of the rod can be represented as a direction, making him a line vector . However, at reference point AA1, he is on the axis of the rod-like object, it cannot be seen, and cannot be detected from the two-dimensional image. The coordinates of the upper edge point A1 (X _A1 , Y _A1 , Z _A1 ) on the surface of the ring A are as shown in Fig. 2 to estimate the previously estimated reference point AA1 (X _AA1 , Y _AA1 , Z _AA1 ) 3D coordinates, edge points The A1 (X _A1 , Y _A1 , Z _A1 ) coordinates can be detected in the 2D image.

Line vector Is the surface and axis vector that intersects the rod The trajectory projected onto the image plane, as shown in Figure 2, is assumed to be parallel to the axis, assuming that the straight line of the rod is uniform. It should be noted that point A1 is at The intersection of the vector and the upper edge of ring A is shown in Figure 2.

The three angles of the round rod are α, β, γ, as defined in detail in Fig. 1 as follows: (1) The α angle is the X axis and the plane vector projected on the xy (XY) image The angle of the. The Z-axis rotation around the alpha angle is also known as the In-Plan angle. (2) β angle is the X axis and is projected on the X(x)-Z plane line vector The angle of the β, the angle of rotation around the Y axis, also known as the out-plan angle. (3) γ angle is y-axis and projected on ZY(y) plane line vector The angle, the gamma angle rotates around X, also known as the Out-Plan angle. The six parameters (X _AA1 , Y _AA1 , Z _{AA1 and} α, β, γ) described above are sufficient to determine the 3D pose of the round rod.

Estimation of the In-planeα angle:

Line vector in 2D image plane Projecting a 3D line vector on the xy plane . however Angle is The angle between the X axis and the X axis. Line vector Is for the midpoint of the image with Connected. It is the center of gravity of the area where the ring A projects through the image plane through the focal length of the camera lens length λ. same, Ring A is the center of gravity of the area projected onto the image plane by the focal length of the same camera lens length γ. Therefore, the following formula can be obtained: [Math 2]

Where α is an acute angle. Here are four possible alpha angles, as shown in Figure 3: 1. Let x _a < x _b and y _a > y _b hold, and α be the first quadrant angle, ie . 2. Suppose x _a <x _b and y _a >y _b hold, and α is the second quadrant angle, ie . 3. Suppose x _a <x _b and y _a <y _b hold, and α is the third quadrant angle, ie . 4. Suppose x _a >x _b and y _a <y _b hold, and α is the fourth quadrant angle, ie . The table is organized as shown in Table 1 (four-quadrant alpha angle of XY plane): [Table 1]

Estimation of Out-plane β angle and depth Z value:

In the 2D image plane Vector can decide to project in 3D direction line vector The center of mass of the 2D region is projected by the search of the ring A and the ring B, as shown in FIG. Coordinate point is 2D projection The intersection of the line vector with the upper edge and the lower edge of the ring A and the ring B is as shown in FIG. 5, and the same The same is true for the coordinate point.

As defined in Figure 4 in the XZ plane, the rod The angle between the vector and the X-axis is called the β angle, and then the angle β of the round rod in each quadrant is defined. The first quadrant is The second quadrant is The third quadrant is The fourth quadrant is Then, the β angle and depth Z _{A1 of} each quadrant are derived in detail respectively.

Double loop projection point under known conditions And camera focal length , ring A and ring B width , the spacing between ring A and ring B And use the camera's imaging geometry and similar three small proportional relationships to calculate out-plane Corner and depth Z information.

Estimation of the first quadrant out-planeβ and depth Z _A1 :

This section describes the marking of the first quadrant round bar object on the two-dimensional image projection surface and the triangular proportional relationship calculation β angle and depth Z _A1 (the invention uses the virtual plane coordinates to derive, more in line with the orientation of the acquired image).

From Figure 5 (ring A), the imaging geometry of the ring A and the camera will be used to derive the formula: (I-1) Using the phase relationship of the camera, get (I-2) (I-3) Substitute Eq.(I-2) (I-3) to Eq.(I-1), get (I-4) Further simplifying Eq. (I-4) using 3D geometric relations (I-5) get (I-6)

Similarly, from Figure 5 (ring B), the following formula is obtained: (I-7) For a simple derivation order: , (I-8) Note Simplify Eq. (I-6) (I-7) and Eq. (I-8), get (I-9) (I-10) To further simplify (I-9) (I-10), (I-11) and obtain Eq. (I-12) and (I-13) (I-12) (I-13) Eq. (I-12) and (I-13) are divided by A and B, respectively, to obtain: (I-14) (I-15) Eq.(I-14) minus Eq.(I-15), get (I-16) Set (I-17) get (I-18)

Using the similar triangle relationship of Ring A and Ring B in Figure 5, we get: (I-19) Eq. (I-18) can be simplified using Eq. (I-19) (I-20) (I-21) Let (I-21-A) Eq.(I-21) is simplified as follows (I-22) due to , using Eq. (I-22), get with , and so, (due to assumptions The angle is an acute angle, so v ₁ takes a positive value) (I-23) And (I-24) (I-25-1) (I-25-1) is as follows: (I-25-2) (I-26)

Estimation of the second quadrant out-planeβ and depth Z _A1 :

This section introduces the mark of the second quadrant round rod object on the two-dimensional image projection surface difference and the triangular proportional relationship calculation β angle and depth Z _A1 (the invention uses the virtual plane coordinates to derive, which is more in line with the orientation of the acquired image) .

From Figure 6 (ring A), the imaging geometry of the ring A and the camera will be used to derive the formula: (II-1) Using the phase relationship of the camera, get (II-2) (II-3) Substitute Eq. (II-2) (II-3) to Eq. (II-1), get (II-4) Further simplification of Eq. (I-4) Using 3D geometric relations (II-5) get (II-6)

Similarly, from Figure 6 (ring B), the following formula is obtained: (II-7) For a simple derivation order: , (II-8) Note Simplify Eq. (II-6) (II-7) and Eq. (II-8), get (II-9) (II-10) To further simplify (II-9)(II-10), (II-11) and obtain Eq. (II-12) and (II-13) (II-12) (II-13) Eq. (II-12) and (II-13) are divided by A and B, respectively, to obtain: (II-14) (II-15) Eq. (II-14) minus Eq. (II-15), obtained (II-16) Set (II-17) get (II-18)

Using the similar triangle relationship of Ring A and Ring B in Figure 6, we get: (II-19) Eq. (II-18) can be simplified using Eq. (II-19) (II-20) (II-21) Let (II-21-A) Eq. (II-21) is simplified as follows (II-22) due to , using Eq. (II-22), get with and so, (due to assumptions The angle is an acute angle, so v ₁ takes a positive value) (II-23) And (II-24) From eq.(II-14) (II-25-1) (II-25-1) Details are as follows: (II-25-2) (II-26)

Estimation of the third quadrant out-planeβ and depth Z _A1 :

This section describes the marking of the third quadrant round bar object on the two-dimensional image projection surface and the triangular proportional relationship calculation β angle and depth Z _A1 (the invention uses the virtual plane coordinates to derive, more in line with the orientation of the acquired image).

From Figure 7 (ring A), the imaging geometry of the ring A and the camera will be used to derive the formula: (III-1) Using the phase relationship of the camera, get (III-2) (III-3) Substitute Eq. (III-2) (III-3) to Eq. (III-1), get (III-4) Further simplifying Eq. (III-4) using 3D geometric relations (III-5) get (III-6)

Similarly, from Figure 7 (ring B), the following formula is obtained: (III-7) For a simple derivation order: , (III-8) Note Simplify Eq.(III-6) (III-7) and Eq.(III-8), get (III-9) (III-10) To further simplify (III-9)(III-10), (III-11) and obtained (III-12) (III-13) Eq. (III-12) and (III-13) are divided by A and B, respectively, to obtain: (III-14) (III-15) Eq. (III-14) minus Eq. (III-15), (III-16) Set (III-17) get (III-18)

Using the similar triangle relationship of Ring A and Ring B in Figure 7, we get: (III-19) Eq.(III-18) can be simplified using Eq.(III-19) (III-20) (III-21) Let (III-21-A) Eq. (III-21) is simplified as follows (III-22) due to , using Eq. (III-22), get And , thus, (due to assumptions The angle is an acute angle, so v ₁ takes a positive value) (III-23) And (III-24) From eq.(III-14) (III-25-1) (III-25-1) is as follows: (III-25-2) (III-26)

Estimation of the fourth quadrant out-planeβ and depth Z _A1 :

This section describes the marking of the fourth quadrant round rod object on the two-dimensional image projection surface and the triangular proportional relationship calculation β angle and depth Z _A1 (the invention uses virtual plane coordinates to derive, more in line with the orientation of the acquired image).

From Figure 8 (ring A), the imaging geometry of the ring A and the camera will be used to derive the formula: (IV-1) Using the phase relationship of the camera, get (IV-2) (IV-3) Substitute Eq. (I-2) (I-3) to Eq. (I-1), get (IV-4) Further simplifying Eq. (IV-4) using 3D geometric relationships (IV-5) get (IV-6)

Similarly, from Figure 8 (ring B), the following formula is obtained: (IV-7) For a simple derivation order: , (IV-8) Note Simplify Eq. (IV-6) (IV-7) and Eq. (IV-8), get (IV-9) (IV-10) To further simplify (IV-9)(IV-10), (IV-11) and obtain Eq. (IV-12) and (IV-13) (IV-12) (IV-13) Eq. (IV-12) and (IV-13) are divided by A and B, respectively, to obtain: (IV-14) (IV-15) Eq.(IV-14) minus Eq.(IV-15), get (IV-16) Set (IV-17) get (IV-18)

Using the similar triangle relationship of Ring A and Ring B in Figure 8, we get: (IV-19) Eq.(IV-18) Eq.(IV-19) can be used for simplification (IV-20) (IV-21) Let (IV-21-A) Eq.(IV-21) is simplified as follows (IV-22) due to , using Eq. (IV-22), get with ,and so, (due to assumptions The angle is an acute angle, so v ₁ takes a positive value) And (IV-24) (IV-25-1) (IV-25-1) is as follows: (IV-25-2) (IV-26)

The following is a list of possible beta angles as shown in Figure 9, SD = size(ringA)-size(ringB), where size() is the total area of ring A or ring B (number of pixels) 1. Suppose SD <0 and x _a >x _{b ,} established, the round rod faces the first quadrant, And (I-24) (I-25) (I-26) 2. Suppose SD<0 and x _a <x _{b , which are} true, the round rod faces the second quadrant And (II-24) (II-25) (II-26) 3. Assuming SD>0 and x _a <x _{b ,} established, the round rod faces the third quadrant And (III-24) (III-25) (III-26) 4. Suppose SD>0 and x _a >x _{b ,} established, the round rod faces the fourth quadrant And (IV-24) (IV-25) (IV-26)

Comparison of the derivation results of one to four quadrants β: According to the above-mentioned attitude of the circular rod on the XZ plane, the angles and depths of each quadrant Z _A1 are different, as shown in the following Table 2: [Table 2]

Out-plane γ angle and depth Z value estimation

In the 2D image plane Vector can decide to project in 3D direction line vector The center of mass of the 2D region is projected by the search of the ring A and the ring B, as shown in FIG. Coordinate point is 2D projection The intersection of the line vector with the upper edge and the lower edge of the ring A and the ring B is as shown in FIG. The same is true for the coordinate point.

As shown in Figure 10, defined in the YZ plane, the rod The angle between the vector and the X-axis is called the angle, which defines the angle of the round rod in each quadrant. The first quadrant is The second quadrant is The third quadrant is The fourth quadrant is After that, each quadrant will be deduced in detail. Angle and depth Z _A1 .

Double loop projection point And camera focal length , ring A and ring B width , the spacing between ring A and ring B The out-plane angle and depth Z information are calculated using the camera's imaging geometry and similar three small scale relationships.

First quadrant out-plane gamma and depth Z _A1 estimate

This section describes the angle of the two-dimensional image projection surface and the triangular proportional relationship calculation angle and depth Z _{A1 of the} first quadrant round rod object (the invention uses the virtual plane coordinates to derive the orientation of the acquired image).

From Figure 11 (ring A), the imaging geometry of the ring A and the camera will be used to derive the formula: (I-1) Using the phase relationship of the camera to (I-2) (I-3) Substitute Eq. (I-2) (I-3) to Eq. (I-1), get (I-4) Further simplifying Eq. (I-4) Using 3D geometric relations (I-5) get (I-6)

Similarly, from Figure 11 (ring B), the following formula is obtained: (I-7) For a simple derivation order: , (I-8) Note Simplify Eq. (I-6) (I-7) and Eq. (I-8), get (I-9) (I-10) To further simplify (I-9) (I-10), (I-11) and obtain Eq. (I-12) and (I-13) (I-12) (I-13) Eq. (I-12) and (I-13) are divided by A and B, respectively, to obtain: (I-14) (I-15) Eq.(I-14) minus Eq.(I-15), get (I-16) Set (I-17) get (I-18)

Using the similar triangle relationship of Ring A and Ring B in Figure 11, we get: (I-19) Eq.(I-18) can be simplified using Eq.(I-19) (I-20) (I-21) Let (I-21-A) Eq.(I-21) is simplified as follows (I-22) due to , using Eq.(I-22), get with ,and so, (due to assumptions The angle is an acute angle, so v ₁ takes a positive value) (I-23) And (I-24) (I-25-1) (I-25-1) is as follows: (I-25-2)

Estimation of the second quadrant out-planeγ and depth Z _A1 :

This section describes the projection of the second quadrant round bar object on the 2D image projection surface and the triangular proportional relationship calculation. Angle and depth Z _A1 (The invention uses virtual plane coordinates to derive, which is more in line with the orientation of the acquired image).

From Figure 12 (ring A), the imaging geometry of the ring A and the camera will be used to derive the formula: (II-1) Using the phase relationship of the camera, get (II-2) (II-3) Substitute Eq. (II-2) (II-3) to Eq. (II-1), get (II-4) Further simplification of Eq. (II-4) Using 3D geometric relations (II-5) get (II-6)

Similarly, from Figure 12 (Ring B), the following formula is obtained: (II-7) For a simple derivation order , (II-8) Note Simplify Eq. (II-6) (II-7) and Eq. (II-8), get (II-9) (II-10) To further simplify (II-9) (II-10), (II-11) and obtain Eq. (II-12) and (II-13) (II-12) (II-13) Eq. (II-12) and (II-13) are divided by A and B, respectively, to obtain: (II-14) (II-15) Eq.( II-14) Subtract Eq.( II-15) and get (II-16) Set (II-17) get (II-18)

Using the similar triangle relationship of Ring A and Ring B in Figure 12, we get: (II-19) Eq.(II-18) can be simplified using Eq.(II-19) (II-20) (II-21) Let (II-21-A) Eq. (II-21) is simplified as follows (II-22) due to , using Eq. (II-22), get with ,and so, (due to assumptions The angle is an acute angle, so v ₁ takes a positive value) (II-23) And (II-24) From eq.(II-14) (II-25-1) (II-25-1) Details are as follows: (II-25-2) (II-26)

Estimation of the third quadrant out-planeγ and depth Z _A1 :

This section describes the marking of the third quadrant round bar object on the two-dimensional image projection surface and the triangular proportional relationship calculation angle and depth Z _A1 (the invention uses virtual plane coordinates to derive, more in line with the orientation of the acquired image).

From Figure 13 (ring A), the imaging geometry of the ring A and the camera will be used to derive the formula: (III-1) Using the phase relationship of the camera, get (III-2) (III-3) Substitute Eq. (III-2) (III-3) to Eq. (III-1), get (III-4) Further simplification of Eq. (III-4) Using 3D geometric relations (III-5) get (III-6)

Similarly, from Figure 13 (ring B), the following formula is obtained: (III-7) For a simple derivation order: , (III-8) Note Simplify Eq. (III-6) (III-7) and Eq. (III-8), get (III-9) (III-10) To further simplify (III-9) (III-10), (III-11) and obtained (III-12) (III-13) Eq. (III-12) and (III-13) are divided by A and B, respectively, to obtain: (III-14) (III-15) Eq. (III-14) minus Eq. (III-15), (III-16) Set (III-17) get (III-18)

Using the similar triangle relationship of Ring A and Ring B in Figure 13, we get: (III-19) Eq.(III-18) can be simplified using Eq.(III-19) (III-20) (III-21) Let (III-21-A) Eq. (III-21) is simplified as follows (III-22) due to , using Eq. (III-22), get And ,thus, (due to assumptions The angle is an acute angle, so v ₁ takes a positive value) (III-23) And (III-24) From eq.(III-14) (III-25-1) (III-25-1) is as follows: (III-25-2)

Estimation of the fourth quadrant out-planeγ and depth Z _A1 :

This section describes the marking of the fourth quadrant round rod object on the two-dimensional image projection surface and the triangular proportional relationship calculation β angle and depth Z _A1 (the invention uses virtual plane coordinates to derive, more in line with the orientation of the acquired image).

From Figure 14 (ring A), the imaging geometry of the ring A and the camera will be used to derive the formula: (IV-1) Using the phase relationship of the camera, get (IV-2) (IV-3) Substitute Eq. (IV-2) (IV-3) to Eq. (IV-1), get (IV-4) Further simplification of Eq. (IV-4) using 3D geometric relations (IV-5) get (IV-6)

Similarly, from Figure 14 (ring B), the following formula is obtained: (IV-7) For a simple derivation order: , (IV-8) Note Simplify Eq.(IV-6) (IV-7) and Eq.(IV-8), get (IV-9) (IV-10) To further simplify (IV-9)(IV-10), (IV-11) and obtain Eq. (IV-12) and (IV-13) (IV-12) (IV-13) Eq. (IV-12) and (IV-13) are divided by A and B, respectively, to obtain: (IV-14) (IV-15) Eq.(IV-14) minus Eq.(IV-15), get (IV-16) Set (IV-17) get (IV-18)

Using the similar triangle relationship of Ring A and Ring B in Figure 14, you get: (IV-19) Eq.(IV-18) Eq.(IV-19) can be used for simplification (IV-20) (IV-21) Let (IV-21-A) Eq.(IV-21) is simplified as follows (IV-22) due to , using Eq.(IV-22), get with ,and so, (due to assumptions The angle is an acute angle, so v ₁ takes a positive value) And (IV-24) (IV-25-1) (IV-25-1) is as follows: (IV-25-2)

Listed below The possibility of the angle, as shown in Figure 15, is SD=size(ringA)-size(ringB), where size() is the total area of ring A or ring B: 1. Assuming SD<0 and y _a >y _{b , it is} true. Then the round rod faces the first quadrant, And (I-24) (I-25) (I-26) 2. Assuming SD<0 and y _a <x _{b ,} established, the round rod faces the second quadrant And (II-24) (II-25) (II-26) 3. Assuming SD>0 and y _a <y _{b ,} established, the round rod faces the third quadrant And (III-24) (III-25) (III-26) 4. Suppose SD>0 and y _a >y _{b ,} established, the round rod faces the fourth quadrant And (IV-24) (IV-25) (IV-26)

Comparison of gamma derivation results from one to four quadrants:

According to the above γ of the circular rod attitude on the YZ plane, the results derived from the γ angle and the depth Z _A1 of each quadrant are independent, as shown in Table 3 below: [Table 3]

Estimate 3D coordinate reference point AA1:

In Fig. 16, the axis reference point coordinates (X _AA1 , Y _{AA1 ,} Z _AA1 ) are found based on the 3D coordinates A1 (X _A1 , Y _{A1 ,} Z _A1 ) of the estimated surface. Give an image coordinate , Z _A1 and If you want to estimate the 3D coordinates A1 (X _A1 , Y _{A1 ,} Z _A1 ) of the rod surface and the reference point AA1, the steps are as follows:

Step1: When estimating _{A1 (X A1, Y A1,} Z A1) 3D coordinates on the surface of the rod-like member, since there is no point A1 _A1 the Z depth information, the process is determined by an angle β and γ obtained the Z _A1, can be used The inverse perspective transformation in image geometry yields X _A1 and Y _A1 with the following formula: (27) (28)

Step2: Estimate 3D coordinates. From Figure 16, (29) , , , R is the radius of the rod. Therefore, it is possible to obtain a 3D attitude 6 parameter defining a double-ring round rod: X _AA1 , Y _{AA1 ,} Z _AA1 and .

AA1 derivation results extending to other quadrants:

The above derivation of the rounded rod at the first quadrant AA1 point, this section will make a comparison of the derivation results for each quadrant. The round rod is placed in four quadrant positions centered on the center of the mirror, and the surface is seen when the rod is on the left side of the mirror core. When the rod is on the right side of the mirror, the surface is completely Blocked, so according to Figure 17, the formula for all the different AA1 coordinates is deduced as shown in Table 4: [Table 4]

Regardless of the X-axis or Y-axis, the position relative to the meditation is positive (right or top) If it is negative (left or bottom), it must be added .

3D eight-quadrant three-angle attitude and position derivation relationship of the total round bar:

It is conceivable that the round rod has eight quadrants in the three-dimensional space, as shown in Fig. 18. Since the three-dimensional space has three planes, the three planes are also defined as three angles α, β, γ, α angle. It is defined by the XY plane, the β angle is defined by the YZ plane, the γ angle is defined by the XZ plane, and then the relationship table of the eight quadrant poses is analyzed by Fig. 18, as shown in Fig. 19, it can be more clearly known. The conditional relationship required for the eight postures. Among them, Q1~Q4 are in the same direction as the lens, and Q5~Q8 are pointing to the lens.

To sum up, the original algorithm has only a single attitude quadrant derivation formula, which is not enough to calculate the arbitrary posture of the round rod object. Therefore, the 3D space is defined as eight-quadrant, and eight quadrants are displayed. The α angle is XY. The plane definition, the γ angle is defined by the YZ plane, and the β angle is defined by the XZ plane. The derivation results α, β, γ, and Z _A1 are listed in Tables 1 to 3, respectively. In addition, the rod axis point AA1 will be based on the four quadrants centered on the lens, regardless of the X-axis or Y-axis, the position relative to the meditation is positive (right or above). If it is negative (left or bottom), then ∆z must be added.

About the double-loop color selection of the marked round rod object in the MIS minimally invasive surgery environment:

The invention will extend to the actual environment of MIS minimally invasive surgery, and find the complete contour of the double loop in the complex organ background, thereby obtaining the 2D information required for the 3D positioning information, and performing color model analysis on the complex organ background to determine the color of the double loop. Analyze the color that can be easily identified in the laparoscopic environment to find the complete contour of the circular marker, quickly and completely identify the contour of the double loop in the complex organ background, and obtain accurate two-dimensional information.

Another object of the present invention is to find two marker ring colors that are readily identifiable in a real surgical environment, using the following steps:

Step1: Collect a large number of representative endoscopic complex organ images, and do the collage action and Histogram analysis to find the least 2 colors to use; Step2: Then use the double-ring scalpel with monotonous background to find any ring of double ring. Make a precise answer image (here is the A-ring as a precise answer); Step3: Use the drawing software PhotoImpactX3 to combine the object-oriented surgical instrument knife with the real environment and the possible colors on the ring (except for the first step) Color, then add 8 easily distinguishable extreme colors as possible color options called possible colors), make him an experimental image; Step4: Apply the experimental image to the proposed double-loop contour algorithm proposed by the present invention to obtain the A-ring contour Let him filter out the A-ring experimental image; Step5: Perfect answer R_P(x,y) and filter out the A-loop experimental image R_j^A(x,y) to do XOR to find the difference pixel and calculate the variance analysis (j =1~50); Step6: Make a double ring color selection according to the step 5 analysis result. Fig. 21 is a flow chart of the aforementioned Steps 1 to 5.

Get the double loop contour algorithm:

The method for obtaining a double loop contour algorithm proposed by the present invention defines the color on the image as (R, G, B), and the color defined for the ring defines the A ring as (Ra, Ga, Ba) and the B ring as (Rb). , Gb, Bb), using the concept of channel difference, each Pixel in the image is subtracted from the absolute value of the three channels, and the threshold values are respectively set in the three channels after subtraction, respectively defined as: [Math 3]

The threshold value is set primarily for the tolerance of each color to the light source. It is assumed that the above conditions are true and belong to the ring Pixel. The flow chart is shown in Figure 20 (for example: ring).

Performance analysis of each color:

1. Let the exact Marker position answer be R _p (x, y), and let the actual complex organ background Marker position of Step 5 be R _IJ (x, y) for logical operation XOR (exclusive or) mutual exclusion or gate: .

2. Calculate every N Cases, the exact Marker position R _p (x, y) binary image and the actual complex organ background Marker position binary image R _IJ (x, y) do XOR (exclusive or), find the difference Pixels, as shown in Figure 22: [Math 4]

3. In operation side while variance (Variance), the variation of the number of the following formula: [Mathematical Formula 5]

Among the formulas Equivalent to my actual complex organ background Marker position binary image R _IJ (x, y), μ corresponds to the exact Marker position R _p (x, y) binary image, n is the experimental number of sheets.

4. Record the experimental data table for the total number and variance of each color XOR (exclusive or) operation.

The following is the experimental data calculated using 50 complex organ backgrounds for each possible experimental color. Next, adjust the threshold values TR, TG, and TB to observe each color. With the Variance, the two most suitable colors are selected as the double ring color of the present invention.

From the data in Table 5 below, it is found that the two total colors (255, 255, 255) and (0, 0, 0) have been affected. The number becomes larger when the threshold value is set to 1, and the variation (Variance) also becomes larger (255, 255, 255). The effect of the all white color is the largest, and can basically be eliminated (255, 255, 255). With (0,0,0). 【table 5】

The data observations in Table 6 below (the threshold values are all set to 10) and Table 7 (the threshold values are all set to 30) are in addition to (255, 255, 255) and (0, 0, 0). With the increase of the threshold (Variance), the other colors are not affected. [Table 6] [Table 7]

It can be seen from Table 6 to Table 7 above that in the case where the threshold value 10 is adjusted to 30, the two colors to be selected are not seen, so it is decided to directly adjust the threshold value to 120 observation data. The following table 8 (the threshold value is set to 10) data observation except the extreme value color (0, 255, 255) and (0, 255, 0) are less affected, the rest of the color Variant (Variance) increases with the threshold value, then you can choose the two colors that the marker ring is most suitable for in the real operating environment. Figure 23 also shows the ideal color ranking of the real environment marker ring. [Table 8]

The MIS minimally invasive surgical environment determines the RGB threshold of the marker ring:

Due to the actual surgical environment, the surgical instrument will be reflected by the illumination. It will be simulated by the drawing software (PhotoimpacX3t). According to the invention, the double loop contour algorithm is obtained, and the appropriate RGB threshold value (light source tolerance) is adjusted and determined to obtain a clear and complete double loop contour. The method flow chart, as shown in Figure 24, aims to find a suitable RGB threshold value suitable for use in a real surgical environment.

Step1: Use the monotonous all black as the background to find the position of the A ring and the B ring, as the experimental perfect (R _P (x, y); Step 2: combine multiple surgical instruments with the drawing soft PhotoimpactX3 Into the reflection, the image is the test image; Step3: The experimental image is applied to the double loop contour algorithm proposed by the present invention to obtain the double loop contour, so that he can filter out the A loop experimental image; Step 5: perfect answer (x,y) and filtered double loop experimental images Do XOR to find the difference pixel and calculate the variance analysis (j=1~20); and Step6: Make the threshold value selection according to the above step 5 analysis result.

Simulate real-world methods:

The invention has a monotonous all black background as shown in Fig. 25, and finds the positions of the A ring and the B ring.

Combine multiple surgical instruments with the drawing software PhotoImpactX3 to adjust the intensity of the light source. In the ambient light parameters of PhotoImpactX3, the intensity of the light source can be adjusted. The original image (zero-level light source) of this team will be used to adjust the intensity, medium and weak of the light source. And define strong, medium, and weak, as follows: when the ambient light parameter is 30, make him a primary light source (weak), as shown in Figure 26-1; when the ambient light parameter is 60, make him a secondary light source (middle), such as Figure 26-2; when the ambient light parameter is 90, make him a three-level light source (strong), as shown in Figure 26-3.

Get the double loop contour capture algorithm:

Let the color on the image be defined as (R, G, B), and the color set for the ring defines the A ring as (Ra, Ga, Ba) and the B ring as (Rb, Gb, Bb), using the concept of channel difference, Each Pixel in the image is subtracted from the absolute value of the three channels, and the threshold values are set respectively for the three channels after subtraction, which are respectively defined as: [Math 6] Among them, the threshold value is set mainly for the tolerance of each color to the light source. It is assumed that the above conditions are true and belong to the ring Pixel. The flow chart is shown in Figure 27 (for example: ring).

Performance analysis of each RGB threshold:

Let the exact Marker position answer be R _P (x, y), so that the actual complex organ background Marker position is R _J (x, y) to do the logical operation XOR (exclusive or) mutual exclusion or gate: [Math 7] Its schematic diagram is shown in Figure 28. Its purpose is to calculate the number of points that differ from each other. . (XOR rule: both are the same as 0, and the difference is 1).

[Formula 8] obtained by mutually exclusive ORing the above R _P (x, y) and R _j (X, Y) , Change the threshold and repeat the above steps and compare each . Its purpose: to select the best RGB threshold value for each level of light source.

The experimental results of the zero-level light source are as follows:

In order to find the ideal threshold value of the zero-order light source, first look for it from a wide range, as shown in Table 9, from the start of 1 to the increase of 10 to 250. It can be observed that when the threshold value is 1, the D _TH is the smallest. At this time, the range can be narrowed to about 30 threshold values to find the ideal threshold value. The result is shown in Table 10. [Table 9] [Table 10]

Therefore, from Table 9 to expand the scope of the search, it can be observed that the ideal threshold value may be again around 30, then narrow the range from the threshold value of 30 to find, as shown in Table 10, found from the threshold value of 25 It is an incremental state, and it can be known that the ideal threshold value of the zero-order light source is 25.

The experimental results of the primary light source are as follows:

In order to find the ideal threshold for the first-order light, first look for it from a wide range, as shown in Table 11, from the start of 1 to the increase of 10 to 250. It can be observed that when the threshold value is 130, the D _TH is the smallest. At this time, the range can be narrowed down to the threshold value of 130 to find the ideal threshold value. The result is shown in Table 12. [Table 11] [Table 12]

Looking at the expanded range from Table 11, it can be observed that the ideal threshold value may be again near 130, and then narrow the range from the threshold value 130 to find, as shown in Table 12, found above 130 or below. It is an incremental state, and it can be known that the ideal threshold value of the primary light source is 130.

The experimental results of the secondary light source are as follows:

In order to find the ideal threshold for secondary light, first look for it from a wide range, as shown in Table 13, from the start of 1 to the increase of 10 to 250. It can be observed that the D _TH is the smallest when the threshold value is 60. At this time, the range can be narrowed down to the threshold value of about 60 to find the ideal threshold value. The results are shown in Table 14. [Table 13] [Table 14]

Looking up from the expanded range of Table 13, it can be observed that the ideal threshold value may be again around 60. At this time, the narrowing range is further searched from the threshold value of 60, as shown in Table 14, and it is found that the threshold value is above 62. It is an incremental state, and it can be known that the ideal threshold value of the secondary light source is 62.

The experimental results of the three-level light source are as follows:

In order to find the ideal threshold for the three-level ray, first look for it from a wide range, as shown in Table 15, from the start of 1 to the increase of 10 to 250. It can be observed that when the threshold value is 1, the D _TH is the smallest. At this time, the range can be narrowed down to the threshold value of 1 to find the ideal threshold value. The result is shown in Table 16. [Table 15] [Table 16]

Looking up from the expanded range of Table 15, it can be observed that the ideal threshold value may be again near 1, then narrow the range from the threshold value of 1 to find, as shown in Table 16, found that the threshold value is above 1 It is an incremental state, and it can be known that the ideal threshold value of the three-level light source is 1.

After the above experiments, the light source is adjusted to be strong, medium, weak, and non-deterministic, and the ideal RGB threshold value of each light source level (zero, first, second, third) is determined. 1. Zero-level light source (no light source effect) threshold value is 25. 2. First-level light source (weak) threshold value is 130. 3. Secondary light source (middle) threshold value is 62. 4. Three-level light source (strong) threshold value Is 1.

Therefore, a large number of complex organ background images will be color model analyzed to find two colors that can be easily identified in the laparoscopic environment as a double ring. The A ring is (0, 255, 255) and the B ring is (0, 255, 0). For the illumination of the surgical instrument due to light illumination, using the drawing software PhotoimpactX3 to simulate the light source and defining the intensity of the light source (no light source, strong, medium, weak), the ideal RGB threshold value of each level of light source is determined, and the zero-level light source (no light source) Effect) The threshold is 25. The first-order glare (weak) threshold is 130. The secondary glare (middle) threshold is 62. The third-level glare (strong) threshold is 1. The receiver poses the instrument and observes the effect of the light source on the six parameters. Observing the result of the light source intensity on the 3D six parameters, it is still a small error. Possible reason: because the different positions of the filtered contours of the light rays and the perfect answer are distributed at the edge of the double loop when acquiring the double loop contour, resulting in acquisition There are differences between the upper and lower edges of the double loop, resulting in minor errors.

Marked round rod object three-dimensional positioning algorithm applied to endoscopic surgery:

The invention is based on the three-dimensional eight-image limited-bit algorithm flow of the two-dimensional image of the labeled round rod object, and influences the three-dimensional positioning parameters of the round rod in different postures and distances. Based on the above derivation formula, a three-dimensional eight-quadrant parameter estimation algorithm based on a single two-dimensional image is proposed and applied to the endoscopic surgery environment. The detailed steps are introduced.

First, the experimental results of the double-loop color selection in the aforementioned MIS minimally invasive surgery environment, as shown in Fig. 23 (only the top ten are listed) determine the color of the double ring, and the present invention selects the colors of the first and second names, respectively: (1) A ring: (0, 255, 255); (2) B ring: (0, 255, 0), these two colors can effectively distinguish the round rod in the real operating environment (Figure 5.1) The double loop mark of the object.

The marker-type round rod object is based on a three-dimensional image of a three-dimensional eight-image limited-bit algorithm flow:

The two-dimensional image of the marked round rod object of the invention is input, and the two-dimensional information of the round rod is obtained through the technology of modern image processing, and the marked round rod can be quickly calculated through the formula derived in the third chapter. The eight-quadrant arbitrary pose parameter of the object in three-dimensional space. Algorithm flow chart, as shown in Figure 29.

Step1. Take out the double loop outline:

Let the color on the image be defined as (R, G, B), and the color set for the ring defines the A ring as (Ra, Ga, Ba) and the B ring as (Rb, Gb, Bb), using the concept of channel difference, Each Pixel in the image is subtracted from the absolute value of the three channels, and the threshold values are respectively set in the three channels after subtraction, respectively defined as Mathematical Formula 6 as described above; wherein the threshold value is set mainly for each color for the light source Tolerance. It is assumed that the above conditions are true and belong to the ring Pixel, as shown in FIG. For example: Pixel∈A ring.

Step2. Image coordinate conversion:

Because the image development software (0,0) starts from the top left of the picture, but the process of deriving the formula is based on the center of the image (0,0), so the coordinates must be converted to move the origin to the center, as shown in the figure. 31, the input is an MXN size image or video frame f (i, j), to f (i, j) into f (x, y), the formula is as follows: [Math 9] f ( x,y)=(iM/2,N/2-j)

Step3. Find the center of gravity of the double ring:

The A and B rings on the round rod object are cut out by the aforementioned Step 1 and then the center of gravity coordinates are calculated respectively, as shown in Fig. 32, and the coordinates of the center of gravity of the object (Xc, Yc) can be calculated by the mathematical formula 9, wherein P is pixel. [Math 10]

Step4. Get double-loop 2D information:

Using the aforementioned Step 3, the two-ring center of gravity coordinates are connected to form a straight line equation L of the axis segment, as shown in FIG. 33, and the two-ring binarized images are respectively obtained by Canny edge detection or morphological method to obtain all the edge point coordinates of the two rings. The linear equation L brought into the axis segment calculates the coordinate positions A (X _a1 , Y _a1 ), A (X _a2 , Y _a2 ), B (X _b1 , Y _b1 ) at both ends of the ring mark by the edge points. B(X _b2 , Y _b2 ), as shown in Figure 34.

Step5.pixel is converted into mm units:

The position coordinates A (X _a1 , Y _a1 ), A (X _a2 , Y _a2 ), B (X _b1 , Y _b1 ), B (X _b2 , Y _b2 ) obtained by the above Step 4 are obtained from the pixel. The point Pixel is converted into mm units (10 mm = 1 cm), and the resolution of the image and the camera lens size are used. The present invention uses the Iphone5 camera resolution: 3264x2448, the camera lens size is: 4.54x3.42mm, the conversion factor is dy=4.54/3264, dx=3.42/2448.

Step6. Perform 3D pose estimation:

Through the Step1 to Step5 method, the parameters in the required two-dimensional space can be derived, and then the three-dimensional eight-image limit position system introduced above can be used to quickly calculate the arbitrary posture of the round rod-like object in the 3D space. Known camera focal length ( ), the length of the ring axis ( ), double ring center of gravity axis ( ), the double-ring projection point )with( ), to find the three-dimensional positioning parameters of the rod object { , , , , , }, the parameter solving steps are as follows:

It is an acute angle. Here are four possible The angles are as shown in Table 17 (the four-quadrant alpha angle of the XY plane), Table 18 (the four-quadrant beta angle of the XZ plane and the Z _A1 ), and Table 19 (the four-quadrant γ angle of the YZ plane and Z _A1 ). [Table 17] [Table 18] [Table 19]

Brought into the mathematical formula 11 [Math 11]

X _A1 and Y _{A1 are} obtained, and the reference point coordinate points AA1 (X _AA1 , Y _AA1 , Z _AA1 ) are obtained, as shown in Table 20 (the derivation results of the _A1 point coordinate of the circular rod-shaped one to four quadrants): [Table 20]

In summary, the present invention relates to a "tracking method for an endoscopic surgical instrument based on a two-dimensional image and a three-dimensional complete eight-image limit position", and the structure thereof has not been seen in various books or publicly used, and is in compliance with a patent application. In the case of the requirements of the patent application, the stipulations of the patent application are based on the specific examples of the patent application and the technical principles applied. Changes in the functions of the products and their functions beyond the scope of this specification and the drawings shall be incorporated in the scope of this patent application.

a, b, c, d, e, f‧ ‧ steps

Figure 1: Schematic diagram of the relationship between a double-ring round rod object and the image plane. Figure 2: Reference point AA1 is a schematic diagram of the intersection of the axis of the round bar and the plane of the upper edge of the ring A. Figure 3: α angle is the X axis and is projected onto the xy (XY) image plane line vector Schematic diagram of the angle. Figure 4: The β angle is the X-axis and the vector projected on the X(x)-Z plane Schematic diagram of the angle (β is an acute angle). Figure 5: Schematic diagram of the marked round rod object in the two-dimensional image X-axis projection plane (first quadrant direction). Figure 6: Schematic diagram of the marked round rod object in the two-dimensional image X-axis projection plane (second quadrant direction). Figure 7: Schematic diagram of the marked round rod object in the two-dimensional image X-axis projection plane (third quadrant direction). Figure 8 is a schematic view of a two-dimensional image X-axis projection surface (fourth quadrant direction) of a marked round rod object. Figure 9: Schematic representation of the possible projection of all beta angles in the X(x)-Z plane (the angle β is acute). Figure 10: The angle is the Y-axis and the vector projected on the Y(y)-Z plane Schematic diagram of the angle (the angle is an acute angle). Fig. 11 is a schematic view showing the two-dimensional image Y-axis projection plane (first quadrant direction) of the marked round rod object. Fig. 12: Schematic diagram of the marked round rod object on the two-dimensional image Y-axis projection surface (second quadrant direction. Fig. 13: Schematic diagram of the marked round rod object on the two-dimensional image Y-axis projection surface (third quadrant direction) Figure 14: Schematic diagram of the marked round bar object in the two-dimensional image Y-axis projection plane (fourth quadrant direction) Figure 15: Possible gamma angle projections on the Y(y)-Z plane (γ angle is acute) Fig. 16: Schematic diagram of the geometric relationship between coordinate points A1 and AA1. Figure 17: Schematic diagram of the derivation of the coordinates of the AA1 point of the one-four quadrant of the circular rod. Figure 18: Conceptual view of the three-dimensional eight-quadrant attitude (the positive direction of the Z-axis is Schematic diagram of the lens direction.Figure 19: Schematic diagram of the three-dimensional eight-quadrant attitude relationship table. Figure 20: Schematic diagram of the proposed double-loop contour algorithm proposed by the present invention. Figure 21: Step 1 to Step 5 flow chart of the double-loop color selection of the MIS minimally invasive surgery environment. 22: The present invention is a schematic diagram of XOR of R _P (x, y) and R _IJ (x, y). Figure 23: Schematic diagram of the applicable color ranking of the double loop of the present invention. Figure 24: Flow chart of selecting the RGB threshold value of the present invention. 25: The invention is based on monotonous all black, 撷Bicyclic schematic of FIG. 26: a plurality of surgical instrument of FIG synthesized using graphics software to adjust the light source intensity PhotoImpactX3 schematic FIG. 27: a flowchart of the algorithm of the present invention, a schematic view of another profile acquisition bicyclic Figure 28: Another R _P ( X, y) and R _J (x, y) are logically operated XOR diagrams. Figure 29: Flow diagram of the three-dimensional eight-image limit bit algorithm based on two-dimensional images of the labeled round rod object of the present invention. Figure 31 is a schematic diagram showing the coordinate conversion of the algorithm of Figure 29 of the present invention. Figure 32 is a schematic diagram of the calculation of the coordinates of the center of gravity of the algorithm of Figure 29 of the present invention. Figure 33: AB of the algorithm of Figure 29 of the present invention Schematic diagram of the coordinate connection of the two-ring center of gravity. Figure 34: The algorithm of Figure 29 of the present invention acquires the upper and lower points of the two rings of AB.

Claims (6)

An endoscopic surgical instrument tracking method using a three-dimensional complete eight-image limited position includes: a double-loop contour removal step for capturing the first ring body (A) and the image of the input surgical instrument image projected onto the plane a planar image of the second ring body (B); an image coordinate conversion step of setting the center of the first ring body and the second ring body plane image as an origin, and performing coordinate conversion; a step of finding the center of gravity of the double ring, Calculating the coordinates of the center of gravity of the first ring body and the second ring body respectively; and obtaining the two-dimensional information step of the double ring, connecting the center of gravity coordinates of the first ring body and the second ring to form a linear equation (L) of the axis segment and Calculating the two center endpoint coordinates A1, A2 of the first ring body and the two center endpoint coordinates B1, B2 of the second ring body through the edge points; a pixel conversion millimeter unit step, the four end points are Coordinates A1, A2, B1, B2 are converted into millimeter units; a three-dimensional attitude estimation step is performed using the endpoint coordinates A1, A2, B1, B2 at the end of the previous step and the known camera focal length (λ), the ring axis Long (L), double ring center of gravity axis length (L _AB ), the parameters of the double-ring projection points (x _{a1 ,} x _{a2 ,} x _{b1 ,} x _b2 ) and ( y _{a1 ,} y _{a2 ,} y _{b1 ,} y _b2 ) are used together with the three-dimensional eight-image limit system. The parameters of the three-dimensional positioning parameters {X _A1 , Y _A1 , Z _A1 , α, β, γ} of the surgical instrument are obtained, and the arbitrary posture of the surgical instrument in space can be quickly calculated. An endoscopic surgical instrument tracking method using a three-dimensional complete eight-image limit according to claim 1, wherein the double-loop contour extraction step defines the first ring body as (Ra, Ga, Ba) and the second ring body For (Rb, Gb, Bb), using the concept of channel difference, each pixel in the first ring and the second ring image is subtracted from the absolute value of the three channels, and the three channels after subtraction are respectively Set its threshold value and define the following mathematical formula: [Math 1] Where the threshold value is set primarily for the tolerance of the light source for each color, assuming that the above conditions are true, the pixels belonging to the ring body. The endoscopic surgical instrument tracking method using the three-dimensional complete eight-image limit position as claimed in claim 1, wherein in the image coordinate conversion step, a frame of MXN size image or video is set as f(i, j), and the application is as follows The mathematical formula converts f(i,j) into f(x,y): [Math 2] . The endoscopic surgical instrument tracking method using the three-dimensional complete eight-image limit position as claimed in claim 1, wherein in the step of finding the double-loop center of gravity, the coordinate point coordinate value (Xc, Yc) is calculated by the following mathematical formula, wherein P For the number of pixels (pixel): [Math 3] . The endoscopic surgical instrument tracking method using the three-dimensional complete eight-image limit position as claimed in claim 1, wherein the two-loop binary image is obtained by the Canny edge detection or the morphological method respectively. All the edge coordinates of the ring, the linear equation (L) brought into the axis segment calculates the coordinate positions A1 (X _a1 , Y _a1 ), A2 (X _a2 , Y _a2 ) at both ends of the ring mark through the edge points, B1 (X _b1 , Y _b1 ), B2 (X _b2 , Y _b2 ). The endoscopic surgical instrument tracking method using the three-dimensional complete eight-image limit position as claimed in claim 1, wherein the three-dimensional posture estimation step is performed by using the mathematical formula of the following steps to obtain the three-dimensional positioning parameter of the rod object. , , , , , }: Step (1) first find the possible four quadrants Angle: [Math 4] Step (2) then find the possible β angle of the four quadrants and Z _A1 : [Math 5] Step (3) then find the possible gamma angle of four quadrants and Z _A1 : [Math 6] Step (3) to find X _A1 and Y _{A1 again} [Math 7] Thus, the reference point coordinate point AA1 (X _AA1 , Y _AA1 , Z _AA1 ) can be obtained. [Math 8] .