WO2020210967A1

WO2020210967A1 - Optical tracking system and training system for medical instruments

Info

Publication number: WO2020210967A1
Application number: PCT/CN2019/082803
Authority: WO
Inventors: 孙永年; 周一鸣; 朱敏慈; 沈庭立; 邱昌逸; 蔡博翔
Original assignee: 孙永年
Priority date: 2019-04-16
Filing date: 2019-04-16
Publication date: 2020-10-22

Abstract

An optical tracking system (1) for medical instruments (21-24) comprises multiple optical markers (11), multiple optical sensors (12) and a computer device (13). The optical markers (11) are provided at the medical instruments (21-24). The respective optical sensors (12) optically sense the optical markers (11) to obtain multiple sensed signals. The computer device (13) has a three-dimensional surgical scenario model (14), and is connected to the optical sensors (12) to receive the sensed signals. The computer device adjusts, according to the sensed signals, relative positions of medical instrument representations (141-144) and a surgical target representation (145) in the three-dimensional surgical scenario model (14).

Description

Optical tracking system and training system for medical appliances

Technical field

The invention relates to an optical tracking system and a training system, in particular to an optical tracking system and a training system for medical appliances.

Background technique

It takes some time to train the operation of medical devices to make the learners become proficient. For minimally invasive surgery, in addition to operating a scalpel, ultrasound imaging probes are usually operated. The tolerance for minimally invasive surgery is not large. It usually takes a wealth of experience to proceed smoothly. Therefore, training before surgery is extremely important.

Therefore, how to provide an optical tracking system and training system for medical devices that can assist or train users to operate medical devices has become one of the important issues.

Summary of the invention

In view of the above-mentioned problems, the object of the present invention is to provide an optical tracking system and training system for medical devices, which can assist or train users to operate medical devices.

An optical tracking system for medical appliances, which includes a plurality of optical markers, a plurality of optical sensors, and a computer device. The optical markers are arranged on the medical appliance. The optical sensor optically senses the optical markers to generate a plurality of sensings respectively. signal. The computer device is coupled to the optical sensor to receive the sensing signal, and has a three-dimensional model of the surgical situation, and adjusts the relative position between the medical appliance present and the surgical target present in the three-dimensional model of the surgical situation according to the sensing signal.

In one embodiment, there are at least two optical sensors, which are arranged above the medical appliance and facing the optical marker.

In one embodiment, the computer device and the optical sensor perform a pre-operation procedure, and the pre-operation procedure includes: calibrating the coordinate system of the optical sensor; and adjusting the scaling ratio of the medical appliance and the surgical target object.

In one embodiment, the computer device and the optical sensor perform a coordinate calibration program, and the calibration program includes an initial calibration step, an optimization step, and a correction step. The initial correction step is to perform initial correction between the coordinate system of the optical sensor and the coordinate system of the three-dimensional model of the surgical situation to obtain initial conversion parameters. The optimization step is to optimize the degrees of freedom of the initial conversion parameters to obtain the optimized conversion parameters. The correction step is to correct the setting error caused by the optical marker in the optimized conversion parameter.

In one embodiment, the initial calibration step is to use singular value decomposition (SVD), triangular coordinate registration (triangle coordinate registration), or linear least square estimation (linear least square estimation).

In one embodiment, the initial calibration step is to use singular value decomposition to find the transformation matrix between the feature points of the medical appliance and the optical sensor as the initial transformation parameter. The transformation matrix includes a covariance matrix and rotation For the rotation matrix, the optimization step is to obtain multiple Euler angles with multiple degrees of freedom from the rotation matrix, and use Gauss-Newton method to iteratively optimize the parameters of the multiple degrees of freedom to obtain optimized conversion parameters.

In one embodiment, the computer device sets the positions of the medical appliance presentation and the surgical target presentation in the three-dimensional model of the surgical situation according to the optimized conversion parameters and the sensing signal.

In one embodiment, the correcting step is to use the reverse conversion and sensing signals to correct the positions of the medical appliance presentation and the surgical target presentation in the three-dimensional model of the surgical situation.

In one embodiment, the computer device outputs display data, and the display data is used to present the 3D images of the medical appliance presentation and the surgical target presentation.

In one embodiment, the computer device generates the medical image according to the three-dimensional model of the surgical situation and the medical image model.

In one embodiment, the surgical target object is an artificial limb, and the medical image is an artificial medical image for the surgical target object.

In one embodiment, the computer device deduces the position of the medical appliance inside and outside the surgical target object, and adjusts the relative position between the medical appliance presentation and the surgical target presentation in the three-dimensional model of the surgical situation accordingly.

A training system for operating medical appliances includes medical appliances and the aforementioned optical tracking system for medical appliances.

In one embodiment, the medical appliance includes a medical probe and a surgical appliance, and the medical appliance presentation includes a medical probe presentation and a surgical appliance presentation.

In one embodiment, the computer device scores the detection object found by the medical probe present and the operation of the surgical instrument present.

A method for calibrating an optical tracking system for medical appliances includes a sensing step, an initial calibration step, an optimization step, and a correction step. The sensing step uses a plurality of optical sensors of the optical tracking system to optically sense a plurality of optical markers arranged on the optical tracking system on the medical appliance to generate a plurality of sensing signals; the initial calibration step performs the optical sensor according to the sensing signals The initial correction between the coordinate system of the surgical scene and the coordinate system of the three-dimensional model of the surgical situation to obtain the initial conversion parameters; the optimization step is to optimize the degrees of freedom of the initial conversion parameters to obtain the optimized conversion parameters; the correction step is to modify the leading factors in the optimized conversion parameters For the setting error of the optical marker.

In one embodiment, the calibration method further includes a pre-operation procedure, the pre-operation procedure includes calibrating the coordinate system of the optical sensor; and adjusting the zoom ratio for the medical appliance and the surgical target object.

In one embodiment, in the calibration method, the initial calibration step is to use singular value decomposition to find the transformation matrix between the feature points of the medical appliance presentation of the three-dimensional model of the surgical situation and the optical sensor as the initial transformation parameter. Variation matrix and rotation matrix. The optimization step is to obtain multiple Euras angles with multiple degrees of freedom from the rotation matrix, and use Gauss-Newton method to iteratively optimize the parameters of multiple degrees of freedom to obtain optimized conversion parameters.

In one embodiment, the positions of the medical appliance presentation and the surgical target presentation in the three-dimensional model of the surgical situation are set according to the optimized conversion parameters and the sensing signal. The correction step is to use the reverse conversion and sensing signals to correct the positions of the medical appliance presentation and the surgical target presentation in the three-dimensional model of the surgical situation.

In summary, the optical tracking system of the present disclosure can assist or train users to operate medical appliances, and the training system of the present disclosure can provide a realistic surgical training environment for the trainees to effectively assist the trainees in completing surgical training.

Description of the drawings

FIG. 1A is a block diagram of the optical tracking system of the embodiment.

1B and 1C are schematic diagrams of the optical tracking system of the embodiment.

Fig. 1D is a schematic diagram of a three-dimensional model of the surgical situation of the embodiment.

Fig. 2 is a flow chart of the pre-operation procedure of the optical tracking system of the embodiment.

FIG. 3A is a flowchart of the coordinate correction program of the optical tracking system of the embodiment.

Fig. 3B is a schematic diagram of the coordinate system correction of the embodiment.

Fig. 3C is a schematic diagram of the degrees of freedom of the embodiment.

Fig. 4 is a block diagram of the training system for medical appliance operation according to the embodiment.

Fig. 5A is a schematic diagram of a three-dimensional model of the operation situation of the embodiment.

FIG. 5B is a schematic diagram of a three-dimensional model of an entity medical image according to an embodiment.

FIG. 5C is a schematic diagram of the three-dimensional model of the artificial medical image of the embodiment.

6A to 6D are schematic diagrams of the direction vector of the medical appliance of the embodiment.

7A to 7D are schematic diagrams of the training process of the training system of the embodiment.

Fig. 8A is a schematic diagram of the finger structure of the embodiment.

FIG. 8B is a schematic diagram of applying principal component analysis on bones from computed tomography images in this embodiment.

Fig. 8C is a schematic diagram of applying principal component analysis on the skin from a computed tomography image in an embodiment.

Fig. 8D is a schematic diagram of calculating the distance between the bone spindle and the medical appliance according to the embodiment.

Fig. 8E is a schematic diagram of the artificial medical image of the embodiment.

Fig. 9A is a block diagram for generating artificial medical images according to an embodiment.

Fig. 9B is a schematic diagram of the artificial medical image of the embodiment.

10A and 10B are schematic diagrams of the artificial hand model and the correction of the ultrasonic volume of the embodiment.

Fig. 10C is a schematic diagram of ultrasonic volume and collision detection of the embodiment.

Fig. 10D is a schematic diagram of an artificial ultrasound image of the embodiment.

detailed description

Hereinafter, the optical tracking system and the training system for medical appliance operation according to the preferred embodiments of the present invention will be described with reference to related drawings, in which the same elements will be described with the same reference numerals.

As shown in FIG. 1A, FIG. 1A is a block diagram of the optical tracking system of the embodiment. The optical tracking system 1 for medical appliances includes a plurality of optical markers 11, a plurality of optical sensors 12, and a computer device 13. The optical markers 11 are arranged on one or more medical appliances, and here are a plurality of medical appliances 21-24 For example, the optical marker 11 can also be set on the surgical target object 3. The medical appliances 21-24 and the surgical target object 3 are placed on the platform 4, and the optical sensor 12 optically senses the optical marker 11 to generate multiple senses. Test signal. The computer device 13 is coupled to the optical sensor 12 to receive the sensing signal, and has a three-dimensional model 14 of the surgical context, and adjusts the gap between the medical appliance presents 141-144 and the surgical target present 145 in the three-dimensional model 14 of the surgical context according to the sensed signals relative position. The medical appliance presentation objects 141 to 144 and the surgery target presentation object 145 are shown in FIG. 1D, and represent the medical appliances 21 to 24 and the surgery target object 3 in the three-dimensional model 14 of the surgery situation. Through the optical tracking system 1, the three-dimensional model 14 of the surgical situation can obtain the current positions of the medical appliances 21-24 and the surgical target object 3 and reflect the medical appliance presentation and the surgical target presentation accordingly.

There are at least two optical sensors 12, which are arranged above the medical appliances 21-24 and facing the optical markers 11 for real-time tracking of the medical appliances 21-24 to know their positions. The optical sensor 12 may be a camera-based linear detector. For example, in FIG. 1B, FIG. 1B is a schematic diagram of the optical tracking system of the embodiment, four optical sensors 121 to 124 are installed on the ceiling and facing the optical marker 11, medical appliances 21 to 24 and the surgical target on the platform 4 Object 3.

For example, the medical tool 21 is a medical probe, such as a probe for ultrasonic imaging detection or other devices that can detect the inside of the surgical target object 3. These devices are actually used clinically, and the probe for ultrasonic imaging detection is, for example, ultrasound. Transducer (Ultrasonic Transducer). The medical appliances 22-24 are surgical appliances, such as needles, scalpels, hooks, etc., which are actually used clinically. If used for surgical training, the medical probe can be a device that is actually used in clinical practice or a simulated device that simulates clinical practice, and the surgical instrument can be a device that is actually used in clinical practice or a simulated device that simulates clinical practice. For example, in Figure 1C, Figure 1C is a schematic diagram of the optical tracking system of the embodiment. The medical appliances 21-24 and the surgical target 3 on the platform 4 are used for surgical training, such as minimally invasive finger surgery, which can be used for triggers. Refers to treatment surgery. The material of the clamps of the platform 4 and the medical appliances 21-24 can be wood. The medical appliance 21 is a realistic ultrasonic transducer (or probe), and the medical appliance 22-24 includes a plurality of surgical instruments, such as expanders ( dilator, needle, and hook blade. The surgical target 3 is a hand phantom. Three or four optical markers 11 are installed on each medical appliance 21-24, and three or four optical markers 11 are also installed on the surgical target object 3. For example, the computer device 13 is connected to the optical sensor 12 to track the position of the optical marker 11 in real time. There are 17 optical markers 11, including 4 that are linked on or around the

surgical target object

3, and 13 optical markers 11 are on medical appliances 21-24. The optical sensor 12 continuously transmits real-time information to the computer device 13. In addition, the computer device 13 also uses the movement judgment function to reduce the computational burden. If the moving distance of the optical marker 11 is less than the threshold value, the position of the optical marker 11 is not updated , The threshold value is, for example, 0.7mm.

In FIG. 1A, the computer device 13 includes a processing core 131, a storage element 132, and a plurality of I/O interfaces 133, 134. The processing core 131 is coupled to the storage element 132 and I/O interfaces 133, 134. The I/O interface 133 can receive optical sensors. 12, the I/O interface 134 communicates with the output device 5, and the computer device 13 can output the processing result to the output device 5 through the I/O interface 134. The I/O interfaces 133 and 134 are, for example, peripheral transmission ports or communication ports. The output device 5 is a device capable of outputting images, such as a display, a projector, a printer, and so on.

The storage element 132 stores program codes for execution by the processing core 131. The storage element 132 includes a non-volatile memory and a volatile memory. The non-volatile memory is, for example, a hard disk, a flash memory, a solid state disk, an optical disc, etc. The volatile memory is, for example, dynamic random access memory, static random access memory, and so on. For example, the program code is stored in the non-volatile memory, and the processing core 131 can load the program code from the non-volatile memory to the volatile memory, and then execute the program code. The storage component 132 stores the program code and data of the operation situation three-dimensional model 14 and the tracking module 15, and the processing core 131 can access the storage component 132 to execute and process the operation situation three-dimensional model 14 and the program code and data of the tracking module 15.

The processing core 131 is, for example, a processor, a controller, etc., and the processor includes one or more cores. The processor may be a central processing unit or a graphics processor, and the processing core 131 may also be the core of a processor or a graphics processor. On the other hand, the processing core 131 may also be a processing module, and the processing module includes multiple processors.

The operation of the optical tracking system includes the connection between the computer device 13 and the optical sensor 12, the pre-operation program, the coordinate correction program of the optical tracking system, the real-time rendering program, etc. The tracking module 15 represents the relevant program codes of these operations and Data, the storage element 132 of the computer device 13 stores the tracking module 15, and the processing core 131 executes the tracking module 15 to perform these operations.

After the computer device 13 performs the pre-work and the coordinate correction of the optical tracking system, the optimized conversion parameters can be found, and then the computer device 13 can set the medical appliance presentation objects 141-144 and the surgical target presentation objects 145 according to the optimized conversion parameters and the sensing signal The position in the three-dimensional model 14 of the surgical situation. The computer device 13 can deduce the position of the medical appliance 21 inside and outside the surgical target object 3, and adjust the relative position between the medical appliance presenting objects 141 to 144 and the surgical target presenting object 145 in the three-dimensional model 14 of the operation situation accordingly. In this way, the medical appliances 21-24 can be tracked in real time from the detection results of the optical sensor 12 and correspondingly presented in the three-dimensional model 14 of the surgical context. The presentation of the three-dimensional model 14 in the surgical context is shown in FIG. 1D, for example.

The three-dimensional model 14 of the operation situation is a native model, which includes models established for the surgical target object 3 and also includes models established for the medical appliances 21-24. The method of establishment can be that the developer directly uses computer graphics technology to construct it on the computer, such as using drawing software or special application development software.

The computer device 13 can output the display data 135 to the output device 5. The display data 135 is used to present 3D images of the medical appliance presentation objects 141-144 and the surgical target presentation object 145. The output device 5 can output the display data 135. The output method is, for example, Display or print etc. The result of outputting in a display mode is shown in FIG. 1D, for example.

As shown in FIG. 2, FIG. 2 is a flowchart of the pre-operation procedure of the optical tracking system of the embodiment. The computer device 13 and the optical sensor 12 perform a pre-operation procedure. The pre-operation procedure includes steps S01 and S02 for calibrating the optical sensor 12 and re-adjusting the scale of all medical appliances 21-24.

Step S01 is to calibrate the coordinate system of the optical sensor 12. A plurality of calibration sticks have a plurality of optical markers, and the area enclosed by them is used to define the working area. The optical sensor 12 senses the optical markers on the calibration stick. When all optical markers are detected by each optical sensor 12, the area enclosed by the correction rod is the effective working area. The calibration bar is manually placed by the user, and the user can adjust the position of the calibration bar to modify the effective working area. The sensitivity detected by the optical sensor 12 can be about 0.3 mm. Here, the coordinate system where the detection result of the optical sensor 12 is located is called the tracking coordinate system.

Step S02 is to adjust the scaling ratio of the medical appliances 21 to 24 and the surgical target object 3. The medical appliances 21-24 are usually rigid bodies, and the coordinate correction adopts a rigid body correction method to avoid distortion. Therefore, the medical appliances 21-24 must be rescaled to the tracking coordinate system to obtain correct calibration results. The calculation of scaling ratio can be obtained by the following formula:

Track _G : Tracking the center of gravity in the coordinate system

Track _i : Track the position of the optical marker in the coordinate system

Mesh _G : the center of gravity in the mesh point coordinate system

Mesh _i : the position of the optical marker in the mesh point coordinate system

The tracking coordinate system is the coordinate system adopted by the detection result of the optical sensor 12, and the dot coordinate system is the coordinate system adopted by the three-dimensional model 14 of the surgical situation. Step S02 first calculates the center of gravity in the tracking coordinate system and the dot coordinate system, and then calculates the distance between the optical marker and the center of gravity in the tracking coordinate system and the dot coordinate system. Then, for the individual ratios of the dot coordinate system to the tracking coordinate system, add up all the individual ratios and divide by the number of optical markers to obtain the ratio of the dot coordinate system to the tracking coordinate system.

As shown in FIG. 3A, FIG. 3A is a flowchart of the coordinate correction program of the optical tracking system of the embodiment. The computer device and the optical sensor perform a coordinate calibration program, and the calibration program includes an initial calibration step S11, an optimization step S12, and a correction step S13. The initial calibration step S11 performs an initial calibration between the coordinate system of the optical sensor 12 and the coordinate system of the three-dimensional model 14 of the surgical situation to obtain the initial conversion parameters. The calibration between the coordinate systems is shown in FIG. 3B for example. The optimization step S12 is to optimize the degrees of freedom of the initial conversion parameters to obtain the optimized conversion parameters. For example, the degrees of freedom are shown in FIG. 3C. The correcting step S13 is to correct the setting error caused by the optical marker in the optimized conversion parameter.

Since there is also a conversion between the tracking coordinate system and the coordinate system of the three-dimensional model 14 of the surgical situation, the optical marker attached to the platform 4 can be used to correct the two coordinate systems.

The initial correction step S11 is to find the transformation matrix between the feature points of the medical appliance and the optical sensor as the initial transformation parameter. The initial correction step is to use singular value decomposition (SVD) and triangular coordinate alignment (Triangle coordinate). registration) or linear least square estimation (linear least square estimation). The transformation matrix includes, for example, a covariance matrix and a rotation matrix.

For example, in step S11, singular value decomposition can be used to find the optimal transformation matrix between the feature points of the medical appliance exhibits 141 to 144 and the optical sensor as the initial transformation parameter, and the covariance matrix H can be obtained from These feature points are obtained and can be regarded as the objective function to be optimized. The rotation matrix M can be found by the following formula:

[U, Σ, V] = SVD (H); M = VU T

After obtaining the rotation matrix M, the translation matrix T can be found by the following formula:

T=-M×centroid _A +centroid _B

The optimization step S12 is to obtain multiple Euler angles with multiple degrees of freedom from the rotation matrix M, and use Gauss-Newton algorithm to iteratively optimize the parameters of multiple degrees of freedom to obtain optimized conversion parameters. The multiple degrees of freedom are, for example, six degrees of freedom, and other numbers of degrees of freedom, such as nine degrees of freedom, etc., are also possible to modify the expression appropriately. Since the conversion result obtained from the initial calibration step S11 may not be accurate enough, performing the optimization step S12 can improve the accuracy and obtain a more accurate conversion result.

Let γ represent the angle relative to the X axis, α represent the angle relative to the Y axis, and β represent the angle relative to the Z axis. The rotation of each axis of the world coordinate axis can be expressed as follows:

m ₁₁ =cosαcosβm ₁₂ =sinγsinαcosβ-corγsinβ

m ₁₃ =cosγsinαcosβ+sinγsinβ

m ₂₁ =cosαsinβm ₂₂ =sinγsinαsinβ+corγcosβ

m ₂₃ =cosγsinαsinβ-sinγcosβ

m ₃₁ =-sinα

m ₃₂ =sinγcosα

m ₃₃ ＝cosγcosα

The rotation matrix M can be obtained from the above formula. In general, multiple Euler angles can be obtained from the following formula:

γ=tan ^-1 (m ₃₂ , m ₃₃ )

β=tan ^-1 (sin(γ)m ₁₃ -cos(γ)m ₁₂ , cos(γ)m ₂₂ -sin(γ)m ₂₃ )

After taking out multiple Euler angles, the rotation of the world coordinate system is assumed to be orthogonal. Since the parameters of six degrees of freedom have been obtained, these parameters can be iteratively optimized by the Gauss-Newton method to obtain optimized conversion parameters.

Is the objective function to be minimized.

Among them, b represents the least square error between the reference target point and the current point, n is the number of feature points,

It is a transformation parameter which has translation and rotation parameters, and iteratively transforms the parameters by using Gauss Newton method

Will adjust to find the best value and change the parameters

The update function is as follows:

Δ is the Jacobian matrix from the objective function (Jacobian matrix)

Δ=(J ^T J) ^-1 J ^T b

The stop condition is defined as follows:

The correcting step S13 is to correct the setting error caused by the optical marker in the optimized conversion parameter. The correction step S13 includes a determination step S131 and an adjustment step S132.

In step S13, the source feature point correction procedure can be used to overcome the error caused by manually selecting feature points. This is because the medical appliance presentation objects 141 to 144 of the surgical scene three-dimensional model 14 and the surgical target presentation object 145 have feature points and medical treatments. The error of the feature points of the tools 21-24 and the surgical target object 3, these feature points are selected by the user. The feature points of the medical tools 21-24 and the surgical target object 3 may include points where the optical marker 11 is set. Since the optimal transformation can be obtained from step S12, the target position transformed from the source point will approach the reference target point V ^T after the nth iteration as follows:

The conversion matrix from the source point to the target point in the nth iteration

The source point of the nth iteration

The target point after the nth iteration

In the source point correction step, first calculate the inverse transformation of the transformation matrix, and then obtain the new source point from the reference target point. The calculation formula is as follows:

Inverse transformation of transformation matrix

The new source point after the inverse transformation of the nth iteration

The target point after the nth iteration

Assuming that the exact transformation of the two coordinate systems is as above, after n iterations, the new source point will be the ideal position of the original source point. However, the original source point and the ideal source point are somewhat shifted. In order to correct the original source point to minimize the manual selection error, each iteration can be set to a constraint step size (constraint step size) c ₁ , and set the constraint area box size ( constraint region box size) c ₂ It can be a constant value to limit the distance moved by the original source point. This correction is as follows:

In each iteration, if the distance between these two points is less than c ₁ , the source point will move to the new point, otherwise the source point will only move to the new point by the length c ₁ . If the following situation occurs, the iteration will be aborted. V _T from the source point is the target point after V _S transformation.

Through the correction of the foregoing three steps, the coordinate position of the three-dimensional model 14 of the surgical situation can be accurately transformed to correspond to the optical marker 11 in the tracking coordinate system, and vice versa. As a result, the medical tools 21-24 and the surgical target object 3 can be tracked in real time based on the detection results of the optical sensor 12, and the positions of the medical tools 21-24 and the surgical target object 3 in the tracking coordinate system can be processed after the aforementioned processing. In the contextual three-dimensional model 14, the medical appliance presentation objects 141-144 correspond to the surgical target presentation object 145 accurately. As the medical appliances 21-24 and the surgical target object 3 actually move, the medical appliance presentation objects 141-144 and the surgical target presentation The object 145 will follow the movement of the three-dimensional model 14 of the operation situation in real time.

As shown in FIG. 4, FIG. 4 is a block diagram of the training system for the operation of the medical appliance according to the embodiment. The training system for medical appliance operation (hereinafter referred to as the training system) can truly simulate the surgical training environment. The training system includes an optical tracking system 1a, one or more medical appliances 21-24, and the surgical target object 3. The optical tracking system 1a includes a plurality of optical markers 11, a plurality of optical sensors 12, and a computer device 13. The optical markers 11 are arranged on medical appliances 21-24 and surgical target objects 3, and medical appliances 21-24 and surgical target objects 3 are placed On platform 4. For the medical appliances 21-24 and the surgical target object 3, the medical appliance presents 141 to 144 and the surgical target presents 145 are correspondingly presented on the three-dimensional model 14a of the surgical context. The medical tools 21-24 include medical probes and surgical tools. For example, the medical tools 21 are medical probes, and the medical tools 22-24 are surgical tools. The medical appliance presentations 141-144 include medical probe presentations and surgical appliance presentations. For example, the medical appliance presentation 141 is a medical probe presentation, and the medical appliance presentations 142-144 are surgical appliance presentations. The storage component 132 stores the program code and data of the operation situation three-dimensional model 14a and the tracking module 15, and the processing core 131 can access the storage component 132 to execute and process the operation situation three-dimensional model 14a and the program code and data of the tracking module 15. The implementations and changes of the elements corresponding to or with the same numbers in the preceding paragraphs and drawings can be referred to the descriptions in the previous paragraphs, so they will not be repeated here.

The surgical target object 3 is an artificial limb, such as artificial upper limbs, hand phantoms, artificial palms, artificial fingers, artificial arms, artificial upper arms, artificial forearms, artificial elbows, artificial upper limbs, artificial feet, artificial toes, artificial ankles, artificial Calf, false thigh, false knee, false torso, false neck, false head, false shoulder, false chest, false abdomen, false waist, false hip or other false parts, etc.

In this embodiment, the training system takes the minimally invasive surgery training of the fingers as an example. The surgical target 3 is a prosthetic hand, the surgery is for example a trigger finger treatment surgery, and the medical probe 21 is a realistic ultrasonic transducer (or probe). ), the surgical instruments 22-24 are a needle, a dilator, and a hook blade. In other embodiments, other surgical target objects 3 may be used for other surgical training.

The storage element 132 also stores the program codes and data of the physical medical image 3D model 14b, the artificial medical image 3D model 14c, and the training module 16. The processing core 131 can access the storage element 132 to execute and process the physical medical image 3D model 14b and artificial medicine. The program code and data of the image 3D model 14c and the training module 16. The training module 16 is responsible for the following surgical training procedures and the processing, integration and calculation of related data.

The image model for surgical training is pre-established and imported into the system before the surgical training process. Taking minimally invasive finger surgery training as an example, the content of the image model includes finger bones (metacarpal and proximal phalanx) and flexor tendons. These image models can refer to FIGS. 5A to 5C. FIG. 5A is a schematic diagram of the three-dimensional model of the surgical scene of the embodiment, FIG. 5B is a schematic diagram of the three-dimensional physical medical image model of the embodiment, and FIG. 5C is the three-dimensional model of the artificial medical image Schematic. The content of these three-dimensional models can be output or printed by the output device 5.

The solid medical image three-dimensional model 14b is a three-dimensional model established from medical images, which is a model established for the surgical target object 3, such as the three-dimensional model shown in FIG. 5B. The medical image is, for example, a computer tomography image, and the image of the surgical target object 3 actually generated after the computer tomography is used to build the solid medical image three-dimensional model 14b.

The artificial medical image three-dimensional model 14c contains an artificial medical image model. The artificial medical image model is a model established for the surgical target object 3, such as the three-dimensional model shown in FIG. 5C. For example, the artificial medical imaging model is a three-dimensional model of artificial ultrasound images. Since the surgical target 3 is not a real living body, although computer tomography can obtain images of the physical structure, other medical imaging equipment such as ultrasound imaging can still be used. Effective or meaningful images cannot be obtained directly from the surgical target object 3. Therefore, the ultrasound image model of the surgical target object 3 must be artificially generated. Selecting an appropriate position or plane from the three-dimensional model of artificial ultrasound images can generate two-dimensional artificial ultrasound images.

The computer device 13 generates a medical image 136 according to the three-dimensional model 14a of the surgical situation and the medical image model. The medical image model is, for example, a solid medical image three-dimensional model 14b or an artificial medical image three-dimensional model 14c. For example, the computer device 13 generates a medical image 136 based on the three-dimensional model 14a of the surgical situation and the three-dimensional model 14c of an artificial medical image. The medical image 136 is a two-dimensional artificial ultrasound image. The computer device 13 scores the detection object found by the medical probe 141 and the operation of the surgical instrument representation 145, such as a specific surgical site.

6A to 6D are schematic diagrams of the direction vector of the medical appliance of the embodiment. The direction vectors of the medical device presentation objects 141-144 corresponding to the medical devices 21-24 will be rendered in real time. For the medical device presentation object 141, the direction vector of the medical probe can be calculated by calculating the center of gravity of the optical marker And get, and then project from another point to the xz plane, and calculate the vector from the center of gravity to the projection point. Other medical appliance presentations 142-144 are relatively simple, and the direction vector can be calculated using the sharp points in the model.

In order to reduce the burden of the system and avoid delays, the amount of image rendering can be reduced. For example, the training system can only draw the model of the area where the surgical target presenting object 145 is located instead of drawing all the medical appliance presenting objects 141-144.

In addition, in the training system, the transparency of the skin model can be adjusted to observe the internal anatomical structure of the surgical target present 145, and to see ultrasound image slices or computed tomography image slices of different cross-sections, such as horizontal cross-sections. plane or axial plane), sagittal plane (sagittal plane) or coronal plane (coronal plane), which can help the operator during the operation. The bounding boxes of each model are constructed for collision detection. The surgical training system can determine which medical appliances have contacted tendons, bones and/or skin, and can determine when to start scoring.

Before performing the calibration procedure, the optical marker 11 attached to the surgical target object 3 must be clearly seen or detected by the optical sensor 12. If the optical marker 11 is covered, the position of the optical marker 11 must be detected accurately The degree will decrease, and the optical sensor 12 needs at least two to see all the optical markers at the same time. The calibration procedure is as described above, for example, three-stage calibration, which is used to accurately calibrate two coordinate systems. The correction error, the iteration count and the final position of the optical marker can be displayed in the window of the training system, for example by the output device 5. The accuracy and reliability information can be used to remind users that the system needs to be recalibrated when the error is too large. After the coordinate system is corrected, the three-dimensional model is drawn at a frequency of 0.1 times per second, and the drawn result can be output to the output device 5 for display or printing.

After the training system is ready, the user can start the surgical training process. In the training process, first use a medical probe to find the site to be operated on. After finding the site to be operated on, the site is anesthetized. Then, expand the path from the outside to the surgical site, and after expansion, the scalpel is deepened along this path to the surgical site.

FIGS. 7A to 7D are schematic diagrams of the training process of the training system of the embodiment. The surgical training process includes four stages and the minimally invasive surgery training of the fingers is taken as an example for illustration.

As shown in FIG. 7A, in the first stage, the medical probe 21 is used to find the site to be operated on to confirm that the site to be operated on is in the training system. The surgical site is, for example, the pulley area (pulley), which can be judged by looking for the position of the metacarpophalangeal joints, the anatomical structures of the bones and tendons of the fingers, and the focus at this stage is whether the first pulley area (A1 pulley) is found. In addition, if the trainee does not move the medical probe for more than three seconds to determine the position, then the training system will automatically enter the next stage of scoring. During surgical training, the medical probe 21 is placed on the skin and kept in contact with the skin at the metacarpal joints (MCP joints) along the midline of the flexor tendon.

As shown in FIG. 7B, in the second stage, the surgical instrument 22 is used to open the path of the surgical area. The surgical instrument 22 is, for example, a needle. The needle is inserted to inject local anesthetic and expand the space. The process of inserting the needle can be performed under the guidance of continuous ultrasound images. This continuous ultrasound image is an artificial ultrasound image, which is like the aforementioned medical image 136. Since it is difficult to simulate regional anesthesia with a prosthetic hand, anesthesia is not specifically simulated.

As shown in FIG. 7C, in the third stage, the surgical instrument 23 is pushed in along the same path as the surgical instrument 22 in the second stage to create the trajectory required for hooking the knife in the next stage. The surgical instrument 23 is, for example, a dilator. In addition, if the trainee does not move the surgical instrument 23 for more than three seconds to determine the position, then the training system will automatically enter the next stage of scoring.

As shown in FIG. 7D, in the fourth stage, the surgical instrument 24 is inserted along the trajectory created in the third stage, and the pulley is divided by the surgical instrument 24. The surgical instrument 24 is, for example, a hook blade. The focus of the third stage is similar to that of the fourth stage. During the surgical training process, the vessels and nerves near both sides of the flexor tendon may be easily miscut. Therefore, the third stage and the fourth stage The focus of the stage is not only not contacting tendons, nerves and blood vessels, but also opening a track that is at least 2mm larger than the first pulley area to leave space for the hook knife to cut the pulley area.

In order to score the user's operations, the operations of each training phase must be quantified. First of all, the operation area during the operation is defined by the finger anatomy as shown in Figure 8A, which can be divided into an upper boundary and a lower boundary. Because most of the tissue on the tendon is fat and does not cause pain, the upper boundary of the surgical area can be defined by the skin of the palm, and the lower boundary is defined by the tendon. The proximal depth boundary is 10mm (average length of the first trochlear zone) from the metacarpal head-neck joint. The remote depth boundary is not important, because it has nothing to do with tendons, blood vessels, and nerves. The left and right boundaries are defined by the width of the tendon, and nerves and blood vessels are located on both sides of the tendon.

After the surgical area is defined, the scoring method for each training stage is as follows. In the first stage of FIG. 7A, the focus of the training is to find the target, for example, the target to be excised. Taking the finger as an example is the first pulley area (A1 Pulley). In the actual operation process, in order to have good ultrasound image quality, the angle between the medical probe and the bone spindle should be close to vertical, and the allowable angle deviation is ±30°. Therefore, the scoring formula for the first stage is as follows:

The first stage score = the score of the target object × its weight + the angle score of the probe × its weight

In the second stage as shown in Figure 7B, the focus of training is to use the needle to open the path of the surgical area. Since the pulley area surrounds the tendon, the distance between the main axis of the bone and the needle should be small. Therefore, the calculation formula for the second stage scoring is as follows:

Second stage score = opening score × its weight + needle angle score × its weight + distance from the main axis of the bone score × its weight

In the third stage, the focus of training is to insert a dilator that enlarges the surgical area into the finger. During the operation, the trajectory of the dilator must be close to the main axis of the bone. In order not to damage tendons, blood vessels and nerves, the dilator will not exceed the previously defined boundary of the surgical area. In order to expand a good trajectory of the surgical area, the angle between the expander and the main axis of the bone should be approximately parallel, and the allowable angle deviation is ±30°. Due to the space left for the hook knife to cut the first trolley area, the expander must be at least 2mm higher than the first trolley area. The third stage scoring formula is as follows:

The third stage score = higher than the pulley area score × its weight + expander angle score × its weight + distance from the main axis of the bone score × its weight + not leaving the surgical area score × its weight

In the fourth stage, the scoring conditions are similar to those in the third stage, except that the hook needs to be rotated 90°. This rule is added to the scoring at this stage. The scoring formula is as follows:

The fourth stage score = higher than the pulley area score × its weight + hook angle score × its weight + distance from the main axis of the bone score × its weight + not leaving the surgical area score × its weight + rotating hook score × its weight

In order to establish a scoring standard to score the user's surgical operation, it is necessary to define how to calculate the angle between the bone spindle and the medical appliance. For example, this calculation method is the same as calculating the angle between the palm normal and the direction vector of the medical appliance. First, find the main axis of the bone. As shown in Figure 8B, the three axes of the bone can be found by using Principal Components Analysis (PCA) on the bone from the computed tomography image. Among the three axes, the longest axis is taken as the main axis of the bone. However, the shape of the bone in the computed tomography image is not uniform, which causes the axis found by the principal component analysis and the palm normal to be not perpendicular to each other. Thus, as shown in FIG. 8C, instead of using principal component analysis on the bone, the skin on the bone can be used to find the palm normal using principal component analysis. Then, the angle between the bone spindle and the medical appliance can be calculated.

After calculating the angle between the bone main axis and the appliance, the distance between the bone main axis and the medical appliance also needs to be calculated. The distance calculation is similar to calculating the distance between the top and the plane of the medical appliance. The plane refers to the plane containing the bone main axis vector vector and the palm normal. , The schematic diagram of distance calculation is shown in Figure 8D. This plane can be obtained by the cross product of the palm normal vector D2 and the bone principal axis vector D1. Since these two vectors can be obtained in the previous calculation, the distance between the main axis of the bone and the appliance can be easily calculated.

As shown in FIG. 8E, FIG. 8E is a schematic diagram of the artificial medical image of the embodiment, and the tendon section and the skin section in the artificial medical image are marked with dotted lines. The tendon section and the skin section can be used to construct the model and the bounding box, the bounding box is used for collision detection, and the pulley area can be defined in the static model. By using collision detection, it is possible to determine the surgical area and determine whether the medical appliance crosses the pulley area. The average length of the first pulley area is about 1mm, and the first pulley area is located at the proximal end of the metacarpal head-neck (MCP) joint. The average thickness of the pulley area is about 0.3mm and surrounds the tendons.

FIG. 9A is a flowchart of generating artificial medical images according to an embodiment. As shown in FIG. 9A, the generated flow includes step S21 to step S24.

Step S21 is to extract the first set of bone skin features from the cross-sectional image data of the artificial limb. An artificial limb is the aforementioned surgical target object 3, which can be used as a limb for minimally invasive surgery training, such as a prosthetic hand. The cross-sectional image data report contains multiple cross-sectional images, and the cross-sectional reference images are computed tomography images or solid cross-sectional images.

Step S22 is to extract the second set of bone skin features from the medical image data. The medical image data is a three-dimensional ultrasound image, such as the three-dimensional ultrasound image of FIG. 9B, which is created by multiple planar ultrasound images. Medical image data are medical images taken of real organisms, not artificial limbs. The first group of bone skin features and the second group of bone skin features include multiple bone feature points and multiple skin feature points.

Step S23 is to establish feature registration data (registration) based on the first set of bone and skin features and the second set of bone and skin features. Step S23 includes: taking the first set of bone-skin features as a reference target (target); finding out the correlation function as the spatial alignment correlation data, where the correlation function satisfies the second set of bone-skin features to align with the reference target without being due to the first set of bones Disturbance caused by skin features and the second set of bone skin features. The correlation function is found through the algorithm of the maximum likelihood estimation problem (maximum likelihood estimation problem) and the maximum expectation algorithm (EM Algorithm).

Step S24 is to perform deformation processing on the medical image data according to the feature alignment data to generate artificial medical image data suitable for artificial limbs. The artificial medical image data is, for example, a three-dimensional ultrasound image, which still retains the characteristics of the organism in the original ultrasound image. Step S24 includes: generating a deformation function based on the medical image data and feature alignment data; applying a grid to the medical image data and obtaining multiple dot positions accordingly; deforming the dot positions according to the deformation function; based on the deformed dot positions, The medical image data is supplemented with corresponding pixels to generate a deformed image, and the deformed image is used as artificial medical image data. The deformation function is generated using the moving least square (MLS) method. The deformed image is generated using affine transform.

Through step S21 to step S24, by capturing the image characteristics of the real ultrasound image and the artificial hand computed tomography image, the corresponding point relationship of the deformation is obtained by image alignment, and then the image close to the real ultrasound is generated based on the artificial hand through the deformation The ultrasound retains the characteristics of the original live ultrasound image. If the artificial medical image data is a three-dimensional ultrasound image, a plane ultrasound image of a specific position or a specific section can be generated based on the position or section mapped by the three-dimensional ultrasound image.

As shown in FIG. 10A and FIG. 10B, FIG. 10A and FIG. 10B are schematic diagrams of the correction of the artificial hand model and the ultrasound volume of the embodiment. The physical medical image 3D model 14b and the artificial medical image 3D model 14c are related to each other. Since the model of the prosthetic hand is constructed by the computed tomography image volume, the positional relationship between the computed tomography image volume and the ultrasound volume can be directly used to integrate the artificial hand. Establish correlation with ultrasound volume.

As shown in FIGS. 10C and 10D, FIG. 10C is a schematic diagram of ultrasonic volume and collision detection according to an embodiment, and FIG. 10D is a schematic diagram of an artificial ultrasound image according to an embodiment. The training system must be able to simulate a real ultrasonic transducer (or probe) and generate slice image fragments from the ultrasonic volume. Regardless of the angle of the transducer (or probe), the simulated transducer (or probe) must depict the corresponding image segment. In the implementation, the angle between the medical probe 21 and the ultrasonic body is first detected, and then the collision detection of the segment surface is based on the width of the medical probe 21 and the ultrasonic volume, which can be used to find the corresponding image segment being drawn. Value, the resulting image is shown in Figure 10D. For example, if the artificial medical image data is a three-dimensional ultrasound image, the three-dimensional ultrasound image has a corresponding ultrasound volume, and the content of the image segment to be depicted by the simulated transducer (or probe) can be generated according to the position of the three-dimensional ultrasound image mapping.

The above description is merely illustrative and not restrictive. Any equivalent modifications or changes made without departing from the spirit and scope of the present invention shall be included in the appended claims.

Claims

An optical tracking system for medical appliances, including:

A plurality of optical markers arranged on the medical appliance;

A plurality of optical sensors, optically sensing the optical markers to generate a plurality of sensing signals respectively; and

A computer device, coupled to the optical sensor to receive the sensing signal, has a three-dimensional model of the surgical situation, and adjusts the relative between the medical appliance present and the surgical target present in the three-dimensional model of the surgical situation according to the sensing signal position.
The system according to claim 1, wherein there are at least two optical sensors, which are arranged above the medical appliance and face the optical marker.
The system according to claim 1, wherein the computer device and the optical sensor perform a pre-operation program, and the pre-operation program includes:

Correcting the coordinate system of the optical sensor; and

Adjust the zoom ratio for the medical appliance and the surgical target object.
The system according to claim 1, wherein the computer device and the optical sensor perform a coordinate calibration program, and the calibration program includes:

The initial calibration step is to perform an initial calibration between the coordinate system of the optical sensor and the coordinate system of the three-dimensional model of the surgical situation to obtain initial conversion parameters;

An optimization step, optimizing the degrees of freedom of the initial conversion parameters to obtain optimized conversion parameters; and

The correcting step is to correct the setting error of the optical marker caused by the optimized conversion parameter.
The system according to claim 4, wherein the initial correction step uses singular value decomposition, triangular coordinate alignment, or linear least mean square estimation.
The system according to claim 4, wherein the initial correction step is to use singular value decomposition to find a transformation matrix between the characteristic points of the medical appliance presentation and the optical sensor as the initial transformation parameter, The transformation matrix includes a covariance matrix and a rotation matrix, wherein the optimization step is to obtain multiple Euler angles of multiple degrees of freedom from the rotation matrix, and use Gauss-Newton method to iteratively optimize the parameters of multiple degrees of freedom, To obtain the optimized conversion parameters.
The system according to claim 4, wherein the computer device sets the medical appliance presentation and the surgical target presentation in the three-dimensional model of the surgical situation according to the optimized conversion parameter and the sensing signal s position.
The system according to claim 4, wherein the correcting step is to correct the positions of the medical appliance presentation and the surgical target presentation in the three-dimensional model of the surgical situation by using reverse conversion and the sensing signal .
The system according to claim 1, wherein the computer device outputs display data, and the display data is used to present 3D images of the medical appliance presentation and the surgical target presentation.
The system according to claim 1, wherein the computer device generates medical images according to the three-dimensional model of the surgical situation and the medical image model.
The system according to claim 10, wherein the surgical target object is an artificial limb, and the medical image is an artificial medical image for the surgical target object.
The system according to claim 1, wherein the computer device deduces the position of the medical appliance inside and outside the surgical target object, and adjusts the medical appliance presentation and the medical appliance present in the three-dimensional model of the surgical situation accordingly. The relative position between the surgical target presents.
A training system for medical appliance operation, including:

Medical appliances; and

The optical tracking system according to one of claims 1 to 12, used in the medical appliance.
The system according to claim 13, wherein the medical appliance includes a medical probe and a surgical instrument, and the medical appliance presentation includes a medical probe presentation and a surgical instrument presentation.
The system according to claim 14, wherein the computer device performs a score based on the detection object found by the medical probe present and the operation of the surgical instrument present.
A correction method of an optical tracking system for medical appliances, including:

A sensing step, using a plurality of optical sensors of the optical tracking system to optically sense a plurality of optical markers arranged on the optical tracking system on the medical appliance to generate a plurality of sensing signals respectively;

The initial calibration step is to perform an initial calibration between the coordinate system of the optical sensor and the coordinate system of the three-dimensional model of the surgical situation according to the sensing signal to obtain initial conversion parameters;

An optimization step, optimizing the degrees of freedom of the initial conversion parameters to obtain optimized conversion parameters; and

The correction step is to correct the setting error of the optical marker caused by the optimized conversion parameter.
16. The method of claim 16, further comprising a pre-operation program, the pre-operation program comprising:

Correcting the coordinate system of the optical sensor; and

Adjust the zoom ratio for the medical appliance and the surgical target object.
The method according to claim 16, wherein the initial correction step uses singular value decomposition, triangular coordinate alignment or linear least mean square estimation.
The method of claim 16, wherein:

The initial correction step is to use singular value decomposition to find the transformation matrix between the feature points of the medical appliance presentation of the three-dimensional model of the surgical situation and the optical sensor as the initial transformation parameter, and the transformation matrix includes Variance matrix and rotation matrix,

The optimization step is to obtain multiple Euras angles with multiple degrees of freedom from the rotation matrix, and use Gauss-Newton method to iteratively optimize the parameters of multiple degrees of freedom to obtain the optimized conversion parameters.
The method of claim 16, wherein:

The positions of the medical appliance presentation and the surgical target presentation in the three-dimensional model of the surgical situation are set according to the optimized conversion parameter and the sensing signal,

The correcting step is to use the reverse conversion and the sensing signal to correct the positions of the medical appliance present and the surgical target present in the three-dimensional model of the surgical situation.