CN115454254A - Endoscopic guided transnasal virtual simulated surgery method and device - Google Patents

Endoscopic guided transnasal virtual simulated surgery method and device Download PDF

Info

Publication number
CN115454254A
CN115454254A CN202211167273.6A CN202211167273A CN115454254A CN 115454254 A CN115454254 A CN 115454254A CN 202211167273 A CN202211167273 A CN 202211167273A CN 115454254 A CN115454254 A CN 115454254A
Authority
CN
China
Prior art keywords
model
virtual
force feedback
transnasal
force
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202211167273.6A
Other languages
Chinese (zh)
Inventor
杨峰
蒋嘉熙
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Ariemedi Medical Science Beijing Co ltd
Original Assignee
Ariemedi Medical Science Beijing Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Ariemedi Medical Science Beijing Co ltd filed Critical Ariemedi Medical Science Beijing Co ltd
Priority to CN202211167273.6A priority Critical patent/CN115454254A/en
Publication of CN115454254A publication Critical patent/CN115454254A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/016—Input arrangements with force or tactile feedback as computer generated output to the user
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
    • G06F3/0487—Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser
    • G06F3/0488—Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser using a touch-screen or digitiser, e.g. input of commands through traced gestures
    • G—PHYSICS
    • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09B—EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS
    • G09B23/00—Models for scientific, medical, or mathematical purposes, e.g. full-sized devices for demonstration purposes
    • G09B23/28—Models for scientific, medical, or mathematical purposes, e.g. full-sized devices for demonstration purposes for medicine
    • G09B23/285—Models for scientific, medical, or mathematical purposes, e.g. full-sized devices for demonstration purposes for medicine for injections, endoscopy, bronchoscopy, sigmoidscopy, insertion of contraceptive devices or enemas
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F2203/00—Indexing scheme relating to G06F3/00 - G06F3/048
    • G06F2203/01—Indexing scheme relating to G06F3/01
    • G06F2203/014—Force feedback applied to GUI
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F2203/00—Indexing scheme relating to G06F3/00 - G06F3/048
    • G06F2203/041—Indexing scheme relating to G06F3/041 - G06F3/045
    • G06F2203/04105—Pressure sensors for measuring the pressure or force exerted on the touch surface without providing the touch position

Abstract

The method and the device for the virtual simulation operation through the nasal cavity under the guidance of the endoscope comprise the following steps: (1) Obtaining a CT image of a patient, and obtaining a preliminary three-dimensional virtual patch model with physiological characteristics based on threshold segmentation; (2) Obtaining a patient-based patch model, then tetrahedrizing the patient-based patch model, changing the hardness degree of different regions of the model according to physiological characteristics, setting specific flexibility coefficients for the different regions of the model, obtaining boundary states of the different regions, and combining a soft force feedback model with a visual model to synchronize visual change and force sense change of the model; (3) And traversing detection to obtain the position information of the collision point of the surgical tool and the organ tissue, calculating the physical information of speed, force and damping at different moments, and transmitting the physical information to a force feedback device so that a user can feel the mechanical properties of the model in real time.

Description

Endoscopic guided transnasal virtual simulated surgery method and device
Technical Field
The invention relates to the technical field of medical image processing, in particular to a transnasal virtual simulation operation method under the guidance of an endoscope and a transnasal virtual simulation operation device under the guidance of the endoscope.
Background
Virtual reality medical training techniques have become mature and are used by a variety of surgeons, and the main purpose of the virtual reality medical training techniques is to provide a real, immersive and risk-free surgical training environment for doctors and shorten the learning period of the doctors. With the development of biomedicine, the requirement of 'precision' and 'minimally invasive' in the operation process is increasing, which leads to the increase of operations guided by endoscopes year by year. However, a physician who is skilled in the operation of an endoscope requires a lot of practice and trial and error, which causes many ethical and even safety problems. Therefore, the operation training device under the guidance of the endoscope has strong practical significance.
A large number of operations are performed by using an operation tool to enter the head and neck through the nasal cavity, so that the invention is dedicated to the training of the transnasal operations under the guidance of the endoscope, and the existing training device for the operations of the head and neck under the guidance of the endoscope is few and has a plurality of problems. The most important of them are: 1. the authenticity of a pure simulation model based on computer graphics is difficult to guarantee; 2. the method is lack of a system for performing checkpoint design on key points of different operators based on the guidance of professional doctors, so that the feedback scoring system of operators can be corrected and recorded in real time; 3. the details of the specific anatomy are unclear, the force feedback operation is difficult to correspond to the real picture position, and the consumed computing resources are too large to meet the real-time performance.
Chinese patent 202111533867.x makes a complete set of medical simulation teaching system with force feedback, which has a three-dimensional modeling module, a soft tissue modeling module, and a collision detection and force feedback module similar to this patent, and it is mentioned that because of limited computing resources, in order to ensure the real-time performance of visual and tactile simulation, the simulation of tissue and organ has to be simplified, and such processing method is quite normal, but usually results in the interaction with a finite element complex model, which is impossible for force feedback equipment, with high reality; in addition, the most important thing of medical training simulation is that 'teaching of a doctor' hand is simulated, the doctor usually corrects errors in student operations by the hand, and the real-time error correction function in operation training is hardly mentioned by the existing nasal cavity operation simulation equipment under the guidance of an endoscope; secondly, for example, chinese patent 201520482322.4 manufactures a set of nose endoscope operation navigation guideline training system with force feedback, and a real nose endoscope entity and a guideline operation instrument entity are adopted, so that the force feedback difficulty is greatly reduced, and the cost of a simulation system is reduced.
Disclosure of Invention
In order to overcome the defects of the prior art, the technical problem to be solved by the invention is to provide a method for performing virtual simulated surgery through nasal cavity under the guidance of an endoscope, which can integrate key points in the real surgery process, pertinently construct a virtual model and a real model with high-precision anatomical features, and realize highly immersive virtual simulated surgery drilling by combining virtual vision and real tactile sensation in one-to-one manner by combining a force feedback device with a real surgical tool.
The technical scheme of the invention is as follows: the endoscopic-guided virtual simulation operation method for the nasal cavity comprises the following steps:
(1) Obtaining a CT image of a patient, and obtaining a preliminary three-dimensional virtual patch model with physiological characteristics based on threshold segmentation;
(2) After a patient-based patch model is obtained, the patch model is tetrahedrized, the hardness degree of different regions of the model is changed according to physiological characteristics, specific flexibility coefficients are set for the different regions of the model, the boundary states of the different regions are obtained, and a soft force feedback model and a visual model are combined to enable the visual change and the force sense change to be synchronous;
(3) And traversing, detecting and acquiring the position information of the collision point of the surgical tool and the organ tissue, calculating the physical information of speed, force and damping at different moments, and transmitting the physical information to a force feedback device so that a user can feel the mechanical properties of the model in real time.
The method comprises the steps of firstly obtaining a CT image of a patient, and obtaining a preliminary three-dimensional virtual patch model with physiological characteristics based on threshold segmentation; after a patient-based patch model is obtained, the patch model is tetrahedrized, the hardness degree of different regions of the model is changed according to physiological characteristics, specific flexibility coefficients are set for the different regions of the model, the boundary states of the different regions are obtained, and a soft force feedback model and a visual model are combined to enable the visual change and the force sense change to be synchronous; traversing detection to obtain the position information of the collision point of the surgical tool and the organ tissue, calculating the physical information of speed, force and damping at different moments, and transmitting the information to a force feedback device so that a user can feel the mechanical properties of the model in real time; therefore, the invention can integrate key points in the real operation process, pertinently construct a virtual model and a real model with high-precision anatomical features, and realize highly-immersive virtual simulation operation drill combining virtual vision and real touch by combining one-to-one force feedback equipment with a real operation tool.
There is also provided an endoscopically guided transnasal virtual simulated surgical device comprising:
the three-dimensional modeling module is used for obtaining a CT image of a patient and obtaining a preliminary three-dimensional virtual patch model with physiological characteristics based on threshold segmentation;
the soft tissue modeling module is used for obtaining a patient-based patch model, then making the patient-based patch model tetrahedron, changing the hardness degree of different regions of the model according to physiological characteristics, setting specific flexibility coefficients for the different regions of the model, obtaining the boundary states of the different regions, and combining the soft force feedback model with the visual model to synchronize the visual change and the force sense change of the model;
and the collision detection and force feedback module is used for traversing detection to obtain the position information of the collision point of the surgical tool and the organ tissue, calculating the physical information of speed, force and damping at different moments, and transmitting the physical information to the force feedback device so that a user can feel the mechanical properties of the model in real time.
Drawings
Fig. 1 is a flow chart of a method of endoscopic guided trans-nasal virtual simulation surgery according to the present invention.
Detailed Description
As shown in fig. 1, the endoscopic guided virtual simulation surgery method for nasal cavity comprises the following steps:
(1) Obtaining a CT image of a patient, and obtaining a preliminary three-dimensional virtual patch model with physiological characteristics based on threshold segmentation;
(2) After a patient-based patch model is obtained, the patch model is tetrahedrized, the hardness degree of different regions of the model is changed according to physiological characteristics, specific flexibility coefficients are set for the different regions of the model, the boundary states of the different regions are obtained, and a soft force feedback model and a visual model are combined to enable the visual change and the force sense change to be synchronous;
(3) And traversing detection to obtain the position information of the collision point of the surgical tool and the organ tissue, calculating the physical information of speed, force and damping at different moments, and transmitting the physical information to a force feedback device so that a user can feel the mechanical properties of the model in real time.
The method comprises the steps of firstly obtaining a CT image of a patient, and obtaining a preliminary three-dimensional virtual patch model with physiological characteristics based on threshold segmentation; after a patient-based patch model is obtained, the patch model is tetrahedrized, the hardness degree of different regions of the model is changed according to physiological characteristics, specific flexibility coefficients are set for the different regions of the model, the boundary states of the different regions are obtained, and a soft force feedback model and a visual model are combined to enable the visual change and the force sense change to be synchronous; traversing detection to obtain the position information of the collision point of the surgical tool and the organ tissue, calculating the physical information of speed, force and damping at different moments, and transmitting the information to a force feedback device so that a user can feel the mechanical properties of the model in real time; therefore, the invention can integrate key points in the real operation process, pertinently construct a virtual model and a real model with high-precision anatomical features, and realize highly-immersive virtual simulation operation drill combining virtual vision and real touch by combining one-to-one force feedback equipment with a real operation tool.
It will be understood by those skilled in the art that all or part of the steps in the method of the above embodiments may be implemented by hardware instructions related to a program, the program may be stored in a computer-readable storage medium, and when executed, the program includes the steps of the method of the above embodiments, and the storage medium may be: ROM/RAM, magnetic disks, optical disks, memory cards, and the like. Thus, in accordance with the method of the present invention, the invention also includes an endoscopically guided virtual simulated nasal surgical device, typically in the form of functional modules corresponding to the steps of the method. The device includes:
the three-dimensional modeling module is used for obtaining a CT image of a patient and obtaining a preliminary three-dimensional virtual patch model with physiological characteristics based on threshold segmentation;
the soft tissue modeling module is used for obtaining a patient-based patch model, then making the patient-based patch model tetrahedron, changing the hardness degree of different regions of the model according to physiological characteristics, setting specific flexibility coefficients for the different regions of the model, obtaining the boundary states of the different regions, and combining the soft force feedback model with the visual model to synchronize the visual change and the force sense change of the model;
and the collision detection and force feedback module is used for traversing, detecting and acquiring the position information of the collision point of the surgical tool and the organ tissue, calculating the physical information of speed, force and damping at different moments, and transmitting the physical information to the force feedback device so that a user can feel the mechanical properties of the model in real time.
Preferably, the three-dimensional model modeling module is used for modeling by using CT data or MRI data in DICOM format, wherein the CT data is set to be 1.00mm thick for shooting, and the target face of the object is located below the sinus ostium of the nasopteria and stored in DICOM format.
Preferably, the soft tissue modeling module performs threshold screening and segmentation according to different gray scale ranges of different tissues based on an MITK class library to obtain a preliminary model, performs grid refinement on key parts based on the opinion of a professional doctor, performs grid deletion on non-key parts, and adopts stretching and rotation to make the final model conform to the physiological structure characteristics, and finally stores the final model as a stl-format triangular patch file.
Preferably, the soft tissue modeling module divides tissues into four types, namely muscle, fat, mucous membrane and bone, different biomechanical parameters and constraints of the models in an external force application state are given to different types of models, and the biomechanical parameters of different parts in the same organ are adjusted at will.
Preferably, the collision detection and force feedback module judges a collision position between the virtual surgical instrument and the virtual model and a collision physical parameter so as to guide the input of the values of the model deformation and the force feedback.
Preferably, the virtual surgical instrument is modeled according to a real surgical instrument model 1 to 1, wherein the collision detection model is given to an AABB algorithm to judge whether collision occurs, in each frame of program operation, the collision point between the surgical tool and the physical model is detected in a circulating mode, the information of the collision point is obtained, collision detection boxes of all the instruments are manufactured to be larger than the diameter of the collision detection boxes by 0.1 mm, and after the information of collision speed, direction, force magnitude, momentum and the like of certain points at a certain moment is obtained, numerical values are input into the force feedback model to complete the physical simulation of force feedback.
Preferably, the collision detection and force feedback module uses a 3D-system Touch pen device, which is connected with a computer through a USB interface and is provided with an open source OpenHaptics driver kit.
Preferably, the model and the scene thereof are rendered in color, texture and illumination, and are updated in real time when the model is deformed or displaced.
Preferably, the apparatus further comprises: the device comprises a display device, a force feedback device, a real body model device and an auxiliary input device;
a display device is linked with the virtual surgery simulation for basic visual interaction; the force feedback device is used for inputting and interacting with the sense of touch; the real body models correspond to virtual models in virtual simulation one by one through position calculation and are fixed in the center of the equipment; the auxiliary input device is connected with the process auxiliary model.
The embodiment of the application provides an endoscope-guided nasal cavity virtual simulation surgical device, which has physiological characteristics, is designed according to customized force feedback caused by different biomechanical characteristics of different regions of a nasal cavity, is designed according to visual rendering, meets the force feedback virtual surgical requirement in a real scene, has simple repeatability and convenient designability in the process, and can flexibly design a surgical scheme and a training flow according to different operations.
The above description is only a preferred embodiment of the present invention, and is not intended to limit the present invention in any way, and all simple modifications, equivalent variations and modifications made to the above embodiment according to the technical spirit of the present invention still belong to the protection scope of the technical solution of the present invention.

Claims (10)

1. The endoscopic guided transnasal virtual simulation operation method is characterized in that: which comprises the following steps:
(1) Obtaining a CT image of a patient, and obtaining a preliminary three-dimensional virtual patch model with physiological characteristics based on threshold segmentation;
(2) After a patient-based patch model is obtained, the patch model is tetrahedrized, the hardness degree of different regions of the model is changed according to physiological characteristics, specific flexibility coefficients are set for the different regions of the model, the boundary states of the different regions are obtained, and a soft force feedback model and a visual model are combined to enable the visual change and the force sense change to be synchronous;
(3) And traversing detection to obtain the position information of the collision point of the surgical tool and the organ tissue, calculating the physical information of speed, force and damping at different moments, and transmitting the physical information to a force feedback device so that a user can feel the mechanical properties of the model in real time.
2. The endoscope-guided transnasal virtual simulation surgical device is characterized in that: it includes: the three-dimensional modeling module is used for obtaining a CT image of a patient and obtaining a preliminary three-dimensional virtual patch model with physiological characteristics based on threshold segmentation;
the soft tissue modeling module is used for obtaining a patient-based patch model, then making the patient-based patch model tetrahedron, changing the hardness degree of different regions of the model according to physiological characteristics, setting specific flexibility coefficients for the different regions of the model, obtaining the boundary states of the different regions, and combining the soft force feedback model with the visual model to synchronize the visual change and the force sense change of the model;
and the collision detection and force feedback module is used for traversing detection to obtain the position information of the collision point of the surgical tool and the organ tissue, calculating the physical information of speed, force and damping at different moments, and transmitting the physical information to the force feedback device so that a user can feel the mechanical properties of the model in real time.
3. The endoscopically guided, transnasal virtual simulated surgical device of claim 2, wherein: the three-dimensional model modeling module models with CT data or MRI data in DICOM format, the CT data is shot with the set layer thickness of 1.00mm, the target face reaches the lower part of the sinus ostium of the nose butterfly, and the target face is stored in a DICOM format.
4. The endoscopically guided, transnasal virtual simulated surgical device of claim 3, wherein: the soft tissue modeling module is based on an MITK class library, threshold screening and segmentation are carried out according to different gray scale ranges of different tissues, after a preliminary model is obtained, mesh refinement is carried out on key parts based on the opinions of professional doctors, mesh deletion is carried out on non-key parts, stretching and rotating are adopted, the final model is made to accord with the physiological structure characteristics, and finally the final model is stored as a triangular patch file in stl format.
5. The endoscopically guided, transnasal virtual simulated surgical device of claim 4, wherein: the soft tissue modeling module divides tissues into four types, namely muscle, fat, mucous membrane and bone, gives different biomechanical parameters to different types of models and restricts the models in an external force application state, and adjusts the biomechanical parameters of different parts in the same organ at will.
6. The endoscopically guided, transnasal virtual simulated surgical device of claim 5, wherein: and the collision detection and force feedback module is used for judging the collision position and the collision physical parameters between the virtual surgical instrument and the virtual model so as to guide the input of the values of model deformation and force feedback.
7. The endoscopically guided, transnasal virtual simulated surgical device of claim 6, wherein: the virtual surgical instrument is modeled and manufactured according to a real surgical instrument model 1 to 1, wherein a collision detection model is used for judging whether collision is caused by an AABB algorithm, in each frame of program operation, collision points of a surgical tool and a physical model are detected in a circulating mode, collision point information of the surgical tool and the physical model is obtained, collision detection boxes of all the instruments are manufactured to be larger than the diameter of the instruments by 0.1 mm, and after information of collision speed, direction, force magnitude, momentum and the like of certain points at a certain moment is obtained, numerical values are input into a force feedback model to complete physical simulation of force feedback.
8. The endoscopically guided, transnasal virtual simulated surgical device of claim 7, wherein: the collision detection and force feedback module uses a 3D-system Touch pen device which is connected with a computer through a USB interface and is provided with an open-source OpenHaptics driving tool kit.
9. The endoscopically guided, transnasal virtual simulated surgical device of claim 8, wherein: and rendering the model and the scene thereof by color, texture and illumination, and updating the model in real time when the model is deformed or displaced.
10. The endoscopically guided, transnasal virtual simulated surgical device of claim 9, wherein: the device also includes: the device comprises a display device, a force feedback device, a real body model device and an auxiliary input device;
the display device is linked with the virtual surgery simulation and used for basic visual interaction; the force feedback device is used for inputting and interacting with the sense of touch; the real body models are in one-to-one correspondence with virtual models in virtual simulation through position calculation and are fixed in the center of the equipment; the auxiliary input device is connected with the process auxiliary model.
CN202211167273.6A 2022-09-23 2022-09-23 Endoscopic guided transnasal virtual simulated surgery method and device Pending CN115454254A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202211167273.6A CN115454254A (en) 2022-09-23 2022-09-23 Endoscopic guided transnasal virtual simulated surgery method and device

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202211167273.6A CN115454254A (en) 2022-09-23 2022-09-23 Endoscopic guided transnasal virtual simulated surgery method and device

Publications (1)

Publication Number Publication Date
CN115454254A true CN115454254A (en) 2022-12-09

Family

ID=84306405

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202211167273.6A Pending CN115454254A (en) 2022-09-23 2022-09-23 Endoscopic guided transnasal virtual simulated surgery method and device

Country Status (1)

Country Link
CN (1) CN115454254A (en)

Similar Documents

Publication Publication Date Title
Condino et al. How to build a patient-specific hybrid simulator for orthopaedic open surgery: benefits and limits of mixed-reality using the Microsoft HoloLens
CN107067856B (en) Medical simulation training system and method
Basdogan et al. VR-based simulators for training in minimally invasive surgery
Ruthenbeck et al. Virtual reality for medical training: the state-of-the-art
Delingette et al. Hepatic surgery simulation
Tolsdorff et al. Virtual reality: a new paranasal sinus surgery simulator
Linke et al. Assessment of skills using a virtual reality temporal bone surgery simulator
US20210233429A1 (en) Mixed-reality endoscope and surgical tools with haptic feedback for integrated virtual-reality visual and haptic surgical simulation
KR102536732B1 (en) Device and method for the computer-assisted simulation of surgical interventions
CN113035038A (en) Virtual orthopedic surgery exercise system and simulation training method
Tai et al. Intelligent intraoperative haptic-AR navigation for COVID-19 lung biopsy using deep hybrid model
CN116869651A (en) Device and method for providing operation simulation based on virtual reality
Riener et al. VR for medical training
CN112587233B (en) Method and system for simulating endotracheal intubation
WO2007019546A2 (en) System, device, and methods for simulating surgical wound debridements
CN106409103B (en) A kind of VR medical treatment experiencing system based on transient noise noise-removed technology and correcting technology
Wagner et al. Intraocular surgery on a virtual eye
CN115457008A (en) Real-time abdominal puncture virtual simulation training method and device
CN114267219B (en) Virtual operation simulation device and simulation system
Müller-Wittig Virtual reality in medicine
CN115454254A (en) Endoscopic guided transnasal virtual simulated surgery method and device
Rahman et al. Augmented and virtual reality based approaches in minimally invasive surgery training
Obeid et al. Development and validation of a hybrid nuss procedure surgical simulator and trainer
Rasool et al. Image-driven haptic simulation of arthroscopic surgery
CN115662234B (en) Thoracic surgery teaching system based on virtual reality

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination