CN108492657B - A kind of mixed reality simulation system trained before the surgery for temporal bone - Google Patents
A kind of mixed reality simulation system trained before the surgery for temporal bone Download PDFInfo
- Publication number
- CN108492657B CN108492657B CN201810242877.XA CN201810242877A CN108492657B CN 108492657 B CN108492657 B CN 108492657B CN 201810242877 A CN201810242877 A CN 201810242877A CN 108492657 B CN108492657 B CN 108492657B
- Authority
- CN
- China
- Prior art keywords
- real
- virtual
- force
- soft tissue
- touch device
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Expired - Fee Related
Links
Classifications
-
- G—PHYSICS
- G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
- G09B—EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS
- G09B9/00—Simulators for teaching or training purposes
-
- G—PHYSICS
- G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
- G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- G16H20/00—ICT specially adapted for therapies or health-improving plans, e.g. for handling prescriptions, for steering therapy or for monitoring patient compliance
- G16H20/40—ICT specially adapted for therapies or health-improving plans, e.g. for handling prescriptions, for steering therapy or for monitoring patient compliance relating to mechanical, radiation or invasive therapies, e.g. surgery, laser therapy, dialysis or acupuncture
Landscapes
- Engineering & Computer Science (AREA)
- Health & Medical Sciences (AREA)
- Theoretical Computer Science (AREA)
- Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
- Urology & Nephrology (AREA)
- Educational Technology (AREA)
- General Physics & Mathematics (AREA)
- Physics & Mathematics (AREA)
- Business, Economics & Management (AREA)
- Surgery (AREA)
- Educational Administration (AREA)
- Epidemiology (AREA)
- General Health & Medical Sciences (AREA)
- Medical Informatics (AREA)
- Primary Health Care (AREA)
- Public Health (AREA)
- Processing Or Creating Images (AREA)
- Instructional Devices (AREA)
Abstract
The mixed reality simulation system trained before a kind of surgery for temporal bone is grouped as by real part, virtual part and systems control division；The reality part is made of allodynia device, real model bone structure layer, real part allodynia device is outputting and inputting for entire mixed reality simulation system, wherein operating stick is contacted with real model bone structure layer, and doctor is made to have indeed vibrations sense and operation telepresenc；The virtual part is mainly made of computer and display, for constructing virtual temporal bone model, and calculates soft tissue power everywhere and moment information；Device, Galil motion control card and computer composition is mainly driven by motor in the systems control division point, detect position/appearance of allodynia device in actual environment, collision detection is carried out in conjunction with virtual part data, the power and torque for exporting soft tissue, along with allodynia device gravity, the power of Friction Compensation and torque are converted to expected force and torque output allodynia device by processing.
Description
Technical Field
The invention relates to a mixed reality simulation system for preoperative training of doctors, in particular to a mixed reality simulation system for preoperative training of temporal bones.
Background
The temporal bone is one of the most complex anatomical parts of the human body. It is in close proximity to the outer surface of the brain, protecting and surrounding several important soft tissues, such as arteries, veins and facial nerves. Surgical procedures are required on the temporal bone when removing tumors and cysts, repairing bone fractures, restoring hearing, correcting balance disorders, and preventing infection. Physicians need great training before entering the clinic, or there is a risk that permanent deformities, hearing loss and potential death may result.
The corpse is a main teaching tool for the traditional temporal bone surgery training. However, this approach is limited by the number of cadavers, cost and risk of disease transmission. The more novel teaching method is to use a simulator, and at present, a physical model, a virtual reality simulator and a virtual reality simulator with force feedback are mainly used for training the temporal bone surgery. (1) Physical model: such models approach cadaver conditions, primarily for practical training. These models are mainly rapid prototyping models and plastic models. Such as the "temporalis bone of petasites plastic", is widely considered as a substitute for cadaver training. However, one of the training problems is that the model cannot be reused, is used only once, is expensive and is not cost-effective. (2) Virtual reality simulator: such simulators are composed of a conventional human-computer interaction interface keyboard and mouse. They provide a graphical interpretation of simulated physical conditions, however the main problem with this type of simulator is the inability to provide true vibration and contact forces. (3) Virtual reality simulator with force feedback: this type of simulator has a virtual reality simulator with the capability of contact force feedback. When a intern carries out simulation operation through the virtual environment of the graphical interface, the function enables the intern to have contact force perception capability, but the intern cannot provide vibration sense in the drilling process, and the training effect is greatly reduced. Therefore, the vibration sense in the drilling process is obtained, and the real on-site clinical sense of a doctor has important significance for the effect of doctor training operation.
The invention provides a mixed reality simulation system for training before a temporal bone surgery, which comprises a real part, a virtual part and a system control part. The mixed reality simulation system can provide real vibration feeling and contact force perception, provides a teaching tool for the clinical training of interns, and has important theoretical significance and practical application value in research.
Disclosure of Invention
The invention aims to provide a mixed reality simulation system for preoperative training of temporal bone surgery, which has real vibration sense and contact force sensing capability, comprises a touch device which can provide 3 position degrees of freedom and 2 posture degrees of freedom and can compensate gravity of the touch device, and realizes the same surgery scene as a real surgery by combining soft tissue force feedback and an actual physical model in a virtual environment.
In order to achieve the functional purposes, the technical scheme adopted by the invention is as follows:
a mixed reality simulation system for preoperative training of temporal bone surgery, comprising: a real part, a virtual part and a system control part.
Reality part is including real model bone structure layer and sense of touch device, the sense of touch device includes base, parallel four-bar mechanism, big arm mechanism, forearm mechanism, wrist mechanism and action bars. Real model bone structure layer part is the simple and easy physical model that 3D printed, is equivalent to the temporal bone structural layer, for the action bars provides an object of operation. Parallel four-bar mechanism keeps flat, self guarantees gravity balance, and one end links to each other with the base, and one end links to each other with big arm mechanism, and big arm mechanism, forearm mechanism can feel the power of staff with the gravity of wrist mechanism itself and exert an influence, and the gravity compensation need be carried out to it in the event, nevertheless because big arm mechanism, forearm mechanism are lighter with the weight of wrist mechanism itself, adopts driving motor to carry out the gravity compensation to it, therefore has realized the complete gravity balance of whole sense of touch device, the action bars is the handheld medical instrument of doctor, wherein installs the operation and bores, and action bars and the contact of real model bone structure layer make the doctor have true vibration sense and operation telepresence.
The virtual part is used for constructing a virtual temporal bone model by scanning the temporal bone model and importing data of various tissues, muscles, cartilages and the like of the real temporal bone into Unity3D simulation software, and the virtual temporal bone model is displayed by a display, so that the operation presence is visually increased; by using(FCFor friction, K (x) is the sum of the coefficients of elasticityDamping and viscosity coefficients) to construct a soft tissue mechanics model, and calculating force and moment information of each part of the soft tissue through the soft tissue mechanics model.
The system control part comprises a Galil multi-axis motion control card, a driver and a computer. The Galil multi-axis motion control card controls the motor torque through a driver to generate force feedback, combines a real model part and a virtual part to realize the function of the whole mixed reality simulation system, performs forward and inverse kinematics calculation on the touch device by using a D-H method, establishes a Jacobian matrix of the touch device and provides position/posture data for the movement of the surgical drill in a virtual environment; establishing a touch device dynamic model considering friction force and gravity by applying a Lagrange equation, simplifying the dynamic model by combining with actual conditions, mainly analyzing a gravity item and a friction force item, and performing compensation control on the gravity and the friction force of the touch device; the proportion and the speed controller are adopted for motion control, and the contact force perception and the vibration sense in the operation process of the touch device part are realized by superposing the contact force of the virtual soft tissue and the actual vibration force, so that the purpose of ideal temporal bone operation training is achieved. The control system detects the position/posture of the touch device in the real environment, performs collision detection by combining virtual environment data, outputs the force and moment of soft tissue, and obtains expected force and moment by processing and converting the force and moment compensated by gravity and friction of the touch device and outputs the expected force and moment to the touch device.
The invention has the following advantages:
1. the real part touch device has five degrees of freedom, is combined with a real model bone structure layer, has real vibration sense and flexible movement, can better complete complex actions and generates more real operation presence sense.
2. According to the method, a lagrangian equation is used for establishing a touch device dynamic model considering friction and gravity, the gravity item and the friction force item of the touch device are mainly analyzed, and the gravity and the friction force compensation control of the touch device is realized by adopting motor control, so that the operating rod is basically not bound by other forces when being held, and the hand feeling is more real.
3. According to the invention, a virtual temporal bone model is constructed by adopting Unity3D simulation software, and is displayed by a display, real contact force perception and vibration perception are obtained by combining a soft tissue mechanics model and a real model bone structure layer, the same operation scene as a real operation is realized in vision and touch, and the preoperative training of a temporal bone operation of a doctor is better realized.
4. The invention adopts a closed-loop control strategy, efficiently realizes a control system of the mixed reality simulator, and has higher level in the aspects of real-time performance, high precision and the like of control.
5. The invention only needs to be arranged on a stable plane, is convenient to fix, has small occupied space of the body structure, and is easy to install and miniaturized.
6. The invention can provide real vibration feeling and contact force perception, provides a teaching tool for pre-operation training of interns, and reduces the training cost of interns.
Drawings
FIG. 1 is a mixed reality simulation system experiment platform of the present invention;
FIG. 2 is a block diagram of a real part of the mixed reality simulation system of the present invention;
FIG. 3 is a schematic diagram of the experimental control of the mixed reality simulation system of the present invention;
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is further described with reference to the accompanying drawings.
The mixed reality simulation system experiment platform is shown in figure 1. The system mainly comprises a real part, a virtual part and a system control part. The reality part consists of a touch device and a real model bone structure layer. The joint freedom degree of the touch device mainly comprises a motor, an encoder and a potentiometer. The control system part mainly comprises a motor driver, a Galil multi-axis motion control card and a computer. The virtual part mainly comprises a computer and a display.
The structure diagram of the real part of the mixed reality simulation system of the invention is shown in figure 2. The touch device comprises a base, a parallel four-bar mechanism, a large arm mechanism, a small arm mechanism, a wrist mechanism and an operating rod. The parallel four-bar mechanism is horizontally placed, gravity balance is guaranteed by the parallel four-bar mechanism, one end of the parallel four-bar mechanism is connected with the base, the other end of the parallel four-bar mechanism is connected with the big arm mechanism, gravity of the big arm mechanism, the small arm mechanism and the wrist mechanism can influence the force feeling of hands, so that gravity compensation needs to be carried out on the big arm mechanism, the small arm mechanism and the wrist mechanism, the weight of the big arm mechanism, the weight of the small arm mechanism and the weight of the wrist mechanism are light, the gravity compensation is carried out on the big arm mechanism, the small arm mechanism and the wrist mechanism by adopting the driving motor, complete gravity.
The experimental control principle of the mixed reality simulation system is shown in fig. 3. The real part touch device is the input and output of the whole mixed reality simulation system, wherein the operating rod is in contact with the real model bone structural layer, so that a doctor has real vibration sense and operation on-site sense. Gathering haptic device position/attitude, through haptic deviceCalculating the position/posture of the surgical drill by the aid of the positional kinematics, inputting the position/posture into a virtual environment, calculating soft tissue force and moment information of a contact position in the virtual environment through an established soft tissue mechanical model, obtaining output force and moment of each joint needing to simulate real soft tissue through conversion calculation, establishing the kinematics of the touch device by a D-H method, establishing a temporal bone model in the virtual environment by aid of Unity3D simulation software, and establishing a soft tissue mechanical model by the aid of Unity3D simulation software(FCFor friction, K (x) is the sum of the coefficients of elasticityDamping and viscosity coefficients); the method comprises the steps of collecting positions/postures of a sensing device and calculating surgical drilling positions/postures, calculating force and moment required by gravity and friction force compensation of the touch device by considering touch device dynamics of friction and gravity, wherein the touch device dynamics considering friction and gravity is established by adopting a Lagrange equation; calculating to obtain the output force and moment of each joint needing to simulate real soft tissue and the force and moment needed by the gravity and friction compensation of the touch device, processing and converting to obtain the expected force and moment of each joint and outputting the expected force and moment to a real environment, and further performing motion control by adopting a proportion and speed controller, so that the real perception of a doctor on the contact force of the virtual soft tissue is realized; the real vibration sense of the real model and the real perception of the contact force of the virtual soft tissue are combined to achieve the purpose of mixed reality, so that the real operation environment is simulated, and the preoperative training of the temporal bone operation of a doctor is realized.
Claims (1)
1. A mixed reality simulation system for preoperative training of temporal bone surgery, comprising: real part, virtual part and system control part, its characterized in that: the real part touch device is input and output of the whole mixed reality simulation system, wherein the operating rod is in contact with the real model bone structural layer, so that a doctor has real vibration sense and operation presence sense, the position/posture of the operation drill is calculated through the touch device kinematics, and the operation drill is input into a virtual environment; the virtual part calculates the virtual through the established soft tissue mechanical modelSimulating force and moment information of a soft tissue contact position, obtaining output force and moment of each joint needing to simulate real soft tissue through conversion calculation, constructing a temporal bone model in the virtual environment by adopting Unity3D simulation software, and constructing a soft tissue mechanical model by adopting Unity3D simulation softwareConstruction, where Fc is the friction and K (x) is the coefficient of elasticity,damping and viscosity coefficients; the system control part calculates the position/posture of the operating rod through the collected position/posture of the sensing device, calculates the force and moment required by the compensation of gravity and friction of the touch device through the dynamics of the touch device considering friction and gravity items, calculates the force and moment required by the compensation of the gravity and friction of the touch device through Lagrange's equation, calculates the force and moment required by the compensation of each joint output force and moment required by the simulation of real soft tissue combined with the gravity and friction of the touch device, outputs the expected force and moment of each joint to a real environment through conversion, further performs motion control through a proportional speed controller, thereby realizing the real perception of a doctor on the contact force of the virtual soft tissue, combining the real vibration sense of a bone structure layer of a real model and the real perception on the contact force of the virtual soft tissue, and achieving the purpose of mixed reality, thereby simulating a real operation environment and realizing the preoperative training of the temporal bone operation of the doctor.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810242877.XA CN108492657B (en) | 2018-03-20 | 2018-03-20 | A kind of mixed reality simulation system trained before the surgery for temporal bone |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810242877.XA CN108492657B (en) | 2018-03-20 | 2018-03-20 | A kind of mixed reality simulation system trained before the surgery for temporal bone |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN108492657A CN108492657A (en) | 2018-09-04 |
| CN108492657B true CN108492657B (en) | 2019-09-20 |
Family
ID=63319488
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810242877.XA Expired - Fee Related CN108492657B (en) | 2018-03-20 | 2018-03-20 | A kind of mixed reality simulation system trained before the surgery for temporal bone |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108492657B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113920807A (en) * | 2021-10-20 | 2022-01-11 | 哈尔滨理工大学 | Bone cutting operation teaching and training system |
| CN115024824B (en) * | 2022-08-11 | 2022-11-11 | 珠海康弘医疗科技有限公司 | Working sheath rotating device for percutaneous spinal endoscope robot |
Family Cites Families (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| NL1035303C2 (en) * | 2008-04-16 | 2009-10-19 | Virtual Proteins B V | Interactive virtual reality unit. |
| JP5907596B2 (en) * | 2010-10-12 | 2016-04-26 | キヤノン株式会社 | Three-dimensional measuring apparatus, three-dimensional measuring method and program |
| US9330502B2 (en) * | 2012-09-12 | 2016-05-03 | Eidgenoessische Technische Hochschule Zurich (Eth Zurich) | Mixed reality simulation methods and systems |
| CN103106348A (en) * | 2013-03-08 | 2013-05-15 | 上海交通大学医学院附属第九人民医院 | Virtual surgery simulation method and device thereof |
| CN104658389A (en) * | 2013-11-18 | 2015-05-27 | 上海交通大学医学院附属第九人民医院 | Virtual orthognathic surgery training system and method |
| CN104992582B (en) * | 2015-07-13 | 2017-08-15 | 中国科学院自动化研究所 | A kind of medical science Minimally Invasive Surgery training system based on mixed reality |
| CN105303605B (en) * | 2015-10-26 | 2017-10-13 | 哈尔滨理工大学 | A kind of bone surgery analogue system based on force feedback |
| CN206711474U (en) * | 2017-03-28 | 2017-12-05 | 杭州爱卓科技有限公司 | A kind of medical simulation integrated machine equipment |
| CN107610574B (en) * | 2017-10-17 | 2023-04-07 | 上海褚信医学科技有限公司 | Device and method for simulating puncture surgery |
| CN108154778A (en) * | 2017-12-28 | 2018-06-12 | 深圳科创广泰技术有限公司 | Based on motion-captured and mixed reality ophthalmologic operation training system and method |
-
2018
- 2018-03-20 CN CN201810242877.XA patent/CN108492657B/en not_active Expired - Fee Related
Non-Patent Citations (2)
| Title |
|---|
| 增强/混合现实技术在头颈外科的应用及展望;陈刚;《组织工程与重建外科杂志》;20180228;全文 * |
| 混合现实技术在骨科的最新应用进展;叶哲伟;《临床外科杂志》;20180131;全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN108492657A (en) | 2018-09-04 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| JP6049788B2 (en) | Virtual tool operation system | |
| Ma et al. | Hand rehabilitation learning system with an exoskeleton robotic glove | |
| Rosen et al. | Upper limb powered exoskeleton | |
| CN102207997B (en) | Force-feedback-based robot micro-wound operation simulating system | |
| Srinivasan | What is haptics | |
| O’malley et al. | Haptic interfaces | |
| CN108492657B (en) | A kind of mixed reality simulation system trained before the surgery for temporal bone | |
| Wei et al. | Augmented optometry training simulator with multi-point haptics | |
| Kapur et al. | Vibrotactile feedback system for intuitive upper-limb rehabilitation | |
| Daniulaitis et al. | Medical palpation of deformable tissue using physics-based model for haptic interface robot (HIRO) | |
| Ma et al. | Sensing and force-feedback exoskeleton robotic (SAFER) glove mechanism for hand rehabilitation | |
| Mali et al. | Application of haptic interface for finger exercise | |
| Chen et al. | Dynamic touch‐enabled virtual palpation | |
| Covaciu et al. | VR interface for cooperative robots applied in dynamic environments | |
| Frey et al. | Physical interaction with a virtual knee joint—The 9 DOF haptic display of the Munich knee joint simulator | |
| Riener et al. | Haptic aspects | |
| Bitikofer et al. | Mapping ADL Motion Capture Data to BLUE SABINO Exoskeleton Kinematics and Dynamics | |
| WO2022190738A1 (en) | Electrical stimulation device and electrical stimulation method | |
| Kumar et al. | Haptic System its Constituents and Application in Advanced Technology | |
| Kim et al. | Physically based hybrid approach in real time surgical simulation with force feedback | |
| Kawasaki et al. | HIRO: Multi-fingered Haptic Interface Robot and Its Medical Application Systems | |
| Sharma et al. | Haptic Feedback–A Review | |
| Terada et al. | Virtual Human Body using Haptic Devices for Endoscopic Surgery Training Simulator | |
| Dauren et al. | Structural design of a rehabilitation robot for the upper limb | |
| Chen | The Virtual Haptic Human Upper Body for Palpatory Diagnostic Training |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20190920 Termination date: 20210320 |
|
| CF01 | Termination of patent right due to non-payment of annual fee |