US20170110032A1 - Ultrasound simulation system and tool - Google Patents
Ultrasound simulation system and tool Download PDFInfo
- Publication number
- US20170110032A1 US20170110032A1 US15/294,275 US201615294275A US2017110032A1 US 20170110032 A1 US20170110032 A1 US 20170110032A1 US 201615294275 A US201615294275 A US 201615294275A US 2017110032 A1 US2017110032 A1 US 2017110032A1
- Authority
- US
- United States
- Prior art keywords
- ultrasound
- model
- tool
- anatomy
- simulation
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Abandoned
Links
Images
Classifications
-
- G—PHYSICS
- G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
- G09B—EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS
- G09B23/00—Models for scientific, medical, or mathematical purposes, e.g. full-sized devices for demonstration purposes
- G09B23/28—Models for scientific, medical, or mathematical purposes, e.g. full-sized devices for demonstration purposes for medicine
- G09B23/281—Models for scientific, medical, or mathematical purposes, e.g. full-sized devices for demonstration purposes for medicine for pregnancy, birth or obstetrics
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B8/00—Diagnosis using ultrasonic, sonic or infrasonic waves
- A61B8/13—Tomography
- A61B8/14—Echo-tomography
- A61B8/145—Echo-tomography characterised by scanning multiple planes
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B8/00—Diagnosis using ultrasonic, sonic or infrasonic waves
- A61B8/42—Details of probe positioning or probe attachment to the patient
- A61B8/4245—Details of probe positioning or probe attachment to the patient involving determining the position of the probe, e.g. with respect to an external reference frame or to the patient
- A61B8/4254—Details of probe positioning or probe attachment to the patient involving determining the position of the probe, e.g. with respect to an external reference frame or to the patient using sensors mounted on the probe
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B8/00—Diagnosis using ultrasonic, sonic or infrasonic waves
- A61B8/44—Constructional features of the ultrasonic, sonic or infrasonic diagnostic device
- A61B8/4444—Constructional features of the ultrasonic, sonic or infrasonic diagnostic device related to the probe
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B8/00—Diagnosis using ultrasonic, sonic or infrasonic waves
- A61B8/46—Ultrasonic, sonic or infrasonic diagnostic devices with special arrangements for interfacing with the operator or the patient
- A61B8/461—Displaying means of special interest
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B8/00—Diagnosis using ultrasonic, sonic or infrasonic waves
- A61B8/48—Diagnostic techniques
- A61B8/483—Diagnostic techniques involving the acquisition of a 3D volume of data
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T19/00—Manipulating 3D models or images for computer graphics
- G06T19/006—Mixed reality
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformation in the plane of the image
- G06T3/40—Scaling the whole image or part thereof
- G06T3/4007—Interpolation-based scaling, e.g. bilinear interpolation
-
- G—PHYSICS
- G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
- G09B—EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS
- G09B23/00—Models for scientific, medical, or mathematical purposes, e.g. full-sized devices for demonstration purposes
- G09B23/28—Models for scientific, medical, or mathematical purposes, e.g. full-sized devices for demonstration purposes for medicine
- G09B23/286—Models for scientific, medical, or mathematical purposes, e.g. full-sized devices for demonstration purposes for medicine for scanning or photography techniques, e.g. X-rays, ultrasonics
-
- G—PHYSICS
- G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
- G09B—EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS
- G09B23/00—Models for scientific, medical, or mathematical purposes, e.g. full-sized devices for demonstration purposes
- G09B23/28—Models for scientific, medical, or mathematical purposes, e.g. full-sized devices for demonstration purposes for medicine
- G09B23/30—Anatomical models
-
- G—PHYSICS
- G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
- G09B—EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS
- G09B9/00—Simulators for teaching or training purposes
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B90/00—Instruments, implements or accessories specially adapted for surgery or diagnosis and not covered by any of the groups A61B1/00 - A61B50/00, e.g. for luxation treatment or for protecting wound edges
- A61B90/36—Image-producing devices or illumination devices not otherwise provided for
- A61B90/37—Surgical systems with images on a monitor during operation
- A61B2090/378—Surgical systems with images on a monitor during operation using ultrasound
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B90/00—Instruments, implements or accessories specially adapted for surgery or diagnosis and not covered by any of the groups A61B1/00 - A61B50/00, e.g. for luxation treatment or for protecting wound edges
- A61B90/39—Markers, e.g. radio-opaque or breast lesions markers
- A61B2090/3925—Markers, e.g. radio-opaque or breast lesions markers ultrasonic
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10132—Ultrasound image
- G06T2207/10136—3D ultrasound image
Definitions
- the present invention relates to computerized medical simulation in general, and more specifically to virtual reality and/or augmented reality simulation devices and systems for medical training purposes.
- Medical imaging has become more and more used for both diagnostic/examination and therapeutic purposes in a number of medical applications, such as endoscopy for surgery, or ultrasound imaging for various gynecology and/or obstetrics applications, for instance in the embryo transfer procedure for In Vitro Fertilization (IVF).
- IVF In Vitro Fertilization
- These new techniques may require dedicated training for physicians and surgeons to master the indirect hand-eye coordination required by the imaging system as well as the manipulation of the imaging tools in addition to the conventional medical instruments and procedures for a diversity of patient anatomies as may be encountered in medical practice.
- Computerized medical procedure training simulators may enable the physicians and trainees to develop and improve their practice in a virtual reality environment before actually practicing in the operation room.
- Advanced medical procedure simulators may be based on a virtual reality (“VR”) and/or a mixed or augmented reality (“AR”) simulation apparatus by which the physician can experiment a medical procedure scenario.
- the VR/AR system may compute and display a visual VR/AR model of anatomical structures in accordance with physician gestures and actions to provide various feedback, such as visual feedback.
- a VR system an entire image may be simulated for display to a user, and in an AR system, a simulated image may be overlaid or otherwise incorporated with an actual image for display to a user.
- Various patient models with different pathologies can be selected. Therefore, natural variations as encountered over the years by practicing doctors can be simulated for a user over a compressed period of time for training purposes.
- the medical simulation procedure can be recorded and rehearsed for evaluation purpose.
- the VR/AR simulation system can also compute and provide various metrics and statistics.
- VR/AR simulation systems such as the one described in U.S. Pat. No. 8,992,230 include a human anatomy model of a joint of organ in real size.
- the VR/AR simulation system may further comprise a medical instrument to more realistically simulate the medical procedure.
- the model is further adapted with sensors for tracking the position and/or orientation of both the anatomy model and the medical instrument.
- calibration units may be further used to automatically setup and align the VR/AR simulation system to a diversity of anatomy models and medical procedure training scenarios without requiring a cumbersome, manual calibration procedure each time a new model is adapted to the system.
- a passive feedback VR/AR simulation system such as for instance the one described in U.S. Pat. No. 8,992,230 may also be used with a diversity of medical procedure training scenarios, some of which may possibly result in a mismatch between an anatomy model surface as touched by the trainee and a virtual environment surface as computed by the VR/AR simulation system and rendered on the screen.
- the VR/AR simulation system may be further adapted with space warping methods and systems as described in US patent application US20140071165.
- Dror Aiger and Daniel Cohen-Or described the use of deformable registration techniques for generating large 3D volumes from smaller swept scans in “Real-time ultrasound imaging simulation”, Real - Time Imaging, 4(4):263-274, 1998.
- “Augmented reality simulator for training in two-dimensional echocardiography” Computers and Biomedical Research, 33:11-22, 2000, M. Weidenbach, C. Wick, S. Pieper, K. J. Quast, T. Fox, G. Grunst, and D. A. Redel proposed to register the recorded patient-specific volumes to a generic 3D anatomical heart model, which they thus call augmented training. All the above methods present the user a non-deformable model of the anatomy. In other words, although the images may change with the transducer position/orientation in some simulators, they do not change according to the trainee's interaction with the mannequin or the virtual model, which negatively impacts the simulation realism.
- deformable, interactive ultrasound simulation may generate a better sense of simulator realism and consequent trainee immersion.
- O. Goksel and S. E. Salcudean introduced the first interpolative simulator that allows for deformation by using a fast mapping technique for image pixels to be simulated, from the deformed simulation tissue coordinates to a nominal recorded volumetric coordinate frame.
- This method combines an input (reconstructed) 3D volume and interactive, volumetric tissue deformation models such as the finite element method and has been further applied to prostate brachytherapy simulation as published in Orcun Goksel, Kirill Sapchuk, and Septimiu E.
- Salcudean “Haptic simulator for prostate brachytherapy with simulated needle and probe interaction”, IEEE Trans Haptics, 4(3):188-198, May 2011. It has also been applied to a transrectal ultrasound training application with fluoroscopy imaging as published in Orcun Goksel, Kirill Sapchuk, William James Morris, and Septimiu E. Salcudean, “Prostate brachytherapy training with simulated ultrasound and fluoroscopy images”, IEEE Trans Biomedical Engineering, 60(4):n2002-12, April 2013.
- a physical anatomy model (mannequin) is further used with medical tools or instruments to simulate the medical procedure training as realistically as possible.
- tools such as an ultrasound probe 130 , a speculum 120 , and an embryo transfer catheter 112 may be used.
- a stiffer outer steath (catheter guide) is first used to guide the soft and flexible inner embryo transfer catheter 112 through the cervical canal to the proper location in the uterine cavity 100 .
- the transfer catheter 112 is loaded with the embryos 115 which are propulsed with the embryo transfer syringe 110 into the uterine cavity 100 .
- the whole procedure requires very careful gesture as any misposition or wound to the uterine may result in the failure of the IVF, which is a very costly and time-consuming procedure for the patients.
- the IVF embryo transfer procedure is conducted under ultrasound imaging supervision, by means of an abdominal ultrasound probe 130 .
- the user manipulates the abdominal ultrasound transducer 130 by pressing it on the belly skin and positioning and orienting it to optimize the ultrasound imaging capture.
- a jelly is used to facilitate the ultrasound propagation as well as the manipulation of the ultrasound transducer in contact with the patient skin.
- a moderately full bladder 140 is advisable for better ultrasound imaging quality, which will more or less deform with the probe compression on the above patient skin.
- the speculum tool 120 is usually made of metal and thus acts as a shield to the ultrasound waves, which also results in specific artefacts into the ultrasound images.
- the simulation of such a complex ultrasound procedure may raise specific issues to ensure a realistic virtual-physical dualism, e.g. the image should appear only when the ultrasound probe physically touches the anatomy model also emanating only from the curved surface of contact.
- Deformation of the ultrasound images and its components such as the skin (high compression), the bladder (moderate compression) and the uterine, the speculum and the catheter (no compression) needs to be synchronized with the user manipulation of the ultrasound probe, so the latter interaction remains as realistic as possible.
- Sonosim further introduced a 5-DOF tracking solution for this simulator by adding a translation sensor as a patch that can be applied over a live body or an anatomy model, in association with an electronic tag, so that the probe translation over a curved surface may be further tracked.
- a translation sensor as a patch that can be applied over a live body or an anatomy model, in association with an electronic tag, so that the probe translation over a curved surface may be further tracked.
- this solution is the lack of realism when manipulating the probe compared to a real ultrasound procedure, as no ultrasound gel can be used with this probe according to its instructions of use (http://sonosim.com/support/).
- friction of the rigid plastic virtual probe tip over the anatomy model surface significantly increases when the user has to push the probe to compress the skin and get a better ultrasound image. This friction is much lower in a real procedure where ultrasound gel or jelly is used to improve the ultrasound capture, but in general, it is not desirable to use any liquid or gel in a training setup due to the overhead required to clean it.
- a VR/AR ultrasound simulation system comprising a data processing unit, a display in communication with the data processing unit, an anatomy model and a tool, wherein the tool is adapted to interact with low friction with a surface of the anatomy model.
- the tool may be adapted to roll over a surface of the anatomy model.
- the tool may be a replicate of an ultrasound probe, such as an abdominal probe or a vaginal probe.
- the surface of the anatomy model may be deformable.
- the VR/AR ultrasound simulation system may comprise a position and orientation sensor configured and positioned to sense a position and/or orientation of the anatomy model, and/or a position and orientation sensor configured and positioned to sense a position and/or orientation of the tool.
- the tool tip or end may be adapted with a roller ball or with a low friction material.
- FIG. 1 illustrates an IVF embryo transfer procedure as known in the prior art.
- FIG. 2 illustrates an ultrasound simulator adapted to simulate an IVF embryo transfer procedure.
- FIG. 3 shows four cervix and uterine anatomy models corresponding to some anatomy shapes that may be encountered in IVF embryo transfer medical practice.
- FIG. 4 shows engineering views of an exemplary ultrasound simulation probe adapted to provide realistic passive haptic feedback without requiring the use of ultrasound gel over the anatomy model.
- FIG. 5 shows a first screen capture of an ultrasound simulation rendering for the IVF embryo transfer procedure training.
- FIG. 6 shows a second screen capture of an ultrasound simulation rendering for the IVF embryo transfer procedure training.
- FIG. 7 illustrates a flow chart of an ultrasound simulation method in accordance with different possible embodiments.
- FIG. 2 shows a partial view of an ultrasound simulator system comprising a data processing unit 200 , a display screen 210 , and an anatomy model 220 , according to an embodiment of the invention.
- a pelvic model 220 of the human female anatomy is shown, but other models can be used as well. Examples of models can be found in the catalogues of specialized anatomic model suppliers such as Limbs&Things, Bristol, UK. Some models may be representative of the human anatomy. Other models may be representative of an animal anatomy, e.g. for veterinary training purpose.
- the anatomy model 220 may be made of plastic or any other suitable material.
- the anatomy model 220 is made of a flexible material, such as flexible plastic, so that it can deform under pressure.
- the anatomy model 220 , or parts of the anatomy model may be interchanged with another similar anatomy model corresponding to different simulated patient characteristics or pathologies. For instance in an IVF embryo transfer training simulator, various combinations of cervix canal models (straight, tortuous, with a trap . . . ) and uterus types (axial, anteverted, retroverted . . . ) may be used to simulate more or less challenging variations of the patient pelvic organs anatomy.
- 3 shows four different cervix models: a) base cervix case with axial uterus, b) cervix entrance deflected down 40 degrees in an anteverted uterus, c) tortuous S-shaped cervix with axial uterus and d) trap cervix with 20 degrees anteverted uterus.
- the anatomy model 220 may be manufactured in the training room by using a manufacturing unit.
- manufacturing units are well known to those skilled in the art of rapid prototyping; they may be based on additive manufacturing, such as 3D printing, or substractive manufacturing, such as CNC milling.
- the anatomy model may also be manufactured according to a specific patient anatomy. Examples of manufacturing units as used in emerging medical applications are described for instance in Vietnamese, A. T.; Stan, O.; Miclea, L., “3D printing bone models extracted from medical imaging data”, 2014 IEEE International Conference on Automation, Quality and Testing, Robotics, vol., no., pp. 1,5, 22-24 May 2014.
- Patient specific anatomy models may also comprise specific patient pathologies.
- Patient specific anatomy models may be manufactured and interchanged with the anatomy model 220 .
- Patient specific anatomy models may enable the physicians and trainees to develop and improve their practice in a virtual reality environment before actually performing the medical procedure.
- patient specific anatomy models may be interchanged with the anatomy model 220 , such as for instance a bladder, a womb, an upper torso, a lower torso, or a joint model.
- the patient specific model may comprise parts of the anatomy model 220 .
- the patient specific model may be manufactured by the manufacturing unit.
- the patient specific model may be created with additive manufacturing, such as 3D printing, or substractive manufacturing, such as CNC milling.
- the data processing unit 200 may comprise one or more central processing units (“CPU”), memory modules, controlling modules, and/or communication modules, for example. Other embodiments may include data processing units 200 with other configurations and combinations of hardware and software elements.
- a distributed data processing unit may be used.
- a communication module of the data processing 200 may be connected to a manufacturing unit.
- the data processing unit and the manufacturing unit may be combined in a single unit.
- Some or all of the data processing unit 200 components may be used to compute and display onto a display screen 210 a VR/AR simulation model that may correspond to a chosen medical procedure training scenario. Multiple display screens may also be used.
- the display screen 210 may comprise a touch interface to provide an interface for a physician during a simulation exercise.
- the simulator cart may further comprise a camera.
- the anatomy model 220 may comprise at least one position and orientation sensor (not represented). As described in U.S. Pat. No. 8,992,230, said position and orientation sensor may be associated with at least one calibration unit. For example, six degree of freedom (“6DOF”) miniaturized magnetic tracking sensors may be used. Other embodiments are also possible, e.g. optical tracking or motion tracking may be used.
- the anatomy model 220 may be connected to the data processing unit 200 through a connection such as a USB link, other standard wired or wireless connection, or other communication link.
- the data processing unit 200 may receive the sensor information from the anatomy model 220 and may calculate the virtual anatomy model 220 position and orientation in accordance with the sensor measurement.
- the data processing unit 200 may use the calculated model 220 position and orientation to generate a visual model and display the visual model onto the display screen 210 .
- the data processing unit 200 may compute the VR/AR model position and orientation by combining the absolute sensor measurement received from at least one anatomy model sensor with the pre-computed calibration data from the calibration unit associated with said sensor for the simulated VR/AR model.
- FIG. 2 further shows a replica of an ultrasound probe 230 which may be used to simulate an ultrasound imaging capture.
- the ultrasound probe replica 230 of FIG. 2 replicates the shape of a convex abdominal ultrasound transducer.
- Other probes may be replicated in other ultrasound training procedures, for instance a transvaginal or a transrectal probe.
- the replica may not need to be an exact replica of an ultrasound probe, but it may give a realistic haptic feedback similar to a real probe.
- the ultrasound probe replica 230 may be handled and put in contact with the anatomy model skin surface by the trainee. The trainee may also push the anatomy model 220 skin surface with the ultrasound probe replica 230 as in a real imaging procedure, and the anatomy model 220 skin surface may deform accordingly.
- the ultrasound probe replica 230 may be adapted to comprise at least one position and orientation sensor, possibly in association with a calibration unit, for instance as described in U.S. Pat. No. 8,992,230.
- the data processing unit 200 may receive the sensor information from the ultrasound probe replica 230 and may calculate the VR/AR model position and orientation in accordance with the model sensor measurement and the probe replica sensor measurement respectively.
- FIG. 2 further shows a medical tool 240 appended to the anatomy model 220 .
- the medical tool 240 of FIG. 2 is a speculum that can be inserted into the pelvic model 220 through the vulva portal of anatomy model.
- Other tools may be used, for instance an embryo transfer catheter, an embryo transfer syringe, a tenaculum or, a Hegra dilator.
- the medical tool 240 may be a standard medical tool suitable for various medical procedures, for instance a tenaculum tool that may be used in gynecology or in surgery.
- the tool may be adapted to comprise at least one position and orientation sensor and one calibration unit associated with the position and orientation sensor, for instance as described in U.S. Pat. No. 8,992,230.
- the position and orientation of the tool may be derived from the position and orientation of another element in the simulator system, such as an internal element to which the tool is mechanically bound without any remaining degree of freedom; for instance the tenaculum for steadying the cervix and uterus during the insertion of an intrauterine device is typically bound to the cervix position and orientation once steadying it.
- a part of the anatomy model for instance the cervix, rather than the tool, for instance the tenaculum, may be adapted to comprise at least one position and orientation sensor and one calibration unit associated with the position and orientation sensor, for instance as described for the femur or tibia parts in U.S. Pat. No. 8,992,230.
- Other embodiments are also possible to adapt the VR/AR ultrasound simulator to various medical practices, anatomy models, and tools.
- the data processing unit 200 may receive the sensor information from the anatomy model 220 and may calculate the VR/AR model position and orientation in accordance with the anatomy model sensor measurement, the probe replica sensor measurement, and the tool sensor measurement respectively.
- the tip, or end, attached to the tool body of the ultrasound probe replica 230 is further mechanically adapted to comprise a rolling part 235 , such as a roller ball, which may simulate a more realistic low friction motion, when the ultrasound probe replica 230 tip interacts with the anatomy model 220 surface.
- a rolling part 235 such as a roller ball
- FIG. 4 shows the engineering views of a possible mechanical embodiment of a roller ball 235 which decreases the friction for different translational and rotational movements of the ultrasound probe replica 230 over the anatomy model 220 surface.
- Other embodiments than a roller ball are also possible, for instance a low friction material may be used at the tip 235 of the ultrasound probe replica 230 , or on the anatomy model 220 surface.
- the data processing unit 200 may use the calculated VR/AR model position and orientation to generate a visual model and display the visual model onto the display screen 210 .
- initial alignment of the VR/AR model with the position and orientation of the anatomy model 220 , the ultrasound probe replica 230 or the medical tool 240 may also require calibration.
- the methods and systems as described in U.S. Pat. No. 8,992,230 for an endoscopy simulation system may be used to calibrate the VR/AR ultrasound simulation system.
- the ultrasound image also needs to be rendered as realistically as possible.
- the data processing unit 200 may use the calculated VR/AR model position and orientation to identify a slice to be rendered from a reconstructed 3D ultrasound volume or a 3D ultrasound generation model and to render the simulated ultrasound image in real time onto the display screen 210 .
- FIG. 5 shows a screen capture of the ultrasound image and the VR/AR model as rendered on display screen 210 by the data processing unit 200 in the embryo transfer procedure.
- the embryo transfer catheter tool 540 is rendered into the uterine cavity 500 on the ultrasound image 530 on the left as well as on the VR/AR model computer graphics representation on the right.
- a moderately filled bladder 520 and abdominal skin layers 510 are also visible in the ultrasound image as in a real ultrasound capture.
- FIG. 6 shows a screen capture of the ultrasound image and the VR/AR model as rendered on display screen 210 by the data processing unit 200 at the next step in the embryo transfer procedure, where the trainee transfers the embryos 600 from the embryo transfer catheter into the uterine cavity.
- the data processing unit 200 may be adapted to implement the interpolative ultrasound simulation methods as proposed by Goksel et al. to render realistic ultrasound images by interpolating a 2D slice in a 3D ultrasound volume reconstructed from formerly registered ultrasound image records, such as MR or CT data records, in combination with real-time induced deformations due to the user interaction with the ultrasound simulator system.
- Other methods may be used, for instance the ultrasound simulation may use a ray-based generated ultrasound simulation with segmentation rather that interpolative ultrasound simulation methods, as known to those skilled in the art of ultrasound simulation.
- a diversity of different patient VR/AR models may also be used as the basis for a diversity of ultrasound practice training scenarios, for instance to represent different pathologies.
- the VR/AR ultrasound simulation system and methods may be further adapted with space warping methods and systems as described in US patent application US20140071165.
- FIG. 7 shows a general flowchart of the proposed VR/AR ultrasound simulation method, comprising the steps of:
- While a passive haptic VR/AR ultrasound simulator has been detailed herein as an example for a better illustration of the disclosure, the proposed methods and systems can be generalized to any type of hardware simulator setups.
- various position and orientation tracking means may be used, such as magnetic or optical tracking systems, or a mix of such systems.
- Various calibration procedures may also be used.
- Active haptic hardware may also be integrated into the VR/AR ultrasound simulator.
Landscapes
- Engineering & Computer Science (AREA)
- Health & Medical Sciences (AREA)
- Physics & Mathematics (AREA)
- Life Sciences & Earth Sciences (AREA)
- General Physics & Mathematics (AREA)
- Medical Informatics (AREA)
- General Health & Medical Sciences (AREA)
- Theoretical Computer Science (AREA)
- Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
- Radiology & Medical Imaging (AREA)
- Educational Technology (AREA)
- Business, Economics & Management (AREA)
- Educational Administration (AREA)
- Pure & Applied Mathematics (AREA)
- Mathematical Physics (AREA)
- Mathematical Optimization (AREA)
- Mathematical Analysis (AREA)
- Medicinal Chemistry (AREA)
- Computational Mathematics (AREA)
- Algebra (AREA)
- Chemical & Material Sciences (AREA)
- Heart & Thoracic Surgery (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Pathology (AREA)
- Veterinary Medicine (AREA)
- Public Health (AREA)
- Animal Behavior & Ethology (AREA)
- Surgery (AREA)
- Molecular Biology (AREA)
- Reproductive Health (AREA)
- Gynecology & Obstetrics (AREA)
- Pregnancy & Childbirth (AREA)
- Computer Graphics (AREA)
- Computer Hardware Design (AREA)
- General Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Ultra Sonic Daignosis Equipment (AREA)
Abstract
Embodiments of flexible ultrasound simulation devices, methods, and systems described herein may facilitate various medical procedure training by rendering realistic ultrasound images in accordance with the end user interaction with an anatomy model, an ultrasound probe replica and a medical tool. An ultrasound probe replica may be adapted with a low friction tip, such as a roller ball. A data processing unit may compute and display an accurate Virtual Reality/Augmented Reality (VR/AR) model and an ultrasound image in accordance with the position and orientation tracking of the VR/AR ultrasound simulator elements. The VR/AR ultrasound simulator may be adapted to realistically render the VR/AR model and the ultrasound image as a function of the user interaction with the VR/AR simulator elements.
Description
- This application claims priority to 62/242,322 filed on Oct. 16, 2015, which is herein incorporated by reference in its entirety.
- The present invention relates to computerized medical simulation in general, and more specifically to virtual reality and/or augmented reality simulation devices and systems for medical training purposes.
- Medical imaging has become more and more used for both diagnostic/examination and therapeutic purposes in a number of medical applications, such as endoscopy for surgery, or ultrasound imaging for various gynecology and/or obstetrics applications, for instance in the embryo transfer procedure for In Vitro Fertilization (IVF). These new techniques may require dedicated training for physicians and surgeons to master the indirect hand-eye coordination required by the imaging system as well as the manipulation of the imaging tools in addition to the conventional medical instruments and procedures for a diversity of patient anatomies as may be encountered in medical practice. Computerized medical procedure training simulators may enable the physicians and trainees to develop and improve their practice in a virtual reality environment before actually practicing in the operation room.
- Advanced medical procedure simulators may be based on a virtual reality (“VR”) and/or a mixed or augmented reality (“AR”) simulation apparatus by which the physician can experiment a medical procedure scenario. The VR/AR system may compute and display a visual VR/AR model of anatomical structures in accordance with physician gestures and actions to provide various feedback, such as visual feedback. In a VR system, an entire image may be simulated for display to a user, and in an AR system, a simulated image may be overlaid or otherwise incorporated with an actual image for display to a user. Various patient models with different pathologies can be selected. Therefore, natural variations as encountered over the years by practicing doctors can be simulated for a user over a compressed period of time for training purposes. The medical simulation procedure can be recorded and rehearsed for evaluation purpose. The VR/AR simulation system can also compute and provide various metrics and statistics.
- VR/AR simulation systems such as the one described in U.S. Pat. No. 8,992,230 include a human anatomy model of a joint of organ in real size. The VR/AR simulation system may further comprise a medical instrument to more realistically simulate the medical procedure. The model is further adapted with sensors for tracking the position and/or orientation of both the anatomy model and the medical instrument. As described in U.S. Pat. No. 8,992,230, calibration units may be further used to automatically setup and align the VR/AR simulation system to a diversity of anatomy models and medical procedure training scenarios without requiring a cumbersome, manual calibration procedure each time a new model is adapted to the system.
- A passive feedback VR/AR simulation system such as for instance the one described in U.S. Pat. No. 8,992,230 may also be used with a diversity of medical procedure training scenarios, some of which may possibly result in a mismatch between an anatomy model surface as touched by the trainee and a virtual environment surface as computed by the VR/AR simulation system and rendered on the screen. In order to further improve the passive haptic experience and increase the realism in such medical training scenarios, the VR/AR simulation system may be further adapted with space warping methods and systems as described in US patent application US20140071165.
- Most prior art ultrasound simulation solutions have been developed based on interpolative ultrasound simulation, as used for instance by a number of commercial ultrasound training simulators such as Medsim (http://www.medsim.com/), EchoCom (http://www.echocom.de) and MedCom/Sonofit (http://www.sonofit.de). These prior art simulators use a set of collected 2D images for each case, which are reconstructed into 3D volumes per case in an offline “case generation” stage with minor differences between individual methods in how the 3D volumes and their corresponding graphical models are generated. For instance, Dror Aiger and Daniel Cohen-Or described the use of deformable registration techniques for generating large 3D volumes from smaller swept scans in “Real-time ultrasound imaging simulation”, Real-Time Imaging, 4(4):263-274, 1998. In “Augmented reality simulator for training in two-dimensional echocardiography”, Computers and Biomedical Research, 33:11-22, 2000, M. Weidenbach, C. Wick, S. Pieper, K. J. Quast, T. Fox, G. Grunst, and D. A. Redel proposed to register the recorded patient-specific volumes to a generic 3D anatomical heart model, which they thus call augmented training. All the above methods present the user a non-deformable model of the anatomy. In other words, although the images may change with the transducer position/orientation in some simulators, they do not change according to the trainee's interaction with the mannequin or the virtual model, which negatively impacts the simulation realism.
- Compared to non deformable solutions, deformable, interactive ultrasound simulation may generate a better sense of simulator realism and consequent trainee immersion. In “B-mode ultrasound image simulation in deformable 3-D medium”, IEEE Trans Medical Imaging, 28(11):1657-69, November 2009, O. Goksel and S. E. Salcudean introduced the first interpolative simulator that allows for deformation by using a fast mapping technique for image pixels to be simulated, from the deformed simulation tissue coordinates to a nominal recorded volumetric coordinate frame. This method combines an input (reconstructed) 3D volume and interactive, volumetric tissue deformation models such as the finite element method and has been further applied to prostate brachytherapy simulation as published in Orcun Goksel, Kirill Sapchuk, and Septimiu E. Salcudean, “Haptic simulator for prostate brachytherapy with simulated needle and probe interaction”, IEEE Trans Haptics, 4(3):188-198, May 2011. It has also been applied to a transrectal ultrasound training application with fluoroscopy imaging as published in Orcun Goksel, Kirill Sapchuk, William James Morris, and Septimiu E. Salcudean, “Prostate brachytherapy training with simulated ultrasound and fluoroscopy images”, IEEE Trans Biomedical Engineering, 60(4):n2002-12, April 2013.
- However, the latter methods by Goksel et al. only used a virtual patient model for simplicity. In a passive haptic VR/AR medical simulator, a physical anatomy model (mannequin) is further used with medical tools or instruments to simulate the medical procedure training as realistically as possible. For example in a state of the art embryo transfer IVF medical procedure as represented by
FIG. 1 , tools such as anultrasound probe 130, aspeculum 120, and anembryo transfer catheter 112 may be used. As described for instance by http://www.advancedfertility.com/ivf-embryo-transfer-catheter.htm, a stiffer outer steath (catheter guide) is first used to guide the soft and flexible innerembryo transfer catheter 112 through the cervical canal to the proper location in theuterine cavity 100. Once at the right location, thetransfer catheter 112 is loaded with theembryos 115 which are propulsed with theembryo transfer syringe 110 into theuterine cavity 100. The whole procedure requires very careful gesture as any misposition or wound to the uterine may result in the failure of the IVF, which is a very costly and time-consuming procedure for the patients. The IVF embryo transfer procedure is conducted under ultrasound imaging supervision, by means of anabdominal ultrasound probe 130. The user manipulates theabdominal ultrasound transducer 130 by pressing it on the belly skin and positioning and orienting it to optimize the ultrasound imaging capture. A jelly is used to facilitate the ultrasound propagation as well as the manipulation of the ultrasound transducer in contact with the patient skin. In general, a moderatelyfull bladder 140 is advisable for better ultrasound imaging quality, which will more or less deform with the probe compression on the above patient skin. Thespeculum tool 120 is usually made of metal and thus acts as a shield to the ultrasound waves, which also results in specific artefacts into the ultrasound images. - As known to those skilled in the art, the simulation of such a complex ultrasound procedure may raise specific issues to ensure a realistic virtual-physical dualism, e.g. the image should appear only when the ultrasound probe physically touches the anatomy model also emanating only from the curved surface of contact. Deformation of the ultrasound images and its components such as the skin (high compression), the bladder (moderate compression) and the uterine, the speculum and the catheter (no compression) needs to be synchronized with the user manipulation of the ultrasound probe, so the latter interaction remains as realistic as possible.
- In “Patient-Specific Interactive Ultrasound Image Simulation with Soft-Tissue Deformation”, PhD thesis, University of California Los Angeles, 2013, K. Petrinec proposed to adapt a commercially available ultrasound simulator (www.sonosim.com) to further simulate the deformation of soft tissues when in contact with the ultrasound probe, based on the Goksel method. In this simulator, only 3-DOF motion trackers are used to track the ultrasound probe orientation—that is, the probe is assumed to be positioned at a given 3D static position over a mannequin, and the system does not track its translational movement.
- More recently, as described in US20150154890, Sonosim further introduced a 5-DOF tracking solution for this simulator by adding a translation sensor as a patch that can be applied over a live body or an anatomy model, in association with an electronic tag, so that the probe translation over a curved surface may be further tracked. From a passive haptic perspective, one major drawback of this solution is the lack of realism when manipulating the probe compared to a real ultrasound procedure, as no ultrasound gel can be used with this probe according to its instructions of use (http://sonosim.com/support/). In particular, friction of the rigid plastic virtual probe tip over the anatomy model surface significantly increases when the user has to push the probe to compress the skin and get a better ultrasound image. This friction is much lower in a real procedure where ultrasound gel or jelly is used to improve the ultrasound capture, but in general, it is not desirable to use any liquid or gel in a training setup due to the overhead required to clean it.
- There is therefore a need for better ultrasound simulation devices, methods and systems that facilitate a diversity of ultrasound training procedures without requiring an expensive hardware setup and cumbersome registration and calibration procedures.
- It is an object of the invention to provide a VR/AR ultrasound simulation system comprising a data processing unit, a display in communication with the data processing unit, an anatomy model and a tool, wherein the tool is adapted to interact with low friction with a surface of the anatomy model. The tool may be adapted to roll over a surface of the anatomy model. The tool may be a replicate of an ultrasound probe, such as an abdominal probe or a vaginal probe. The surface of the anatomy model may be deformable. The VR/AR ultrasound simulation system may comprise a position and orientation sensor configured and positioned to sense a position and/or orientation of the anatomy model, and/or a position and orientation sensor configured and positioned to sense a position and/or orientation of the tool.
- It is a further object of the invention to provide a medical simulation tool, wherein the tool is adapted to interact with low friction with a surface of an anatomy model. The tool tip or end, may be adapted with a roller ball or with a low friction material.
-
FIG. 1 illustrates an IVF embryo transfer procedure as known in the prior art. -
FIG. 2 illustrates an ultrasound simulator adapted to simulate an IVF embryo transfer procedure. -
FIG. 3 shows four cervix and uterine anatomy models corresponding to some anatomy shapes that may be encountered in IVF embryo transfer medical practice. -
FIG. 4 shows engineering views of an exemplary ultrasound simulation probe adapted to provide realistic passive haptic feedback without requiring the use of ultrasound gel over the anatomy model. -
FIG. 5 shows a first screen capture of an ultrasound simulation rendering for the IVF embryo transfer procedure training. -
FIG. 6 shows a second screen capture of an ultrasound simulation rendering for the IVF embryo transfer procedure training. -
FIG. 7 illustrates a flow chart of an ultrasound simulation method in accordance with different possible embodiments. - Embodiments of flexible ultrasound simulation devices, methods, and systems will now be described in more detail with reference to an exemplary medical simulation of an IVF embryo transfer procedure, as illustrated on
FIG. 1 .FIG. 2 shows a partial view of an ultrasound simulator system comprising adata processing unit 200, adisplay screen 210, and ananatomy model 220, according to an embodiment of the invention. For the purpose of illustration, inFIG. 2 , apelvic model 220 of the human female anatomy is shown, but other models can be used as well. Examples of models can be found in the catalogues of specialized anatomic model suppliers such as Limbs&Things, Bristol, UK. Some models may be representative of the human anatomy. Other models may be representative of an animal anatomy, e.g. for veterinary training purpose. - The
anatomy model 220 may be made of plastic or any other suitable material. Preferably, theanatomy model 220 is made of a flexible material, such as flexible plastic, so that it can deform under pressure. In some embodiments, theanatomy model 220, or parts of the anatomy model, may be interchanged with another similar anatomy model corresponding to different simulated patient characteristics or pathologies. For instance in an IVF embryo transfer training simulator, various combinations of cervix canal models (straight, tortuous, with a trap . . . ) and uterus types (axial, anteverted, retroverted . . . ) may be used to simulate more or less challenging variations of the patient pelvic organs anatomy.FIG. 3 shows four different cervix models: a) base cervix case with axial uterus, b) cervix entrance deflected down 40 degrees in an anteverted uterus, c) tortuous S-shaped cervix with axial uterus and d) trap cervix with 20 degrees anteverted uterus. - In some embodiments, the
anatomy model 220, or parts of the anatomy model, such as for instance the cervix canal and/or the uterine in an OB/GYN training application, may be manufactured in the training room by using a manufacturing unit. Examples of such manufacturing units are well known to those skilled in the art of rapid prototyping; they may be based on additive manufacturing, such as 3D printing, or substractive manufacturing, such as CNC milling. - In some embodiments, the anatomy model may also be manufactured according to a specific patient anatomy. Examples of manufacturing units as used in emerging medical applications are described for instance in Popescu, A. T.; Stan, O.; Miclea, L., “3D printing bone models extracted from medical imaging data”, 2014 IEEE International Conference on Automation, Quality and Testing, Robotics, vol., no., pp. 1,5, 22-24 May 2014. Patient specific anatomy models may also comprise specific patient pathologies. Patient specific anatomy models may be manufactured and interchanged with the
anatomy model 220. Patient specific anatomy models may enable the physicians and trainees to develop and improve their practice in a virtual reality environment before actually performing the medical procedure. Furthermore, in order to support other medical procedures with ultrasound imaging, patient specific anatomy models may be interchanged with theanatomy model 220, such as for instance a bladder, a womb, an upper torso, a lower torso, or a joint model. In some embodiments the patient specific model may comprise parts of theanatomy model 220. The patient specific model may be manufactured by the manufacturing unit. In some embodiments the patient specific model may be created with additive manufacturing, such as 3D printing, or substractive manufacturing, such as CNC milling. - The
data processing unit 200 may comprise one or more central processing units (“CPU”), memory modules, controlling modules, and/or communication modules, for example. Other embodiments may includedata processing units 200 with other configurations and combinations of hardware and software elements. A distributed data processing unit may be used. In some embodiments, a communication module of thedata processing 200 may be connected to a manufacturing unit. In some embodiments, the data processing unit and the manufacturing unit may be combined in a single unit. Some or all of thedata processing unit 200 components may be used to compute and display onto a display screen 210 a VR/AR simulation model that may correspond to a chosen medical procedure training scenario. Multiple display screens may also be used. Thedisplay screen 210 may comprise a touch interface to provide an interface for a physician during a simulation exercise. In other embodiments (not illustrated) the simulator cart may further comprise a camera. - In
FIG. 2 , theanatomy model 220 may comprise at least one position and orientation sensor (not represented). As described in U.S. Pat. No. 8,992,230, said position and orientation sensor may be associated with at least one calibration unit. For example, six degree of freedom (“6DOF”) miniaturized magnetic tracking sensors may be used. Other embodiments are also possible, e.g. optical tracking or motion tracking may be used. Theanatomy model 220 may be connected to thedata processing unit 200 through a connection such as a USB link, other standard wired or wireless connection, or other communication link. Thedata processing unit 200 may receive the sensor information from theanatomy model 220 and may calculate thevirtual anatomy model 220 position and orientation in accordance with the sensor measurement. Thedata processing unit 200 may use thecalculated model 220 position and orientation to generate a visual model and display the visual model onto thedisplay screen 210. As described in U.S. Pat. No. 8,992,230, thedata processing unit 200 may compute the VR/AR model position and orientation by combining the absolute sensor measurement received from at least one anatomy model sensor with the pre-computed calibration data from the calibration unit associated with said sensor for the simulated VR/AR model. -
FIG. 2 further shows a replica of anultrasound probe 230 which may be used to simulate an ultrasound imaging capture. For the purpose of illustration, in the case of IVF embryo transfer simulation, theultrasound probe replica 230 ofFIG. 2 replicates the shape of a convex abdominal ultrasound transducer. Other probes may be replicated in other ultrasound training procedures, for instance a transvaginal or a transrectal probe. The replica may not need to be an exact replica of an ultrasound probe, but it may give a realistic haptic feedback similar to a real probe. Theultrasound probe replica 230 may be handled and put in contact with the anatomy model skin surface by the trainee. The trainee may also push theanatomy model 220 skin surface with theultrasound probe replica 230 as in a real imaging procedure, and theanatomy model 220 skin surface may deform accordingly. Theultrasound probe replica 230 may be adapted to comprise at least one position and orientation sensor, possibly in association with a calibration unit, for instance as described in U.S. Pat. No. 8,992,230. Thedata processing unit 200 may receive the sensor information from theultrasound probe replica 230 and may calculate the VR/AR model position and orientation in accordance with the model sensor measurement and the probe replica sensor measurement respectively. -
FIG. 2 further shows amedical tool 240 appended to theanatomy model 220. For the purpose of illustration, in the case of IVF embryo transfer simulation, themedical tool 240 ofFIG. 2 is a speculum that can be inserted into thepelvic model 220 through the vulva portal of anatomy model. Other tools may be used, for instance an embryo transfer catheter, an embryo transfer syringe, a tenaculum or, a Hegra dilator. Themedical tool 240 may be a standard medical tool suitable for various medical procedures, for instance a tenaculum tool that may be used in gynecology or in surgery. In some embodiments, the tool may be adapted to comprise at least one position and orientation sensor and one calibration unit associated with the position and orientation sensor, for instance as described in U.S. Pat. No. 8,992,230. In other embodiments, the position and orientation of the tool may be derived from the position and orientation of another element in the simulator system, such as an internal element to which the tool is mechanically bound without any remaining degree of freedom; for instance the tenaculum for steadying the cervix and uterus during the insertion of an intrauterine device is typically bound to the cervix position and orientation once steadying it. Thus, in some embodiments of the simulator system, a part of the anatomy model, for instance the cervix, rather than the tool, for instance the tenaculum, may be adapted to comprise at least one position and orientation sensor and one calibration unit associated with the position and orientation sensor, for instance as described for the femur or tibia parts in U.S. Pat. No. 8,992,230. Other embodiments are also possible to adapt the VR/AR ultrasound simulator to various medical practices, anatomy models, and tools. - The
data processing unit 200 may receive the sensor information from theanatomy model 220 and may calculate the VR/AR model position and orientation in accordance with the anatomy model sensor measurement, the probe replica sensor measurement, and the tool sensor measurement respectively. - In a preferred embodiment, the tip, or end, attached to the tool body of the
ultrasound probe replica 230 is further mechanically adapted to comprise arolling part 235, such as a roller ball, which may simulate a more realistic low friction motion, when theultrasound probe replica 230 tip interacts with theanatomy model 220 surface.FIG. 4 shows the engineering views of a possible mechanical embodiment of aroller ball 235 which decreases the friction for different translational and rotational movements of theultrasound probe replica 230 over theanatomy model 220 surface. Other embodiments than a roller ball are also possible, for instance a low friction material may be used at thetip 235 of theultrasound probe replica 230, or on theanatomy model 220 surface. - The
data processing unit 200 may use the calculated VR/AR model position and orientation to generate a visual model and display the visual model onto thedisplay screen 210. As known to those skilled in the art, initial alignment of the VR/AR model with the position and orientation of theanatomy model 220, theultrasound probe replica 230 or themedical tool 240 may also require calibration. The methods and systems as described in U.S. Pat. No. 8,992,230 for an endoscopy simulation system may be used to calibrate the VR/AR ultrasound simulation system. - In an ultrasound simulation system, the ultrasound image also needs to be rendered as realistically as possible. The
data processing unit 200 may use the calculated VR/AR model position and orientation to identify a slice to be rendered from a reconstructed 3D ultrasound volume or a 3D ultrasound generation model and to render the simulated ultrasound image in real time onto thedisplay screen 210. -
FIG. 5 shows a screen capture of the ultrasound image and the VR/AR model as rendered ondisplay screen 210 by thedata processing unit 200 in the embryo transfer procedure. The embryotransfer catheter tool 540 is rendered into theuterine cavity 500 on theultrasound image 530 on the left as well as on the VR/AR model computer graphics representation on the right. A moderately filledbladder 520 and abdominal skin layers 510 are also visible in the ultrasound image as in a real ultrasound capture.FIG. 6 shows a screen capture of the ultrasound image and the VR/AR model as rendered ondisplay screen 210 by thedata processing unit 200 at the next step in the embryo transfer procedure, where the trainee transfers theembryos 600 from the embryo transfer catheter into the uterine cavity. - In a possible embodiment, the
data processing unit 200 may be adapted to implement the interpolative ultrasound simulation methods as proposed by Goksel et al. to render realistic ultrasound images by interpolating a 2D slice in a 3D ultrasound volume reconstructed from formerly registered ultrasound image records, such as MR or CT data records, in combination with real-time induced deformations due to the user interaction with the ultrasound simulator system. Other methods may be used, for instance the ultrasound simulation may use a ray-based generated ultrasound simulation with segmentation rather that interpolative ultrasound simulation methods, as known to those skilled in the art of ultrasound simulation. A diversity of different patient VR/AR models may also be used as the basis for a diversity of ultrasound practice training scenarios, for instance to represent different pathologies. In order to further improve the passive haptic experience and increase the realism in such diversity of medical training scenarios without requiring too many different setups of the underlying anatomy model, the VR/AR ultrasound simulation system and methods may be further adapted with space warping methods and systems as described in US patent application US20140071165. -
FIG. 7 shows a general flowchart of the proposed VR/AR ultrasound simulation method, comprising the steps of: -
- Acquiring the position and orientation of the VR/AR simulator elements, such as the anatomy model parts, the ultrasound probe replica and any medical tools relevant to the ultrasound procedure training;
- Aligning the VR/AR model to the tracked position and orientation of the VR/AR simulator elements;
- Generating an ultrasound image matching the VR/AR model;
- Rendering the ultrasound image and a computer graphics representation of the VR/AR model onto a display screen.
- While a VR/AR ultrasound simulator for the IVF embryo transfer procedure has been detailed herein as an example for a better illustration of the disclosure, the proposed methods and systems can be generalized to any type of ultrasound simulations.
- While a passive haptic VR/AR ultrasound simulator has been detailed herein as an example for a better illustration of the disclosure, the proposed methods and systems can be generalized to any type of hardware simulator setups. In particular, various position and orientation tracking means may be used, such as magnetic or optical tracking systems, or a mix of such systems. Various calibration procedures may also be used. Active haptic hardware may also be integrated into the VR/AR ultrasound simulator.
- While a VR/AR ultrasound simulation method using interpolation from a reconstructed 3D ultrasound volume has been detailed herein as an example for a better illustration of the disclosure, the proposed methods and systems can be generalized to any type of ultrasound imaging simulations. In particular, generative ray-based simulation systems and methods may also be used for ultrasound image modeling.
- While various embodiments of an IVF embryo transfer simulator have been described above, it should be understood that they have been presented by way of example and not limitation. It will be apparent to persons skilled in the relevant art(s) that the proposed methods and systems can be generalized to any type of ultrasound simulations. Various changes in form and detail can be made therein without departing from the spirit and scope. In fact, after reading the above description, it will be apparent to one skilled in the relevant art(s) how to implement alternative embodiments. Thus, the present embodiments should not be limited by any of the above-described embodiments.
- In addition, it should be understood that any figures which highlight the functionality and advantages are presented for example purposes only. The disclosed methodology and system are each sufficiently flexible and configurable such that they may be utilized in ways other than that shown.
- Although the term “at least one” may often be used in the specification, claims and drawings, the terms “a”, “an”, “the”, “said”, etc. also signify “at least one” or “the at least one” in the specification, claims and drawings.
- Finally, it is the applicant's intent that only claims that include the express language “means for” or “step for” be interpreted under 35 U.S.C. 112, paragraph 6. Claims that do not expressly include the phrase “means for” or “step for” are not to be interpreted under 35 U.S.C. 112, paragraph 6.
Claims (12)
1. A simulation system comprising:
a data processing unit;
a display in communication with the data processing unit;
an anatomy model; and
a tool adapted to interact with low friction with a surface of the anatomy model.
2. The simulation system of claim 1 , wherein the tool is adapted to roll over a surface of the anatomy model.
4. The simulation system of claim 1 , wherein the tool is a replica of an ultrasound probe.
5. The simulation system of claim 4 , wherein the ultrasound probe is an abdominal probe.
6. The simulation system of claim 4 , wherein the ultrasound probe is a vaginal probe.
7. The simulation system of claim 1 , wherein the surface of the anatomy model is deformable.
8. The simulation system of claim 1 , further comprising a position and orientation sensor configured and positioned to sense a position and/or orientation of the anatomy model.
9. The simulation system of claim 1 , further comprising a position and orientation sensor configured and positioned to sense a position and/or orientation of the tool.
10. A tool for a medical simulation system comprising:
a body; and
a tip connected to the body and adapted to interact with low friction with a surface of an anatomy model.
11. The tool of claim 10 , wherein the tool tip is adapted with a roller ball.
12. The tool of claim 10 , wherein the tool tip is adapted with a low friction material.
13. The tool of claim 12 , wherein the surface of the anatomy model is deformable.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US15/294,275 US20170110032A1 (en) | 2015-10-16 | 2016-10-14 | Ultrasound simulation system and tool |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US201562242322P | 2015-10-16 | 2015-10-16 | |
US15/294,275 US20170110032A1 (en) | 2015-10-16 | 2016-10-14 | Ultrasound simulation system and tool |
Publications (1)
Publication Number | Publication Date |
---|---|
US20170110032A1 true US20170110032A1 (en) | 2017-04-20 |
Family
ID=57178401
Family Applications (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US15/768,204 Active 2036-10-24 US10453360B2 (en) | 2015-10-16 | 2016-10-14 | Ultrasound simulation methods |
US15/294,275 Abandoned US20170110032A1 (en) | 2015-10-16 | 2016-10-14 | Ultrasound simulation system and tool |
Family Applications Before (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US15/768,204 Active 2036-10-24 US10453360B2 (en) | 2015-10-16 | 2016-10-14 | Ultrasound simulation methods |
Country Status (3)
Country | Link |
---|---|
US (2) | US10453360B2 (en) |
CN (1) | CN108352132A (en) |
WO (1) | WO2017064249A1 (en) |
Cited By (14)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN108806446A (en) * | 2017-11-24 | 2018-11-13 | 安徽农业大学 | A kind of production method of diagnosis of early gestation teaching mode and application |
WO2019035889A1 (en) * | 2017-08-16 | 2019-02-21 | Gaumard Scientific Company, Inc. | Augmented reality system for teaching patient care |
CN109545054A (en) * | 2017-09-21 | 2019-03-29 | 牛志宏 | Human embryo transplants in-vitro simulated research platform |
US10453360B2 (en) * | 2015-10-16 | 2019-10-22 | Virtamed Ag | Ultrasound simulation methods |
US20210134186A1 (en) * | 2017-01-26 | 2021-05-06 | SonoSim, Inc. | System and Method for Multisensory Psychomotor Skill Training |
CN113450634A (en) * | 2021-05-21 | 2021-09-28 | 重庆市人民医院 | Prostate biopsy training model under supersound |
US11189195B2 (en) * | 2017-10-20 | 2021-11-30 | American Association of Gynecological Laparoscopists, Inc. | Hysteroscopy training and evaluation |
US11315439B2 (en) | 2013-11-21 | 2022-04-26 | SonoSim, Inc. | System and method for extended spectrum ultrasound training using animate and inanimate training objects |
US11495142B2 (en) * | 2019-01-30 | 2022-11-08 | The Regents Of The University Of California | Ultrasound trainer with internal optical tracking |
US11568762B2 (en) | 2017-10-20 | 2023-01-31 | American Association of Gynecological Laparoscopists, Inc. | Laparoscopic training system |
US11600201B1 (en) | 2015-06-30 | 2023-03-07 | The Regents Of The University Of California | System and method for converting handheld diagnostic ultrasound systems into ultrasound training systems |
US11627944B2 (en) | 2004-11-30 | 2023-04-18 | The Regents Of The University Of California | Ultrasound case builder system and method |
US11631342B1 (en) | 2012-05-25 | 2023-04-18 | The Regents Of University Of California | Embedded motion sensing technology for integration within commercial ultrasound probes |
US11810473B2 (en) | 2019-01-29 | 2023-11-07 | The Regents Of The University Of California | Optical surface tracking for medical simulation |
Families Citing this family (11)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
EP3510564B1 (en) | 2016-09-06 | 2022-11-02 | Virtamed AG | Ray-tracing methods for realistic interactive ultrasound simulation |
IT201700059643A1 (en) * | 2017-05-31 | 2018-12-01 | Accurate S R L | Embryo transfer simulator. |
IT201700059635A1 (en) * | 2017-05-31 | 2018-12-01 | Accurate S R L | Embryo transfer simulator. |
IT201700059639A1 (en) * | 2017-05-31 | 2018-12-01 | Accurate S R L | Embryo transfer simulator. |
IT201800006822A1 (en) * | 2018-06-29 | 2019-12-29 | Sonohysterography simulator. | |
CN109523867B (en) * | 2018-11-21 | 2021-03-09 | 杨天烈 | Bracelet-type cow rectum positioning examination teaching device and teaching method |
CN111419272B (en) * | 2019-01-09 | 2023-06-27 | 深圳华大智造云影医疗科技有限公司 | Operation panel, doctor end controlling means and master-slave ultrasonic detection system |
CN110148343B (en) * | 2019-05-31 | 2021-08-03 | 吉林大学 | Abdominal ultrasonic exercise model and lesion model thereof |
CN112991854A (en) * | 2021-02-05 | 2021-06-18 | 四川大学华西医院 | Ultrasonic teaching method, device and system and electronic equipment |
AU2022312475A1 (en) * | 2021-07-16 | 2024-01-18 | Vantari Pty Ltd | Ultrasound simulation |
WO2023183699A2 (en) * | 2022-03-25 | 2023-09-28 | Exo Imaging, Inc. | Method and system for cross-referencing of two-dimensional (2d) ultrasound scans of a tissue volume |
Citations (19)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5704791A (en) * | 1995-03-29 | 1998-01-06 | Gillio; Robert G. | Virtual surgery system instrument |
US6857878B1 (en) * | 1998-01-26 | 2005-02-22 | Simbionix Ltd. | Endoscopic tutorial system |
US20060142657A1 (en) * | 2002-03-06 | 2006-06-29 | Mako Surgical Corporation | Haptic guidance system and method |
US20070122785A1 (en) * | 1996-05-08 | 2007-05-31 | Gaumard Scientific Company, Inc. | Interactive Education System for Teaching Patient Care |
WO2009117419A2 (en) * | 2008-03-17 | 2009-09-24 | Worcester Polytechnic Institute | Virtual interactive system for ultrasound training |
US20100152880A1 (en) * | 2008-10-31 | 2010-06-17 | Searete Llc, A Limited Liability Corporation Of The State Of Delware | Systems, devices, and methods for making or administering frozen particles |
US20130054150A1 (en) * | 2010-05-07 | 2013-02-28 | William Sacks | System for tracking female fertility |
US20130189663A1 (en) * | 2012-01-23 | 2013-07-25 | Eidgenossische Technische Hochschule Zurich | Medical training systems and methods |
US8556635B2 (en) * | 2000-10-23 | 2013-10-15 | Christopher C. Toly | Physiological simulator for use as a brachial plexus nerve block trainer |
US20140071165A1 (en) * | 2012-09-12 | 2014-03-13 | Eidgenoessische Technische Hochschule Zurich (Eth Zurich) | Mixed reality simulation methods and systems |
US8786613B2 (en) * | 2006-04-08 | 2014-07-22 | Alan Millman | Method and system for interactive simulation of materials and models |
US20140277678A1 (en) * | 2013-03-15 | 2014-09-18 | General Electric Company | Methods and systems for improving patient engagement via medical avatars |
US20140277739A1 (en) * | 2013-03-15 | 2014-09-18 | Sri International | Exosuit System |
US20140343416A1 (en) * | 2013-05-16 | 2014-11-20 | Intuitive Surgical Operations, Inc. | Systems and methods for robotic medical system integration with external imaging |
US8961190B2 (en) * | 2011-12-20 | 2015-02-24 | Applied Medical Resources Corporation | Advanced surgical simulation |
US20160262689A1 (en) * | 2015-03-11 | 2016-09-15 | Mirca Christina da Silva BATISTA | Wireless electromyograph equipment and operating system thereof |
US9788905B2 (en) * | 2011-03-30 | 2017-10-17 | Surgical Theater LLC | Method and system for simulating surgical procedures |
US9898937B2 (en) * | 2012-09-28 | 2018-02-20 | Applied Medical Resources Corporation | Surgical training model for laparoscopic procedures |
US9940849B2 (en) * | 2013-03-01 | 2018-04-10 | Applied Medical Resources Corporation | Advanced surgical simulation constructions and methods |
Family Cites Families (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2008100623A2 (en) * | 2007-02-16 | 2008-08-21 | Johns Hopkins University | Robust and accurate freehand 3d ultrasound |
US10380920B2 (en) | 2013-09-23 | 2019-08-13 | SonoSim, Inc. | System and method for augmented ultrasound simulation using flexible touch sensitive surfaces |
JP6596436B2 (en) * | 2013-10-18 | 2019-10-23 | ジーバ メディカル, インコーポレイテッド | Methods and systems for the treatment of polycystic ovary syndrome |
WO2016015994A1 (en) * | 2014-07-29 | 2016-02-04 | Koninklijke Philips N.V. | Ultrasound imaging apparatus |
CN107635474B (en) * | 2015-03-31 | 2022-01-07 | 阿布拉护理公司 | Methods and systems for manipulating ovarian tissue |
WO2017064249A1 (en) * | 2015-10-16 | 2017-04-20 | Virtamed Ag | Ultrasound simulation methods |
-
2016
- 2016-10-14 WO PCT/EP2016/074707 patent/WO2017064249A1/en active Application Filing
- 2016-10-14 US US15/768,204 patent/US10453360B2/en active Active
- 2016-10-14 CN CN201680059568.8A patent/CN108352132A/en active Pending
- 2016-10-14 US US15/294,275 patent/US20170110032A1/en not_active Abandoned
Patent Citations (20)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5704791A (en) * | 1995-03-29 | 1998-01-06 | Gillio; Robert G. | Virtual surgery system instrument |
US20070122785A1 (en) * | 1996-05-08 | 2007-05-31 | Gaumard Scientific Company, Inc. | Interactive Education System for Teaching Patient Care |
US6857878B1 (en) * | 1998-01-26 | 2005-02-22 | Simbionix Ltd. | Endoscopic tutorial system |
US8556635B2 (en) * | 2000-10-23 | 2013-10-15 | Christopher C. Toly | Physiological simulator for use as a brachial plexus nerve block trainer |
US20060142657A1 (en) * | 2002-03-06 | 2006-06-29 | Mako Surgical Corporation | Haptic guidance system and method |
US8786613B2 (en) * | 2006-04-08 | 2014-07-22 | Alan Millman | Method and system for interactive simulation of materials and models |
WO2009117419A2 (en) * | 2008-03-17 | 2009-09-24 | Worcester Polytechnic Institute | Virtual interactive system for ultrasound training |
US20100179428A1 (en) * | 2008-03-17 | 2010-07-15 | Worcester Polytechnic Institute | Virtual interactive system for ultrasound training |
US20100152880A1 (en) * | 2008-10-31 | 2010-06-17 | Searete Llc, A Limited Liability Corporation Of The State Of Delware | Systems, devices, and methods for making or administering frozen particles |
US20130054150A1 (en) * | 2010-05-07 | 2013-02-28 | William Sacks | System for tracking female fertility |
US9788905B2 (en) * | 2011-03-30 | 2017-10-17 | Surgical Theater LLC | Method and system for simulating surgical procedures |
US8961190B2 (en) * | 2011-12-20 | 2015-02-24 | Applied Medical Resources Corporation | Advanced surgical simulation |
US20130189663A1 (en) * | 2012-01-23 | 2013-07-25 | Eidgenossische Technische Hochschule Zurich | Medical training systems and methods |
US20140071165A1 (en) * | 2012-09-12 | 2014-03-13 | Eidgenoessische Technische Hochschule Zurich (Eth Zurich) | Mixed reality simulation methods and systems |
US9898937B2 (en) * | 2012-09-28 | 2018-02-20 | Applied Medical Resources Corporation | Surgical training model for laparoscopic procedures |
US9940849B2 (en) * | 2013-03-01 | 2018-04-10 | Applied Medical Resources Corporation | Advanced surgical simulation constructions and methods |
US20140277678A1 (en) * | 2013-03-15 | 2014-09-18 | General Electric Company | Methods and systems for improving patient engagement via medical avatars |
US20140277739A1 (en) * | 2013-03-15 | 2014-09-18 | Sri International | Exosuit System |
US20140343416A1 (en) * | 2013-05-16 | 2014-11-20 | Intuitive Surgical Operations, Inc. | Systems and methods for robotic medical system integration with external imaging |
US20160262689A1 (en) * | 2015-03-11 | 2016-09-15 | Mirca Christina da Silva BATISTA | Wireless electromyograph equipment and operating system thereof |
Cited By (20)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US11627944B2 (en) | 2004-11-30 | 2023-04-18 | The Regents Of The University Of California | Ultrasound case builder system and method |
US11631342B1 (en) | 2012-05-25 | 2023-04-18 | The Regents Of University Of California | Embedded motion sensing technology for integration within commercial ultrasound probes |
US11315439B2 (en) | 2013-11-21 | 2022-04-26 | SonoSim, Inc. | System and method for extended spectrum ultrasound training using animate and inanimate training objects |
US11594150B1 (en) | 2013-11-21 | 2023-02-28 | The Regents Of The University Of California | System and method for extended spectrum ultrasound training using animate and inanimate training objects |
US11600201B1 (en) | 2015-06-30 | 2023-03-07 | The Regents Of The University Of California | System and method for converting handheld diagnostic ultrasound systems into ultrasound training systems |
US10453360B2 (en) * | 2015-10-16 | 2019-10-22 | Virtamed Ag | Ultrasound simulation methods |
US20210134186A1 (en) * | 2017-01-26 | 2021-05-06 | SonoSim, Inc. | System and Method for Multisensory Psychomotor Skill Training |
US11749137B2 (en) * | 2017-01-26 | 2023-09-05 | The Regents Of The University Of California | System and method for multisensory psychomotor skill training |
WO2019035889A1 (en) * | 2017-08-16 | 2019-02-21 | Gaumard Scientific Company, Inc. | Augmented reality system for teaching patient care |
US11551578B2 (en) | 2017-08-16 | 2023-01-10 | Gaumard Scientific Company, Inc. | Augmented reality system for teaching patient care |
CN111465970A (en) * | 2017-08-16 | 2020-07-28 | 科玛科学公司 | Augmented reality system for teaching patient care |
CN109545054A (en) * | 2017-09-21 | 2019-03-29 | 牛志宏 | Human embryo transplants in-vitro simulated research platform |
US11568762B2 (en) | 2017-10-20 | 2023-01-31 | American Association of Gynecological Laparoscopists, Inc. | Laparoscopic training system |
US11189195B2 (en) * | 2017-10-20 | 2021-11-30 | American Association of Gynecological Laparoscopists, Inc. | Hysteroscopy training and evaluation |
CN108806446A (en) * | 2017-11-24 | 2018-11-13 | 安徽农业大学 | A kind of production method of diagnosis of early gestation teaching mode and application |
US11810473B2 (en) | 2019-01-29 | 2023-11-07 | The Regents Of The University Of California | Optical surface tracking for medical simulation |
US11495142B2 (en) * | 2019-01-30 | 2022-11-08 | The Regents Of The University Of California | Ultrasound trainer with internal optical tracking |
US20230061192A1 (en) * | 2019-01-30 | 2023-03-02 | The Regents Of The University Of California | Ultrasound Trainer with Internal Optical Tracking |
US11769427B2 (en) * | 2019-01-30 | 2023-09-26 | The Regents Of The University Of California | Ultrasound trainer with internal optical tracking |
CN113450634A (en) * | 2021-05-21 | 2021-09-28 | 重庆市人民医院 | Prostate biopsy training model under supersound |
Also Published As
Publication number | Publication date |
---|---|
CN108352132A (en) | 2018-07-31 |
WO2017064249A1 (en) | 2017-04-20 |
US20180301063A1 (en) | 2018-10-18 |
US10453360B2 (en) | 2019-10-22 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US10453360B2 (en) | Ultrasound simulation methods | |
US20200402425A1 (en) | Device for training users of an ultrasound imaging device | |
US9424761B2 (en) | Medical simulation system and method with configurable anatomy model manufacturing | |
US9142145B2 (en) | Medical training systems and methods | |
Sutherland et al. | An augmented reality haptic training simulator for spinal needle procedures | |
US20100179428A1 (en) | Virtual interactive system for ultrasound training | |
Bano et al. | Simulation of pneumoperitoneum for laparoscopic surgery planning | |
TW202038867A (en) | Optical tracking system and training system for medical equipment | |
De Visser et al. | Developing a next generation colonoscopy simulator | |
Riener et al. | VR for medical training | |
Zara et al. | Uterus | |
Faso | Haptic and virtual reality surgical simulator for training in percutaneous renal access | |
US20180055570A1 (en) | Low-Resistance Anatomical Tissue Model | |
Henshall et al. | Towards a high fidelity simulation of the kidney biopsy procedure | |
Li et al. | Haptics-equiped interactive PCI simulation for patient-specific surgery training and rehearsing. | |
ÖZKADİF et al. | Contribution of virtual anatomic models to medical education | |
Sutherland et al. | Towards an augmented ultrasound guided spinal needle insertion system | |
CN105078579A (en) | Simulation training system for nasal endoscopic surgery navigation | |
Nicolau et al. | A low cost simulator to practice ultrasound image interpretation and probe manipulation: Design and first evaluation | |
Abolmaesumi et al. | A haptic-based system for medical image examination | |
Kim et al. | Palpation simulator for laparoscopic surgery with haptic feedback | |
Rasool et al. | Image-driven haptic simulation of arthroscopic surgery | |
TWI441106B (en) | An image based surgery training system | |
Ourahmoune et al. | A survey of echographic simulators. | |
Rajagopal et al. | Finite element modelling of breast biomechanics: Predicting the effects of gravity |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: VIRTAMED AG, SWITZERLAND Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:O'BRIEN, CAROLYN;BEUDEKER, ERIKA;SIGNING DATES FROM 20161122 TO 20161123;REEL/FRAME:040887/0152 |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: RESPONSE TO NON-FINAL OFFICE ACTION ENTERED AND FORWARDED TO EXAMINER |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: FINAL REJECTION MAILED |
|
STCB | Information on status: application discontinuation |
Free format text: ABANDONED -- FAILURE TO RESPOND TO AN OFFICE ACTION |