TWI765370B - Surgery teaching auxiliary system using virtual reality (vr) and method thereof - Google Patents

Abstract

A surgery teaching auxiliary system using virtual reality (VR) and method thereof is disclosed. By establishing a VR surgical environment through three-dimensional reconstruction based on a two-dimensional surgical video, and pre-setting a plurality of operation steps, time point of using instruments and reference information. When performing a surgical training in the VR surgical environment, continuously detects a surgical operation, and triggers an auxiliary support and guidance according to a progress of the surgical training when detecting the surgical operation is abort or delay. The mechanism is help to improve the learning effectiveness of the surgical training of The VR.

Description

Surgical teaching assistant system and method using virtual reality

The invention relates to a surgical teaching assistant system and a method thereof, in particular to a surgical teaching assistant system and a method using virtual reality.

In recent years, with the popularization and vigorous development of virtual reality, various applications based on virtual reality have sprung up. In addition to common applications in games, virtual reality is also often used in teaching, such as: surgery, aviation Flight and other simulation teaching and training.

Generally speaking, the traditional surgical training or teaching method is performed on a general or animal basis, but this method has the problem of irrecoverable in addition to the high cost. Therefore, some manufacturers use virtual reality in surgical teaching, for example: simulating human organs or simply demonstrating the surgical process in virtual reality. However, current simulated human organs are still not realistic enough for teaching, training or training. On the other hand, the degree of freedom of surgical operation provided by virtual reality is also insufficient, so the effectiveness of its application in surgical teaching still has its limitations.

In view of this, some manufacturers have proposed technologies to improve the degree of realism and freedom to increase the immersive experience. However, high fidelity and freedom do not necessarily improve learning effectiveness. For example, when you don’t know how to operate next, even if you provide an operating environment with high fidelity and high degrees of freedom, you still don’t know how to operate. Therefore, if it is possible to actively detect whether help is needed, and to intervene in a timely manner to assist and guide, it can be expected to have a positive effect on improving the effectiveness of surgical learning.

To sum up, it can be seen that in the prior art, the problem of ineffective surgical learning in virtual reality has existed for a long time. Therefore, it is necessary to propose improved technical means to solve this problem.

The present invention discloses a surgical teaching assistant system and method using virtual reality.

First, the present invention discloses a surgical teaching assistant system using virtual reality, which includes: a surgical image database, a building module, a training module and a support module. Among them, the surgical image database stores two-dimensional surgical images, and the two-dimensional surgical images each correspond to the name of the surgical procedure, the operation steps, the time point of using the instrument, and the reference information; a module is established to connect the surgical image database to be used for surgical training. Before, receive the name of the surgery to select and load one of the corresponding two-dimensional surgical images, and create a virtual reality surgical environment in a three-dimensional reconstruction manner according to the loaded two-dimensional surgical images; the training module is connected to the establishment module, using After the virtual reality surgical environment is established, surgical training is allowed, and during the surgical training, the surgical operation behavior is continuously detected. When the surgical operation behavior is abnormally terminated or delayed, the corresponding time point is generated according to the progress of the surgical training. information; and the support module is connected to the training module, so as to play the loaded 2D surgical image according to the generated time-point information, and synchronously display the operation steps, the time-point of using the instrument and the reference information for auxiliary support and guidance.

In addition, the present invention also discloses a surgical teaching assistant method using virtual reality, the steps of which include: providing a plurality of two-dimensional surgical images, the two-dimensional surgical images respectively correspond to the name of the surgery, the operation steps, the time point of using the instrument and the reference information ; Before performing the surgical training, receive the name of the surgical procedure to select and load one of the corresponding two-dimensional surgical images, and create a virtual reality surgical environment in a three-dimensional reconstruction manner according to the loaded two-dimensional surgical image; After the real surgical environment is established, surgical training is allowed, and during the surgical training, the surgical operation behavior is continuously detected, and when the surgical operation behavior is abnormally terminated or delayed, corresponding time-point information is generated according to the progress of the surgical training; and Plays the loaded 2D surgical image according to the generated time point information, and simultaneously displays the operation steps, the time point of using the instrument and the reference information for auxiliary support and guidance.

The system and method disclosed in the present invention are as above, and the difference from the prior art is that the present invention establishes a virtual reality surgical environment by means of three-dimensional reconstruction on the basis of two-dimensional surgical images, and presets corresponding operation steps, use Instrument timing and reference information, when performing surgical training in virtual reality, continuously detect surgical operation behavior, and when detecting abnormal termination or delay of surgical operation behavior, trigger corresponding auxiliary support and guidance according to the progress of surgical training .

Through the above-mentioned technical means, the present invention can achieve the technical effect of improving the surgical learning effect in virtual reality.

The embodiments of the present invention will be described in detail below in conjunction with the drawings and examples, so as to fully understand and implement the implementation process of how the present invention applies technical means to solve technical problems and achieve technical effects.

First, before describing the surgical teaching assistant system and method using virtual reality disclosed in the present invention, the environment in which the present invention is applied will be described first. The present invention is applied in a virtual reality, and the virtual reality refers to Using computer analogy to generate a virtual world of three-dimensional space, it provides users with a simulation of vision and other senses, so that users feel as if they are in the real world, and can observe things in three-dimensional space instantly and without restrictions.

The following is a further description of the auxiliary system for surgery teaching using virtual reality and the method of the present invention in conjunction with the drawings. Please refer to "Figure 1" first. Figure 1 is the system of the auxiliary system for surgery teaching using virtual reality in the present invention. In the block diagram, the system includes: a surgical image database 110 , a building module 120 , a training module 130 and a support module 140 . The surgical image database 110 is used for storing two-dimensional surgical images, and the two-dimensional surgical images respectively correspond to the name of the surgical procedure, the operation steps, the time point of using the instrument, and the reference information. For example, suppose there is a two-dimensional surgical image, and the corresponding surgical name is "myomectomy", then the two-dimensional surgical image also corresponds to the operation steps, the time point of using the instrument, and the reference information. Among them, the operation steps are the execution steps of the surgical procedure, the time point of using the instruments is to record the various surgical instruments that need to be used in this procedure and their use time points (or "time points"), and the reference information can record the needs of this procedure. Information to note.

The establishment module 120 is connected to the surgical image database 110, and is used to receive the name of the surgical procedure to select and load one of the corresponding two-dimensional surgical images before performing the surgical training, and to use the three-dimensional surgical image according to the loaded two-dimensional surgical image. The reconstruction method establishes a virtual reality surgical environment. In practice, the 3D reconstruction refers to the mathematical process and computer technology for recovering the 3D information (shape, etc.) of an object by using 2D projection or images, which can be achieved through algorithms such as photometry, geometry, and deep learning. , which is used to convert the two-dimensional surgical image into three-dimensional information of virtual reality.

The training module 130 is connected to the establishment module 120 to allow surgical training after the virtual reality surgical environment is established, and to continuously detect the surgical operation behavior during the surgical training. When the surgical operation behavior is abnormally terminated or delayed , according to the progress of surgical training to generate corresponding time-point information. In actual implementation, the surgical operation behavior without abnormal termination or delay can be input into the machine learning model for training, and after the training is completed, the detected surgical operation behavior can be input into the machine learning model to identify the surgical operation Whether the behavior is aborted or hysteretic. In addition, the surgical operation behavior can be detected through at least one of a motion sensor, a wrist wearable device, and a hand dynamic input device.

The support module 140 is connected to the training module 130 to play the loaded 2D surgical image according to the generated time point information, and synchronously display the operation steps, the time point of using the instrument and the reference information for auxiliary support and guidance. For example, assuming that the generated time-point message is "10 minutes and 30 seconds", the support module 140 will start to play the loaded 2D surgical image from 10 minutes and 30 seconds, and synchronously display the corresponding operation steps and instruments used. Timing and reference information for auxiliary support and guidance. In other words, when the user's surgical operation is abnormally terminated or delayed, the support module 140 will immediately display this time point (or simply "time point") and the subsequent surgical operation steps, the time point of using the surgical instrument, And the normal physiological data, precautions, etc. at this time point are provided to the user as reference information.

It should be noted that, in practice, the modules described in the present invention can be implemented in various ways, including software, hardware, or any combination thereof. For example, in some embodiments, each module can be implemented by using Software and hardware or one of them can be implemented. In addition, the present invention can also be implemented partially or completely based on hardware. For example, one or more modules in the system can be implemented through integrated circuit chips, system Single chip (System on Chip, SoC), Complex Programmable Logic Device (Complex Programmable Logic Device, CPLD), Field Programmable Gate Array (Field Programmable Gate Array, FPGA) etc. The present invention may be a system, method and/or computer program. The computer program may include a computer-readable storage medium on which computer-readable program instructions for causing a processor to implement various aspects of the present invention are loaded, and the computer-readable storage medium may be a tangible material that may hold and store instructions for use by the instruction execution device equipment. Computer-readable storage media can be, but are not limited to, electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer-readable storage media include: hard disks, random access memory, read-only memory, flash memory, optical disks, floppy disks, and any suitable combination of the foregoing. Computer-readable storage media, as used herein, are not to be construed as transient signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (eg, optical signals through fiber optic cables), or through electrical wires transmitted electrical signals. Additionally, the computer-readable program instructions described herein may be downloaded from computer-readable storage media to various computing/processing devices, or downloaded over a network such as the Internet, a local area network, a wide area network, and/or a wireless network to an external computer device or external storage device. Networks may include copper transmission cables, fiber optic transmissions, wireless transmissions, routers, firewalls, switches, hubs and/or gateways. The network card or network interface in each computing/processing device receives computer-readable program instructions from the network and forwards the computer-readable program instructions for storage on the computer-readable storage medium in each computing/processing device middle. The computer program instructions that perform the operations of the present invention may be assembled language instructions, instruction set architecture instructions, machine instructions, machine dependent instructions, microinstructions, firmware instructions, or source or object code written in any combination of one or more programming languages (Object Code), the programming language includes object-oriented programming languages, such as: Common Lisp, Python, C++, Objective-C, Smalltalk, Delphi, Java, Swift, C#, Perl, Ruby and PHP, etc., as well as conventional programs Procedural programming language, such as C language or similar programming language. The computer program instructions may execute entirely on the computer, partly on the computer, as a stand-alone software, partly on the client computer and partly on the remote computer, or entirely on the remote computer or server execute on.

Please refer to "Fig. 2A" and "Fig. 2B", "Fig. 2A" and "Fig. 2B" are flow charts of the method for the auxiliary method of surgery teaching using virtual reality according to the present invention, and the steps include: providing a plurality of two 2D surgical images, each of which corresponds to the name of the procedure, the operation steps, the time point of using the instrument and the reference information (step 210 ); before the surgical training, the name of the procedure is received to select and load the corresponding two-dimensional One of the surgical images is obtained, and a virtual reality surgical environment is established in a three-dimensional reconstruction manner according to the loaded two-dimensional surgical images (step 220 ); During the surgical training, the surgical operation behavior is continuously detected, and when the surgical operation behavior is abnormally terminated or delayed, corresponding timing information is generated according to the progress of the surgical training (step 230 ); and the loaded two-dimensional surgical image is played according to the generated timing information. , and simultaneously display the operation steps, the time point of using the appliance and the reference information for auxiliary support and guidance (step 240 ). Through the above steps, a virtual reality surgical environment can be established by 3D reconstruction on the basis of 2D surgical images, and the corresponding operation steps, time points and reference information for using instruments can be preset. During surgical training, the surgical operation behavior is continuously detected, and when abnormal suspension or delay of the surgical operation behavior is detected, corresponding auxiliary support and guidance are triggered according to the progress of the surgical training.

In addition, as shown in "Fig. 2B", step 230 can be further divided into two steps: after the virtual reality surgical environment is established, allowing surgical training, and during the surgical training, continuously detecting the surgical operation behavior (Step 231 ); and input the detected surgical operation behavior into a machine learning model that is pre-trained, so as to identify whether the detected surgical operation behavior is abnormally terminated or delayed through the machine learning model, wherein the trained machine The generation method of the learning model is generated by continuously inputting the untrained machine learning model to the untrained machine learning model for training in advance (step 232 ).

The following description will be given in the form of an embodiment in conjunction with "Fig. 3" and "Fig. 4". As shown in "Fig. 3", "Fig. 3" is a schematic diagram of applying the present invention to perform surgical operations in virtual reality. When the user wants to perform a virtual reality surgical operation, the user can first input the name of the surgical procedure, such as myomectomy, through the controller 310b, so as to load the corresponding two-dimensional surgical image from the surgical image database 110 . Next, the creation module 120 uses the loaded 2D surgical images to create a virtual reality surgical environment in a 3D reconstruction manner. Surgical environment, and the virtual surgical instrument (311a, 311b) can be controlled by operating the controller (310a, 310b) to perform surgical training. At this time, the training module 130 will continue to detect the surgical operation behavior, and when the surgical operation behavior is abnormally terminated or delayed, it will generate corresponding timing information according to the progress of the surgical training, so that the support module 140 can play the loaded data according to the timing information. Two-dimensional surgical images. For example, if the surgical training is 10 minutes and 30 seconds and an abnormal termination or delay of the surgical operation is detected, the time-point message is 10 minutes and 30 seconds, and the support module 140 will play the download from 10 minutes and 30 seconds. The imported 2D surgical image, and the simultaneous display of the operation steps, the time point of using the instrument and the reference information for auxiliary support and guidance.

Please refer to “FIG. 4”, “FIG. 4” is a schematic diagram of detecting abnormal or delayed surgical operation behavior by applying the present invention. In actual implementation, when the training module 130 detects that the surgical operation is abnormally terminated or delayed, the warning icon 302 will be displayed as shown in "Fig. 4", and the first display block 301, the second Display block 303 and display block 304 in time. Among them, the second display block 303 will be used to display the 2D surgical image loaded according to the time point information, and synchronously display the operation steps, the time point of using the instrument and the reference information for auxiliary support and guidance; the time point display block 304 displays Time-point information, such as: fifteen minutes can be displayed as "15:00". In actual implementation, the icons 320 of the surgical instruments to be used can also be displayed in sequence according to the time of using the instruments, which is convenient for the user to be mentally prepared to switch the surgical instruments. In other words, when the training module 130 detects that the surgical operation is abnormally terminated or delayed, it no longer simply displays the first display block 301 and the virtual surgical environment in it, but supports and guides the user with various information. For example, the alert icon 302 is used to remind the user that the current surgical operation is abnormally aborted or delayed, and the second display block 303 is used to display the two-dimensional surgical image corresponding to the time point of the abnormal abort or the delay, and display at the time point Block 304 displays the point in time at which the abort or lag occurred, and even displays icons 320 of surgical instruments to be used in sequence according to this point in time. In this way, when the user is not sure about the next steps or is completely unclear about the next steps, he can provide immediate intervention assistance and guidance, greatly improving the effect of virtual reality surgery learning.

To sum up, it can be seen that the difference between the present invention and the prior art lies in that a virtual reality surgical environment is established by means of three-dimensional reconstruction on the basis of two-dimensional surgical images, and the corresponding operation steps, the time point of using the instrument and the Reference information, when performing surgical training in virtual reality, continuously detect surgical operation behavior, and when abnormal termination or delay of surgical operation behavior is detected, corresponding auxiliary support and guidance are triggered according to the progress of surgical training. This technical means can solve the problems existing in the prior art, thereby achieving the technical effect of improving the surgical learning effect in virtual reality.

Although the present invention is disclosed above by the aforementioned embodiments, it is not intended to limit the present invention. Anyone who is familiar with the similar arts can make some changes and modifications without departing from the spirit and scope of the present invention. The scope of patent protection shall be determined by the scope of the patent application attached to this specification.

110: Surgical Image Database 120: Build Mods 130: Training Module 140: Support Mods 300: Graphical User Interface 301: The first display block 302: Warning icon 303: Second display block 304: Time point display block 310a, 310b: Controller 311a, 311b: Virtual Surgical Instruments 320: Illustration of surgical instruments Step 210: Provide a plurality of two-dimensional surgical images, each of which corresponds to a surgical procedure name, an operation step, a time point of using an instrument, and a reference information Step 220: Before performing the surgical training, receive the name of the surgical procedure to select and load one of the corresponding two-dimensional surgical images, and create a virtual reality in a three-dimensional reconstruction manner according to the loaded two-dimensional surgical images surgical environment Step 230: After the surgical environment of the virtual reality is established, allow surgical training, and during the surgical training, continuously detect a surgical operation behavior, when the surgical operation behavior is abnormally terminated or delayed, according to the operation training The progress generates corresponding point-in-time messages Step 231: After the surgical environment of the virtual reality is established, allow surgical training, and continuously detect the surgical operation behavior during the surgical training Step 232: Input the detected surgical operation behavior into a machine learning model that has been pre-trained to identify whether the detected surgical operation behavior is abnormally terminated or delayed by the machine learning model, wherein the trained operation behavior is completed. The generation method of the machine learning model is generated by continuously inputting the untrained machine learning model without abnormal termination or delay of surgical operation behavior in advance for training. Step 240: Play the loaded 2D surgical image according to the generated time point information, and simultaneously display the operation steps, the time point of using the instrument and the reference information for auxiliary support and guidance

FIG. 1 is a system block diagram of a surgical teaching assistant system using virtual reality according to the present invention. FIG. 2A and FIG. 2B are method flow charts of the method for assisting surgery teaching using virtual reality according to the present invention. FIG. 3 is a schematic diagram of applying the present invention to perform surgical operations in virtual reality. FIG. 4 is a schematic diagram of the application of the present invention in detecting abnormal or delayed surgical operation behavior.

110: Surgical Image Database 120: Build Mods 130: Training Module 140: Support Mods

Claims (6)

A surgical teaching assistant system using virtual reality, the system includes: a surgical image database for storing a plurality of two-dimensional surgical images, the two-dimensional surgical images respectively correspond to a surgical name, an operation step, a use An instrument time point and a reference information; a building module, connected to the surgical image database, for receiving the surgical name to select and load one of the corresponding two-dimensional surgical images before performing surgical training, and according to The loaded two-dimensional surgical image creates a virtual reality surgical environment in a three-dimensional reconstruction manner; a training module is connected to the establishment module for allowing surgery to be performed after the virtual reality surgical environment is established training, and during the surgical training, continuously detect a surgical operation behavior, when the surgical operation behavior is abnormally terminated or delayed, a corresponding time-point message is generated according to the progress of the surgical training; and a support module is connected to the training module The group is used to play the loaded two-dimensional surgical image according to the generated time-point information, and to simultaneously display the operation steps, the time-point of using the instrument and the reference information for auxiliary support and guidance. The surgical teaching assistance system using virtual reality as claimed in claim 1, wherein the three-dimensional reconstruction is calculated by at least one of photometry, geometry and deep learning, so as to convert the two-dimensional surgical image into virtual reality three-dimensional information. The surgical teaching assistance system using virtual reality according to claim 1, wherein the surgical operation behavior is detected by at least one of a motion sensor, a wrist wearable device, and a hand dynamic input device. A surgical teaching assistant method using virtual reality, the steps of which include: providing a plurality of two-dimensional surgical images, each of the two-dimensional surgical images corresponding to a surgical name, an operation step, a time point of using an instrument and a reference information; Before performing the surgical training, receive the name of the surgical procedure to select and load one of the corresponding two-dimensional surgical images, and create a virtual reality surgical environment in a three-dimensional reconstruction manner according to the loaded two-dimensional surgical images; After the surgical environment of the virtual reality is established, surgical training is allowed, and during the surgical training, a surgical operation behavior is continuously detected. When the surgical operation behavior is abnormally terminated or delayed, a corresponding operation is generated according to the progress of the surgical training. and play the loaded two-dimensional surgical image according to the generated time-point information, and synchronously display the operation steps, the time-point of using the instrument and the reference information for auxiliary support and guidance. The surgical teaching aid method using virtual reality as claimed in claim 4, wherein the three-dimensional reconstruction is calculated by at least one of photometry, geometry and deep learning, so as to convert the two-dimensional surgical image into virtual reality three-dimensional information. The surgical teaching assistance method using virtual reality according to claim 4, wherein the surgical operation behavior is detected by at least one of a motion sensor, a wrist wearable device, and a hand dynamic input device.

Images