CN114167974B

CN114167974B - Heart operation simulation method and device based on VR

Info

Publication number: CN114167974B
Application number: CN202111260819.8A
Authority: CN
Inventors: 张晓慎; 李小辉; 陆华; 张琰霖; 廖秋英
Original assignee: Jinan University
Current assignee: Jinan University
Priority date: 2021-10-28
Filing date: 2021-10-28
Publication date: 2022-08-09
Anticipated expiration: 2041-10-28
Also published as: CN114167974A; WO2023072190A1

Abstract

The invention discloses a heart surgery simulation method and a heart surgery simulation device based on VR (virtual reality), wherein the method comprises the following steps: receiving an operation instruction transmitted by a user; the operation instructions comprise operation instructions and/or medication instructions; generating a real-time operation response picture according to the operation instruction, a preset response rule and the heart three-dimensional model; the operation response picture is used for being displayed to the user through VR wearable equipment; judging whether the operation instruction accords with a preset safety rule or not, and if not, generating an error prompt picture; and the error prompt picture is used for being displayed to the user through VR wearable equipment. Therefore, the invention can simulate the heart operation through the VR technology and safely judge the operation of the user, thereby realizing more vivid heart operation simulation effect, being beneficial to correcting the error of the user in the heart operation simulation, and further improving the training effect when the invention is used for doctor training.

Description

Heart operation simulation method and device based on VR

Technical Field

The invention relates to the technical field of VR (virtual reality), in particular to a heart surgery simulation method and device based on VR.

Background

With the development of VR technology, VR devices are being applied in more and more fields. Wherein, the medical field is also the hot application scene of VR equipment, and in the operation simulation, more and more institutions adopt VR equipment to provide experience or training of heart operation simulation for the doctor. However, in the prior art, the heart surgery simulation thought based on VR generally only utilizes the real-time response function of VR equipment, but does not further analyze the operation of the user, so that the defects of the prior art are found, and a solution is needed.

Disclosure of Invention

The technical problem to be solved by the invention is to provide a heart surgery simulation method and device based on VR, which can realize more realistic heart surgery simulation effect and is beneficial to correcting the error of a user in the heart simulation surgery, thereby improving the training effect when doctors are trained by using the method and device.

In order to solve the above technical problem, a first aspect of the present invention discloses a VR-based cardiac surgery simulation method, which includes:

receiving an operation instruction transmitted by a user; the operation instructions comprise operation instructions and/or medication instructions;

generating a real-time operation response picture according to the operation instruction, a preset response rule and the heart three-dimensional model; the operation response picture is used for being displayed to the user through VR wearable equipment;

judging whether the operation instruction accords with a preset safety rule or not, and if not, generating an error prompt picture; and the error prompt picture is used for being displayed to the user through VR wearable equipment.

As an optional implementation manner, in the first aspect of the present invention, before the receiving the operation instruction transmitted by the user, the method further includes:

receiving the cardiac surgery type transmitted by the user;

generating a heart simulation operation picture according to the heart operation type and a preset heart three-dimensional model; the heart simulation operation picture is used for being displayed to the user through VR wearable equipment.

As an optional implementation manner, in the first aspect of the present invention, the determining whether the operation instruction meets a preset safety rule includes:

inputting the operation instruction into a trained safety judgment neural network model to obtain the danger probability corresponding to the operation instruction; the safety judgment neural network model is obtained by training a training data set comprising a plurality of training operation instructions marked with safety labels or danger labels;

and when the danger probability is larger than a preset probability threshold value, judging that the operation instruction does not accord with a preset safety rule.

As an optional implementation manner, in the first aspect of the present invention, the safety judgment neural network model is a CNN network model, which is trained through the following steps:

determining a safety judgment training model; the safety judgment training model comprises a safety judgment neural network model and a loss function calculation layer; the loss function of the loss function calculation layer is a cross entropy loss function;

determining a training data set comprising a plurality of training operation instructions labeled with safety labels or danger labels;

inputting the training operation instruction and the labeled labels in the training data set to the safety judgment training model in a paired label mode for training, continuously reducing the loss function value calculated by the loss function layer by using a random gradient reduction method in the training process, and optimizing the safety judgment training model until convergence so as to obtain the trained safety judgment neural network model.

As an alternative implementation, in the first aspect of the present invention, the determining a training data set including a plurality of training operation instructions labeled with security tags or danger tags includes:

receiving a plurality of historical operating instructions of the user and corresponding subsequent instructions; the subsequent instructions are one or more operation instructions in the next preset time period of the occurrence time point of the historical operation instruction;

for each historical operating instruction, judging the instruction type of the historical operating instruction to be a regret instruction or a persistence instruction according to the subsequent instruction corresponding to the historical operating instruction;

and determining a training data set comprising a plurality of training operation instructions marked with safety labels or dangerous labels according to the plurality of historical operation instructions of the user and the corresponding instruction types.

As an optional implementation manner, in the first aspect of the present invention, the determining, according to the subsequent instruction corresponding to the historical operation instruction, that the instruction type of the historical operation instruction is a repentance instruction or a persistence instruction includes:

judging whether the subsequent instruction corresponding to the historical operating instruction and the historical operating instruction are opposite operations in at least one operating dimension; the operation dimension comprises at least one of an operation direction, an operation position, a medication direction, a medication position, a medication dose, an operation attribute and a medication attribute;

if so, determining the instruction type of the historical operation instruction as a regret instruction;

if the judgment result is negative, the instruction type of the historical operation instruction is determined to be a persistence instruction.

As an optional implementation manner, in the first aspect of the present invention, the determining, according to a plurality of historical operating instructions of the user and corresponding instruction types, a training data set including a plurality of training operating instructions labeled with security tags or danger tags includes;

determining a plurality of operation instruction categories;

determining the number proportion of the repentance instructions in the instruction types of a plurality of historical operating instructions corresponding to each operating instruction category;

if the quantity occupation ratio of the regret instructions is higher than a preset occupation ratio threshold, determining the operation instruction type as a dangerous type;

if the quantity proportion of the repentance instructions is lower than a preset proportion threshold, determining the operation instruction type as a safety type;

and determining the plurality of historical operating instructions corresponding to the danger category as training operating instructions marked with danger labels, and determining the plurality of historical operating instructions corresponding to the safety category as training operating instructions marked with safety labels, so as to obtain a training data set comprising the plurality of training operating instructions marked with safety labels or danger labels.

In a second aspect, the present invention discloses a VR-based cardiac surgery simulation apparatus, comprising:

the receiving module is used for receiving an operation instruction transmitted by a user; the operation instructions comprise operation instructions and/or medication instructions;

the response module is used for generating a real-time operation response picture according to the operation instruction, a preset response rule and the heart three-dimensional model; the operation response picture is used for being displayed to the user through VR wearable equipment;

the judging module is used for judging whether the operation instruction accords with a preset safety rule or not, and if not, generating an error prompt picture; and the error prompt picture is used for being displayed to the user through VR wearable equipment.

As an alternative embodiment, in the second aspect of the present invention, the receiving module is further configured to receive the cardiac surgery type transmitted by the user; the response module is also used for generating a heart simulation operation picture according to the heart operation type and a preset heart three-dimensional model; the heart simulation operation picture is used for being displayed to the user through VR wearable equipment.

As an optional implementation manner, in the second aspect of the present invention, a specific manner of determining whether the operation instruction conforms to a preset safety rule by the determining module includes:

and when the danger probability is greater than a preset probability threshold value, judging that the operation instruction does not accord with a preset safety rule.

As an optional implementation manner, in the second aspect of the present invention, the safety judgment neural network model is a CNN network model, and the apparatus further includes a training module, which is trained to obtain the safety judgment neural network model by the following steps:

and inputting the training operation instructions and the labeled labels in the training data set into the safety judgment training model in a paired label mode for training, continuously reducing the loss function value calculated by the loss function layer by using a random gradient reduction method in the training process, and optimizing the safety judgment training model until convergence so as to obtain the trained safety judgment neural network model.

As an alternative embodiment, in the second aspect of the present invention, the specific manner of determining the training data set including a plurality of training operation instructions labeled with security tags or danger tags by the training module includes:

As an optional implementation manner, in the second aspect of the present invention, a specific manner in which the training module determines, according to the subsequent instruction corresponding to the historical operating instruction, that the instruction type of the historical operating instruction is a repentance instruction or a persistent instruction includes:

judging whether the subsequent instruction corresponding to the historical operating instruction and the historical operating instruction are opposite operations in at least one operation dimension; the operation dimension comprises at least one of an operation direction, an operation position, a medication direction, a medication position, a medication dosage, an operation attribute and a medication attribute;

As an optional implementation manner, in the second aspect of the present invention, the training module determines, according to a plurality of historical operating instructions of the user and corresponding instruction types, a specific manner of a training data set including a plurality of training operating instructions labeled with security tags or danger tags, including;

determining a plurality of operation instruction categories;

In a third aspect, the present invention discloses another VR-based cardiac surgery simulation apparatus, the apparatus comprising:

a memory storing executable program code;

a processor coupled with the memory;

the processor invokes the executable program code stored in the memory to perform some or all of the steps of the VR based cardiac procedure simulation method disclosed in the first aspect of the embodiments herein.

The invention discloses a fourth aspect of the invention further discloses a VR-based cardiac surgery simulation device, which comprises VR wearing equipment, operating equipment and data processing equipment which are connected with each other; the data processing device is used for executing part or all of the steps of the VR-based cardiac surgery simulation method disclosed in the first aspect of the embodiments of the invention.

Compared with the prior art, the embodiment of the invention has the following beneficial effects:

in the embodiment of the invention, an operation instruction transmitted by a user is received; the operation instructions comprise surgical operation instructions and/or medication instructions; generating a real-time operation response picture according to the operation instruction, a preset response rule and the heart three-dimensional model; the operation response picture is used for being displayed to the user through VR wearable equipment; judging whether the operation instruction accords with a preset safety rule or not, and if not, generating an error prompt picture; and the error prompt picture is used for being displayed to the user through VR wearable equipment. Therefore, the invention can simulate the heart operation through the VR technology and safely judge the operation of the user, thereby realizing more vivid heart operation simulation effect, being beneficial to correcting the error of the user in the heart operation simulation, and further improving the training effect when the invention is used for doctor training.

Drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings needed to be used in the description of the embodiments will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.

FIG. 1 is a schematic flow chart of a VR-based cardiac surgery simulation method according to an embodiment of the present invention;

FIG. 2 is a schematic structural diagram of a VR-based cardiac surgery simulation apparatus according to an embodiment of the present invention;

FIG. 3 is a schematic structural diagram of another VR-based cardiac surgery simulation apparatus according to an embodiment of the present disclosure.

Detailed Description

In order to make the technical solutions of the present invention better understood, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

The terms "first," "second," and the like in the description and claims of the present invention and in the above-described drawings are used for distinguishing between different objects and not for describing a particular order. Furthermore, the terms "include" and "have," as well as any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, apparatus, article, or article that comprises a list of steps or elements is not limited to only those steps or elements listed, but may alternatively include other steps or elements not listed, or inherent to such process, method, article, or article.

Reference herein to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the invention. The appearances of the phrase in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. It is explicitly and implicitly understood by one skilled in the art that the embodiments described herein can be combined with other embodiments.

The invention discloses a heart surgery simulation method and device based on VR, which can simulate heart surgery through VR technology and carry out safety judgment on the operation of a user, thereby realizing more vivid heart surgery simulation effect, being beneficial to correcting the error of the user in the heart simulation surgery and further improving the training effect when doctors are trained by using the method and device. The following are detailed below.

Example one

Referring to fig. 1, fig. 1 is a schematic flow chart of a VR-based cardiac surgery simulation method according to an embodiment of the present invention. The method described in fig. 1 may be applied to a corresponding VR simulation terminal, VR simulation equipment, or VR simulation server, and the server may be a local server or a cloud server. Specifically, as shown in fig. 1, the VR-based cardiac surgery simulation method may include the operations of:

101. and receiving an operation instruction transmitted by a user.

Optionally, the operation instructions include surgical operation instructions and/or medication instructions. Optionally, the surgical operation instruction may include at least one of a scalpel operation instruction, a surgical scissors operation instruction, a vascular forceps operation instruction, a needle holder operation instruction, a surgical forceps operation instruction, a catheter operation instruction, a guidewire operation instruction, a puncture needle operation instruction, a vascular sheath operation instruction, a balloon operation instruction, and a biopsy needle operation instruction. Alternatively, the administration instructions may include the amount of drug of any drug type that is operated by at least one of an infusion set, a micro-pump, and a blood transfusion set.

102. And generating a real-time operation response picture according to the operation instruction, a preset response rule and the heart three-dimensional model.

Optionally, the operation response screen is used for being displayed to the user through the VR wearable device. Optionally, the preset response rule may be a response rule of the three-dimensional model, for example, a three-dimensional model collision movement rule preset by an operator, which is used for modifying the three-dimensional model of the heart in response to the operation instruction, or may be a medical data presentation rule based on pathological pharmacology, for example, a medication response rule according to medical knowledge, which is used for adaptively modifying some pathological data in response to a medication instruction of the user.

103. And judging whether the operation instruction accords with a preset safety rule or not, and if not, generating an error prompt picture.

Optionally, the error prompt screen is used for being displayed to the user through the VR wearable device.

Optionally, the preset safety rule may be formulated by a physician, for example, the preset safety rule may include an operation result of a specific operation in a specific cardiac operation type, and when the operation result is a dangerous result, such as a fatal result, if the operation instruction is the specific operation, it is determined that the operation instruction does not meet the preset safety rule, and an error prompt screen is generated and displayed to the user.

Therefore, the method described by the embodiment of the invention can simulate the cardiac surgery through VR technology and judge the operation of the user safely, thereby realizing more vivid cardiac surgery simulation effect, being beneficial to correcting the error of the user in the cardiac surgery simulation, and further improving the training effect when the method is used for doctor training.

As an optional implementation manner, before receiving the operation instruction transmitted by the user in the above step, the method further includes:

receiving the cardiac surgery type transmitted by the user;

and generating a heart simulation operation picture according to the heart operation type and a preset heart three-dimensional model.

Optionally, the heart simulation surgery picture is used for displaying to the user through the VR wearable device.

Alternatively, the type of cardiac surgery may be for the purpose of cardiac surgery, such as bypass surgery or valve replacement, or for conditions targeted by cardiac surgery, such as congenital heart disease, valvular heart disease, coronary heart disease, thoracic aortic aneurysm, pericardial disease, and heart tumor. Optionally, the heart simulation operation picture may include a heart three-dimensional model and other peripheral scenes and data presentations, for example, a human body model including the heart three-dimensional model and heart data may be presented, wherein partial features of the human body model and details of the heart data may be determined by the type of the heart operation input by the user.

Therefore, by implementing the optional implementation mode, the heart simulation operation picture can be generated according to the heart operation type and the preset heart three-dimensional model, so that the vivid heart simulation operation picture can be more accurately determined to be displayed to a user, and the immersion feeling during the heart simulation operation by using the method is further improved.

As an optional implementation manner, in the step, determining whether the operation instruction meets a preset safety rule includes:

inputting the operation instruction into the trained safety judgment neural network model to obtain the danger probability corresponding to the operation instruction;

Optionally, the safety judgment neural network model is obtained by training a training data set including a plurality of training operation instructions labeled with safety labels or danger labels.

Therefore, by implementing the optional implementation mode, the operation instruction can be input into the trained safety judgment neural network model to obtain the danger probability corresponding to the operation instruction, and whether the operation instruction meets the preset safety rule or not is further judged according to the danger probability, so that the operation correctness of the user can be judged by using the advantages of the neural network algorithm, the error of the user in the heart simulation operation can be corrected, and the training effect of the invention for training the doctor can be further improved.

As an optional implementation manner, the safety judgment neural network model is a CNN network model, which is obtained by training through the following steps:

determining a safety judgment training model, wherein the safety judgment training model comprises a safety judgment neural network model and a loss function calculation layer, and specifically, a loss function of the loss function calculation layer is a cross entropy loss function;

and inputting training operation instructions in the training data set and labeled labels into a safety judgment training model in a paired label mode for training, continuously reducing a loss function value calculated by a loss function layer by using a random gradient reduction method in the training process, and optimizing the safety judgment training model until convergence so as to obtain a trained safety judgment neural network model.

Therefore, by implementing the optional implementation mode, the training operation instructions in the training data set and the labeled labels can be input to the safety judgment training model in the form of paired labels for training, and the safety judgment training model is optimized until convergence, so that the trained safety judgment neural network model is obtained, and the accurate and efficient neural network model can be determined, so that the operation correctness of the user can be judged by using the advantages of the neural network algorithm in the subsequent process, and the training effect of the invention for the training of doctors is improved.

As an alternative embodiment, in the above step, determining a training data set including a plurality of training operation commands labeled with security tags or danger tags includes:

receiving a plurality of historical operation instructions of a user and corresponding subsequent instructions;

for each historical operating instruction, judging the instruction type of the historical operating instruction to be a regret instruction or a persistence instruction according to a subsequent instruction corresponding to the historical operating instruction;

and determining a training data set comprising a plurality of training operation instructions marked with safety labels or dangerous labels according to a plurality of historical operation instructions of the user and corresponding instruction types.

Optionally, the subsequent instruction is one or more operation instructions in a next preset time period from the occurrence time point of the historical operation instruction.

Therefore, by implementing the optional implementation mode, the instruction type of the historical operation instruction is judged to be a repentance instruction or a persistence instruction according to the subsequent instruction corresponding to the historical operation instruction, and then the training data set is determined according to the plurality of historical operation instructions of the user and the corresponding instruction types, so that an effective training data set can be efficiently determined, a precise and efficient neural network model can be conveniently obtained by subsequent training, and the judgment of the operation correctness of the user by utilizing the advantages of the neural network algorithm is facilitated.

As an optional implementation manner, in the step, determining that the instruction type of the historical operation instruction is a repentance instruction or a persistence instruction according to a subsequent instruction corresponding to the historical operation instruction includes:

judging whether a subsequent instruction corresponding to the historical operation instruction and the historical operation instruction are opposite operations in at least one operation dimension;

Optionally, the operation dimension includes at least one of an operation direction, an operation position, a medication direction, a medication position, a medication amount, an operation attribute, and a medication attribute. Alternatively, the opposite operation is not necessarily used to refer to the opposite of the physical properties of the two, but may also be used to refer to the opposite of the medical concept, for example, the application medication operation and the erase medication operation may be determined as the opposite operation, or the first medicine application operation and the second medicine application operation may be determined as the opposite operation when the medical properties of the first medicine and the second medicine may cancel each other.

Therefore, by implementing the optional implementation mode, the instruction type of the historical operating instruction can be judged to be a repentance instruction or a persistence instruction according to whether the subsequent instruction corresponding to the historical operating instruction and the historical operating instruction are opposite operations in at least one operation dimension, so that the instruction type of the historical operating instruction can be determined according to the self operation reaction of the user, an effective training data set can be determined efficiently subsequently, and an accurate and efficient neural network model can be obtained by facilitating subsequent training.

As an optional implementation manner, in the above step, a training data set including a plurality of training operation instructions marked with security tags or danger tags is determined according to a plurality of historical operation instructions of the user and corresponding instruction types, including;

determining a plurality of operation instruction categories;

determining the quantity proportion of repentance instructions in the instruction types of a plurality of historical operating instructions corresponding to each operating instruction category;

if the quantity proportion of the regret instructions is higher than a preset proportion threshold, determining the operation instruction type as a dangerous type;

if the quantity proportion of the regret instructions is lower than a preset proportion threshold, determining the operation instruction type as a safety type;

determining a plurality of historical operating instructions corresponding to the danger category as training operating instructions marked with danger labels, and determining a plurality of historical operating instructions corresponding to the safety category as training operating instructions marked with safety labels, so as to obtain a training data set comprising a plurality of training operating instructions marked with safety labels or danger labels.

Therefore, by implementing the optional implementation mode, the type of the operation instruction type can be judged according to the quantity proportion of the repentance instructions in the instruction types of the multiple historical operation instructions corresponding to each operation instruction type, and then the training operation instructions and the corresponding labels are further determined, so that an effective training data set can be determined according to the self operation reaction of the user, and the accurate and efficient neural network model can be obtained through subsequent training.

Example two

Referring to fig. 2, fig. 2 is a schematic structural diagram of a VR-based cardiac surgery simulation apparatus according to an embodiment of the present invention. The apparatus described in fig. 2 may be applied to a corresponding VR analog terminal, VR analog device, or VR analog server, and the server may be a local server or a cloud server, which is not limited in the embodiment of the present invention. Specifically, as shown in fig. 2, the apparatus may include:

the receiving module 201 is configured to receive an operation instruction transmitted by a user.

And the response module 202 is configured to generate a real-time operation response picture according to the operation instruction, a preset response rule, and the three-dimensional cardiac model.

Optionally, the operation response screen is used for being displayed to the user through the VR wearable device. Optionally, the preset response rule may be a response rule of the three-dimensional model, for example, a collision and movement rule of the three-dimensional model preset by an operator, which is used for modifying the three-dimensional model of the heart in response to the operation instruction, or a medical data presentation rule based on pathological pharmacology, for example, a medication response rule according to medical knowledge, which is used for adaptively modifying some pathological data in response to a medication instruction of the user.

The judging module 203 is configured to judge whether the operation instruction meets a preset safety rule, and if not, generate an error prompt screen.

Therefore, the device described by the embodiment of the invention can simulate the cardiac surgery through VR technology and judge the operation of the user safely, thereby realizing more vivid cardiac surgery simulation effect, being beneficial to correcting the error of the user in the cardiac surgery simulation, and further improving the training effect when the device is used for training doctors.

As an optional embodiment, the receiving module 201 is further configured to receive the cardiac surgery type transmitted by the user; the response module 202 is further configured to generate a heart simulation surgery picture according to the heart surgery type and the preset three-dimensional model of the heart.

As an optional implementation manner, the specific manner of determining, by the determining module 203, whether the operation instruction meets the preset safety rule includes:

As an optional implementation manner, the safety judgment neural network model is a CNN network model, and the apparatus further includes a training module, which is trained to obtain the safety judgment neural network model by the following steps:

As an alternative embodiment, the specific manner of determining the training data set including a plurality of training operation instructions labeled with security tags or danger tags by the training module includes:

receiving a plurality of historical operating instructions of a user and corresponding subsequent instructions;

As an optional implementation manner, the specific manner in which the training module determines, according to a subsequent instruction corresponding to the historical operating instruction, that the instruction type of the historical operating instruction is a regret instruction or a persistent instruction includes:

judging whether a subsequent instruction corresponding to the historical operating instruction and the historical operating instruction are opposite operations in at least one operating dimension;

As an optional implementation manner, the training module determines a specific manner of a training data set including a plurality of training operation instructions marked with security tags or danger tags according to a plurality of historical operation instructions of a user and corresponding instruction types, including;

determining a plurality of operation instruction categories;

if the quantity occupation ratio of the repentance instructions is higher than a preset occupation ratio threshold, determining the operation instruction type as a dangerous type;

Therefore, by implementing the optional implementation mode, the type of the operation instruction type can be judged according to the quantity proportion of the repentance instructions in the instruction types of the plurality of historical operation instructions corresponding to each operation instruction type, and the training operation instructions and the corresponding labels are further determined, so that an effective training data set can be determined according to the self operation reaction of the user, and the accurate and efficient neural network model can be obtained through subsequent training.

EXAMPLE III

Referring to fig. 3, fig. 3 is a schematic structural diagram of another VR-based cardiac surgery simulation apparatus according to an embodiment of the disclosure. As shown in fig. 3, the apparatus may include:

a memory 301 storing executable program code;

a processor 302 coupled to the memory 301;

the processor 302 invokes the executable program code stored in the memory 301 to perform some or all of the steps of a VR based cardiac procedure simulation method as disclosed in one embodiment of the present invention.

Example four

The embodiment of the invention discloses a computer storage medium, which stores computer instructions, and when the computer instructions are called, the computer instructions are used for executing part or all of the steps in the VR-based heart surgery simulation method disclosed by the embodiment of the invention.

EXAMPLE five

The embodiment of the invention discloses a still another VR-based cardiac surgery simulation device, which can comprise VR wearing equipment, operating equipment and data processing equipment which are connected with each other, wherein the data processing equipment is used for executing part or all of the steps in the VR-based cardiac surgery simulation method disclosed in the embodiment of the invention. Optionally, the data processing apparatus may include a simulation server for rendering the virtual simulation scene, a virtual database connected to the simulation server and for storing the virtual environment data, and a real scene data generation unit connected to the simulation server and for generating the actual operation scene. Optionally, the operating device may comprise a surgical operating means for performing a simulated surgical operation on the heart model and a drug administration operating means for performing a simulated drug administration on the heart model. The virtual database is connected with a communication interface, an operation data acquisition device and an operation data communication module are sequentially connected between the operation device and the real scene data generation unit, a medication flow acquisition device and a medication flow communication module are sequentially connected between the medication operation device and the real scene data generation unit, and the output end of the simulation server is connected with a VR simulation display platform, a simulation sound effect module and the VR wearable equipment. Alternatively, the surgical devices can be divided into two types, one being conventional surgical devices including, but not limited to, scalpels, surgical scissors, vascular clamps, needle holders, and forceps. Another is an interventional surgical device including, but not limited to, catheters, guidewires, puncture needles, vascular sheaths, balloons, and biopsy needles. Optionally, the data acquisition device acquires action state data of the operation device, the action state data is transmitted to the live-action data generation unit through the operation data communication module, the medicine flow acquisition device acquires medicine flow of the medicine operation device, and the medicine flow is transmitted to the live-action data generation unit through the medicine flow communication module. Optionally, the generating unit includes an openscene graph three-dimensional engine and adopts a vital planet builder tool. Optionally, the real-scene data generating unit may generate the simulation posture according to the actual postures of the operation data collecting device and the medication flow collecting device. Optionally, in order to enhance the reality of the simulated operation, the operation data communication module and the medication flow communication module both adopt wireless network communication modules. Optionally, the virtual database stores a surgical scene, a surgical operation model and a medication operation model, the surgical operation model includes a surgical operation device virtual model and surgical operation posture data, the medication operation model includes a medication operation device virtual model and medication data, and the heart simulation data includes heart physiological parameters and heart simulation posture data corresponding to the heart physiological parameters. Optionally, the simulation server combines the simulation posture generated by the real-scene data generation unit and the operation scene, the operation model and the medication operation model in the virtual database, and renders to obtain a virtual-real hybrid simulation scene, so as to achieve the technical effect of combining the actual operation and the virtual scene, so that a user can visually and clearly perceive medical effects of different operation, different medicaments and different medicament dosages on the heart physiological parameters, and the simulation server is suitable for scenes of medical anatomy, medical cognition, operation simulation exercise and the like, is favorable for design of preoperative schemes, is favorable for growth of surgeons in practice, and has a good use effect. Optionally, the operation data acquisition device includes a microcontroller, and a three-axis acceleration sensor, an image positioning module and a displacement sensor connected to the microcontroller. Optionally, the image positioning module includes a positioning camera and an image processing module connected to the positioning camera. Optionally, the three-axis acceleration sensor is configured to detect a posture of the surgical operation device in an XYZ space direction, the image positioning module is configured to detect a position of the surgical operation device relative to the heart model, and the displacement sensor is configured to detect a displacement of the surgical operation device. According to the detection results of the triaxial acceleration sensor, the image positioning module and the displacement sensor, the space posture and the motion of the operation device can be judged. Optionally, the administration operating device comprises an infusion set, a micro pump and a blood transfusion set. In practical use, different medicines or blood can be injected into the heart or vein or artery in the heart three-dimensional model by using different medicine application operating devices, so that a user can conveniently and visually and clearly observe the influence of different medicines on myocardial cell contraction, the influence on heart rhythm, the influence on ventricular fibrillation and the like. Optionally, the administration flow collecting device detects the administration dosage of the administration operation device, so that a user can visually and clearly observe the influence of drugs with different dosages on myocardial cell contraction, the heart rhythm, the ventricular fibrillation and the like by setting the administration flow collecting device. Optionally, this VR wearing equipment is head-mounted VR equipment. Optionally, VR wearing equipment is wear-type VR glasses, perception VR image directly perceived, the operation of simulation reality.

While certain embodiments of the present description have been described above, other embodiments are within the scope of the following claims. In some cases, the actions or steps recited in the claims may be performed in a different order than in the embodiments and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily have to be in the particular order shown or in sequential order to achieve desirable results. In some embodiments, multitasking and parallel processing may also be possible or may be advantageous.

The embodiments in the present specification are described in a progressive manner, and the same and similar parts among the embodiments are referred to each other, and each embodiment focuses on the differences from the other embodiments. In particular, for the apparatus, device, and non-volatile computer-readable storage medium embodiments, since they are substantially similar to the method embodiments, the description is relatively simple, and in relation to the description, reference may be made to some portions of the description of the method embodiments.

The apparatus, the device, the nonvolatile computer readable storage medium, and the method provided in the embodiments of the present specification correspond to each other, and therefore, the apparatus, the device, and the nonvolatile computer storage medium also have similar advantageous technical effects to the corresponding method.

In the 90 s of the 20 th century, improvements in a technology could clearly distinguish between improvements in hardware (e.g., improvements in circuit structures such as diodes, transistors, switches, etc.) and improvements in software (improvements in process flow). However, as technology advances, many of today's process flow improvements have been seen as direct improvements in hardware circuit architecture. Designers almost always obtain the corresponding hardware circuit structure by programming an improved method flow into the hardware circuit. Thus, it cannot be said that an improvement in the process flow cannot be realized by hardware physical modules. For example, a Programmable Logic Device (PLD), such as a Field Programmable Gate Array (FPGA), is an integrated circuit whose Logic functions are determined by programming the Device by a user. A digital system is "integrated" on a PLD by the designer's own programming without requiring the chip manufacturer to design and fabricate application-specific integrated circuit chips. Furthermore, nowadays, instead of manually making an integrated Circuit chip, such Programming is often implemented by "logic compiler" software, which is similar to a software compiler used in program development and writing, but the original code before compiling is also written by a specific Programming Language, which is called Hardware Description Language (HDL), and HDL is not only one but many, such as abel (advanced Boolean Expression Language), ahdl (alternate Language Description Language), traffic, pl (core unified Programming Language), HDCal, JHDL (Java Hardware Description Language), langue, Lola, HDL, laspam, hardsradware (Hardware Description Language), vhjhd (Hardware Description Language), and vhigh-Language, which are currently used in most common. It will also be apparent to those skilled in the art that hardware circuitry that implements the logical method flows can be readily obtained by merely slightly programming the method flows into an integrated circuit using the hardware description languages described above.

The controller may be implemented in any suitable manner, for example, the controller may take the form of, for example, a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro) processor, logic gates, switches, an Application Specific Integrated Circuit (ASIC), a programmable logic controller, and an embedded microcontroller, examples of which include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320, the memory controller may also be implemented as part of the control logic for the memory. Those skilled in the art will also appreciate that, in addition to implementing the controller as pure computer readable program code, the same functionality can be implemented by logically programming method steps such that the controller is in the form of logic gates, switches, application specific integrated circuits, programmable logic controllers, embedded microcontrollers and the like. Such a controller may thus be considered a hardware component, and the means included therein for performing the various functions may also be considered as a structure within the hardware component. Or even means for performing the functions may be regarded as being both a software module for performing the method and a structure within a hardware component.

The systems, devices, modules or units illustrated in the above embodiments may be implemented by a computer chip or an entity, or by a product with certain functions. One typical implementation device is a computer. In particular, the computer may be, for example, a personal computer, a laptop computer, a cellular telephone, a camera phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

For convenience of description, the above devices are described as being divided into various units by function, and are described separately. Of course, the functions of the various elements may be implemented in the same one or more software and/or hardware implementations of the present description.

As will be appreciated by one skilled in the art, the present specification embodiments may be provided as a method, system, or computer program product. Accordingly, embodiments of the present description may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, embodiments of the present description may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and so forth) having computer-usable program code embodied therein.

The description has been presented with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the description. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

In a typical configuration, a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.

The memory may include forms of volatile memory in a computer readable medium, Random Access Memory (RAM) and/or non-volatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM). Memory is an example of a computer-readable medium.

Computer-readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), Read Only Memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), Digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices, or any other non-transmission medium which can be used to store information that can be accessed by a computing device. As defined herein, a computer readable medium does not include a transitory computer readable medium such as a modulated data signal and a carrier wave.

It should also be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in the process, method, article, or apparatus that comprises the element.

This description may be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. The specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media including memory storage devices.

The embodiments in the present specification are described in a progressive manner, and the same and similar parts among the embodiments are referred to each other, and each embodiment focuses on the differences from the other embodiments. In particular, for the system embodiment, since it is substantially similar to the method embodiment, the description is simple, and for the relevant points, reference may be made to the partial description of the method embodiment.

Finally, it should be noted that: the VR-based cardiac surgery simulation method and apparatus disclosed in the embodiments of the present invention are only preferred embodiments of the present invention, which are only used for illustrating the technical solutions of the present invention, and are not limited thereto; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those skilled in the art; the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and the modifications or the substitutions do not make the essence of the corresponding technical solutions depart from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A VR-based cardiac procedure simulation method, the method comprising:

judging whether the operation instruction accords with a preset safety rule or not, and if not, generating an error prompt picture; the error prompt picture is used for being displayed to the user through VR wearable equipment;

the judging whether the operation instruction conforms to a preset safety rule includes:

when the danger probability is larger than a preset probability threshold value, judging that the operation instruction does not accord with a preset safety rule;

the safety judgment neural network model is a CNN network model and is obtained by training through the following steps:

inputting the training operation instructions and the labeled labels in the training data set to the safety judgment training model in a paired label mode for training, continuously reducing the loss function values calculated by the loss function layer by using a random gradient reduction method in the training process, and optimizing the safety judgment training model until convergence so as to obtain the trained safety judgment neural network model;

the determining a training data set including a plurality of training operation instructions labeled with security tags or danger tags includes:

2. The VR-based cardiac procedure simulation method of claim 1, wherein prior to receiving the user-transmitted operating instructions, the method further comprises:

receiving the cardiac surgery type transmitted by the user;

3. The VR-based cardiac surgery simulation method of claim 1, wherein the determining that the instruction type of the historical manipulation instruction is a repentance instruction or a persistence instruction according to the follow-up instruction corresponding to the historical manipulation instruction includes:

4. The VR-based cardiac procedure simulation method of claim 1, wherein the determining a training data set including a plurality of training operational instructions labeled with safety tags or danger tags based on a plurality of historical operational instructions and corresponding instruction types of the user includes;

determining a plurality of operation instruction categories;

5. A VR-based cardiac surgery simulation apparatus for implementing a cardiac surgery simulation method as claimed in any one of claims 1 to 4, the apparatus comprising:

the receiving module is used for receiving an operation instruction transmitted by a user; the operation instructions comprise surgical operation instructions and/or medication instructions;

6. A VR-based cardiac surgery simulation apparatus, the apparatus comprising:

a memory storing executable program code;

a processor coupled with the memory;

the processor invokes the executable program code stored in the memory to perform the VR based cardiac procedure simulation method of any of claims 1-4.

7. A VR-based cardiac surgery simulation device is characterized by comprising VR wearing equipment, operating equipment and data processing equipment which are connected with each other; the data processing device is for performing the VR based cardiac procedure simulation method of any one of claims 1-4.