KR102298417B1 - Program and method for generating surgical simulation information - Google Patents

Abstract

Generating, by a computer, 3D modeling data including the surgical site of the object, obtaining virtual surgical data for the 3D modeling data, and generating cue sheet data including one or more detailed surgical operations using the virtual surgical data A method of generating surgical simulation information is disclosed, including the step of:

Description

Method and program for generating surgical simulation information {PROGRAM AND METHOD FOR GENERATING SURGICAL SIMULATION INFORMATION}

The present invention relates to a method and program for generating surgical simulation information.

In the course of surgery, the development of technologies capable of providing information for assisting a surgeon's operation is required. In order to provide information to assist the operation, it must be able to recognize the operation.

In the past, in order to design a scenario to optimize the surgical process, pre-imaged medical images were referred to or a very experienced doctor was consulted. It was difficult to get advice.

Therefore, it was difficult to utilize medical images or the advice of an experienced doctor as an auxiliary purpose for optimizing the surgical process for the patient to be operated on.

Accordingly, the surgical process is optimized by minimizing unnecessary processes in performing surgery using a 3D medical image (eg, a 3D surgical tool movement and a virtual image of a change in an organ that occurs due to the movement of the tool) and development of a method capable of providing surgical assistance information based on this is required.

In addition, recently, deep learning has been widely used in the analysis of medical images. Deep learning is defined as a set of machine learning algorithms that attempt high-level abstractions (summarizing key contents or functions in large amounts of data or complex data) through a combination of several nonlinear transformation methods. Deep learning can be viewed as a field of machine learning that teaches computers to think in a broad framework.

The problem to be solved by the present invention is to provide a method and a program for generating surgical simulation information.

The problems to be solved by the present invention are not limited to the problems mentioned above, and other problems not mentioned will be clearly understood by those skilled in the art from the following description.

In a method for generating surgical simulation information according to an aspect of the present invention for solving the above-described problems, the computer generates 3D modeling data including a surgical site of an object, and obtains virtual surgery data for the 3D modeling data. and generating cue sheet data including one or more detailed surgical operations using the virtual surgical data.

In addition, the generating of the cue sheet data comprises dividing the virtual surgical data into the one or more detailed surgical operations, but the surgical site included in the virtual surgical data, the type of surgical tool, the number of the surgical tools, the surgical tools It may include dividing the virtual surgical data into the one or more detailed surgical operations based on at least one of the position of the surgical tool, the direction of the surgical tool, and the movement of the surgical tool.

In addition, the acquiring of the 3D modeling data may include acquiring a medical image including the surgical site of the object and generating the 3D modeling data by using the medical image.

Also, the obtaining of the medical image may include obtaining a photographing posture of the object based on an actual surgical posture and obtaining the photographed medical image based on the obtained photographing posture.

In addition, the generating of the 3D modeling data may include generating the 3D modeling data by correcting the obtained medical image based on obtaining the actual surgical posture of the object and the actual surgical posture. have.

In addition, the method may further include determining whether to optimize the generated cue sheet data.

In addition, the step of determining whether to optimize may further include providing surgical guide data based on the cue sheet data determined to be optimized.

In addition, the step of determining whether to optimize may include obtaining optimized cue sheet data and comparing the cue sheet data with the optimized cue sheet data.

In addition, the step of obtaining the optimization cue sheet data includes: obtaining one or more learning cue sheet data; performing reinforcement learning using the one or more learning cue sheet data; and the optimization cue sheet data based on the reinforcement learning result It may include the step of obtaining.

According to an aspect of the present invention for solving the above problems, a computer program stored in a computer-readable recording medium is provided in combination with a computer, which is hardware, to perform the method for generating surgical simulation information according to the disclosed embodiment.

Other specific details of the invention are included in the detailed description and drawings.

According to the disclosed embodiment, surgery simulation information optimized for each patient is generated using information obtained in advance and provided to the doctor during surgery, thereby assisting the operation.

Effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the following description.

1 is a view showing a robotic surgery system according to the disclosed embodiment.
2 is a flowchart illustrating a method of generating surgical simulation information according to an embodiment.
3 is a flowchart illustrating a 3D modeling data generation process by applying a surgical posture according to an embodiment.
4 is a flowchart illustrating a method of generating surgical simulation information according to another exemplary embodiment.
5 is a flowchart illustrating a method for optimizing a surgical process according to an embodiment.

Advantages and features of the present invention and methods of achieving them will become apparent with reference to the embodiments described below in detail in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in various different forms, and only these embodiments allow the disclosure of the present invention to be complete, and those of ordinary skill in the art to which the present invention pertains. It is provided to fully understand the scope of the present invention to those skilled in the art, and the present invention is only defined by the scope of the claims.

The terminology used herein is for the purpose of describing the embodiments and is not intended to limit the present invention. As used herein, the singular also includes the plural unless specifically stated otherwise in the phrase. As used herein, “comprises” and/or “comprising” does not exclude the presence or addition of one or more other components in addition to the stated components. Like reference numerals refer to like elements throughout, and "and/or" includes each and every combination of one or more of the recited elements. Although "first", "second", etc. are used to describe various elements, these elements are not limited by these terms, of course. These terms are only used to distinguish one component from another. Accordingly, it goes without saying that the first component mentioned below may be the second component within the spirit of the present invention.

Unless otherwise defined, all terms (including technical and scientific terms) used herein will have the meaning commonly understood by those of ordinary skill in the art to which this invention belongs. In addition, terms defined in a commonly used dictionary are not to be interpreted ideally or excessively unless specifically defined explicitly.

As used herein, the term “unit” or “module” refers to a hardware component such as software, FPGA, or ASIC, and “unit” or “module” performs certain roles. However, “part” or “module” is not meant to be limited to software or hardware. A “unit” or “module” may be configured to reside on an addressable storage medium or to reproduce one or more processors. Thus, by way of example, “part” or “module” refers to components such as software components, object-oriented software components, class components and task components, processes, functions, properties, Includes procedures, subroutines, segments of program code, drivers, firmware, microcode, circuitry, data, databases, data structures, tables, arrays and variables. Components and functionality provided within “parts” or “modules” may be combined into a smaller number of components and “parts” or “modules” or as additional components and “parts” or “modules”. can be further separated.

In this specification, "image" may mean multi-dimensional data composed of discrete image elements (eg, pixels in a 2D image and voxels in a 3D image). For example, the image may include a medical image of the object obtained by the CT imaging apparatus.

As used herein, an “object” may be a human or an animal, or a part or all of a human or animal. For example, the object may include at least one of organs such as liver, heart, uterus, brain, breast, abdomen, and blood vessels.

As used herein, a “user” may be a medical professional, such as a doctor, a nurse, a clinical pathologist, or a medical imaging specialist, and may be a technician repairing a medical device, but is not limited thereto.

As used herein, "medical image data" is a medical image captured by a medical imaging device, and includes all medical images that can be implemented as a three-dimensional model of the body of an object. "Medical image data" may include a computed tomography (CT) image, a magnetic resonance imaging (MRI), a positron emission tomography (PET) image, and the like.

As used herein, the term "virtual body model" refers to a model generated to match the actual patient's body based on medical image data. The "virtual body model" may be generated by modeling medical image data in 3D as it is, or may be corrected after modeling to be the same as during actual surgery.

As used herein, “virtual surgical data” refers to data including rehearsal or simulation actions performed on a virtual body model. “Virtual surgical data” may be image data on which rehearsal or simulation is performed on a virtual body model in a virtual space, or data recorded on a surgical operation performed on the virtual body model.

As used herein, “actual surgical data” refers to data obtained by actual medical staff performing surgery. "Actual surgical data" may be image data of a surgical site taken during an actual surgical procedure, or may be data recorded on a surgical operation performed in an actual surgical procedure.

As used herein, the term “detailed surgical operation” refers to the smallest unit of surgical operation divided according to specific criteria.

In the present specification, "Cue sheet data" refers to data recorded in order by dividing a specific surgical procedure into detailed surgical operations.

In this specification, "execution cue sheet data" refers to cue sheet data obtained based on virtual surgery data performed by a user simulation.

As used herein, "virtual surgery cue sheet data for training" is included in the execution cue sheet data, and refers to cue sheet data generated based on virtual surgery data obtained by a user performing surgical simulation training.

In this specification, "virtual surgery cue sheet data for reference" refers to cue sheet data for virtual surgery performed by a specific medical person for building big data for learning or guiding the surgical process.

As used herein, "optimized cue sheet data" refers to cue sheet data for an optimized surgical process in terms of operation time or surgical prognosis.

In this specification, "learning cue sheet data" refers to cue sheet data used for learning for calculating the optimized cue sheet data.

As used herein, "surgical guide data" refers to data used as guide information during actual surgery.

As used herein, the term “computer” includes various devices capable of providing a result to a user by performing arithmetic processing. For example, computers include desktop PCs and notebooks (Note Books) as well as smart phones, tablet PCs, cellular phones, PCS phones (Personal Communication Service phones), synchronous/asynchronous A mobile terminal of International Mobile Telecommunication-2000 (IMT-2000), a Palm Personal Computer (PC), a Personal Digital Assistant (PDA), and the like may also be applicable. Also, when a head mounted display (HMD) device includes a computing function, the HMD device may be a computer. In addition, the computer may correspond to a server that receives a request from a client and performs information processing.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a view showing a robotic surgery system according to the disclosed embodiment.

Referring to Figure 1, a schematic diagram of a system capable of performing robotic surgery according to the disclosed embodiment is shown.

According to FIG. 1 , the robotic surgery system includes a medical image capturing device 10 , a server 20 and a controller 30 provided in an operating room, an image capturing unit 36 , a display 32 and a surgical robot 34 . do.

In one embodiment, the robotic surgery is performed by the user using the control unit 30 to control the surgical robot (34). In one embodiment, the robotic surgery may be performed automatically by the control unit 30 without the user's control.

The server 20 is a computing device including at least one processor and a communication unit.

The control unit 30 includes a computing device including at least one processor and a communication unit. In one embodiment, the controller 30 includes hardware and software interfaces for controlling the surgical robot 34 .

The image capturing unit 36 includes at least one image sensor. That is, the image capturing unit 36 includes at least one camera device, and is used to photograph the surgical site. In one embodiment, the imaging unit 36 is used in combination with the surgical robot (34). For example, the imaging unit 36 may include at least one camera coupled to the surgical arm of the surgical robot 34 .

In an embodiment, the image captured by the image capturing unit 36 is displayed on the display 340 .

The control unit 30 receives information necessary for surgery from the server 20 or generates information necessary for surgery and provides it to the user. For example, the controller 30 displays generated or received information necessary for surgery on the display 32 .

For example, the user performs robotic surgery by controlling the movement of the surgical robot 34 by manipulating the controller 30 while viewing the display 32 .

The server 20 generates information necessary for robotic surgery by using medical image data of an object (patient) photographed in advance from the medical imaging device 10 , and provides the generated information to the controller 30 .

The controller 30 provides the information received from the server 20 to the user by displaying it on the display 32 , or controls the surgical robot 34 using the information received from the server 20 .

In one embodiment, the means that can be used in the medical imaging equipment 10 is not limited, for example, CT, X-Ray, PET, MRI, etc., various other medical image acquisition means may be used.

Hereinafter, a method of generating surgical simulation information will be described in detail with reference to the drawings.

2 is a flowchart illustrating a method of generating surgical simulation information according to an embodiment.

Each of the steps shown in FIG. 2 is performed in time series by the server 20 or the controller 30 shown in FIG. 1 . Hereinafter, each step is described as being performed by a computer for convenience of explanation, but the subject of each step is not limited to a specific device, and all or part of it may be performed by the server 20 or the control unit 30 . can

Referring to FIG. 2 , in the method for generating surgical simulation information according to an embodiment of the present invention, the computer generates 3D modeling data including the surgical site of the subject based on medical image data obtained by photographing the surgical site of the subject. step (S200); computer obtaining virtual surgery data for the 3D modeling data from the user (S400); and generating, by the computer, cue sheet data including one or more detailed surgical operations using the virtual surgical data (S600); includes

Hereinafter, detailed description of each step is described.

The computer generates 3D modeling data including the surgical site of the object based on the medical image data obtained by photographing the surgical site of the object (S200). In order for a user to perform a simulation similar to an actual surgery, 3D modeling data identical to that of a real patient is required.

In the past, as medical imaging was performed in a different posture from the actual surgical posture, and the corresponding medical image data was implemented as 3D modeling data as it is, the preoperative simulation failed to provide the effect of practicing the actual surgery in advance.

Specifically, the internal organ arrangement state or organ shape of a patient photographed with a medical imaging device in a state in which the patient is lying down is generally determined by the effect of gravity depending on the angle at which the object 10 is laid. Since the organ arrangement state or the organ shape is different, it cannot provide a simulation situation identical to that of actual surgery.

In addition, since the simulation is performed in a state that does not match the undulation state filled with carbon dioxide inside the body for laparoscopic surgery or robotic surgery, the pre-operative simulation did not provide the effect of practicing the actual operation in advance. In order to solve this problem, it is necessary to generate 3D modeling data identical to the actual surgical state (ie, 3D relief model).

3 is a flowchart illustrating a 3D modeling data generation process by applying a surgical posture according to an embodiment.

In one embodiment, as shown in FIG. 3 , the computer generates 3D modeling data based on medical image data captured by applying an actual surgical posture. Specifically, the computer calculates the patient's imaging posture based on the surgical posture determined based on the patient's location or surgery type (S202); and performing 3D modeling based on the medical image data captured in the photographing posture (S204).

The computer calculates the patient's imaging posture based on the surgical posture determined based on the patient's location of the affected area or the type of surgery (S202). The surgical posture may vary depending on the patient's location of the affected area (ie, the surgical site), the type of surgery, and the like. The computer calculates a photographing posture in which medical images can be taken in the same posture as the patient's surgical posture.

For example, when the upper body is operated in a state where the patient's body is tilted by 15 degrees, the computer may calculate a posture in which the patient's back is supported so that only the upper body is erected by 15 degrees as the photographing posture. By setting only the patient's upper body upright at 15 degrees as the imaging posture, the surgical site may be photographed in the same state as during surgery using a conventional medical imaging apparatus (eg, a CT device).

In addition, for example, the medical imaging apparatus may implement the same imaging posture as the surgical posture, including the function of adjusting the patient's arrangement state. For example, when the medical imaging apparatus is a computed tomography (CT) apparatus, the table of the CT apparatus may be tilted or rotated by a predetermined angle in a predetermined direction. , the gantry may also be inclined by a predetermined angle in a predetermined direction.

Through this, the computer or the user controls the table and the gantry of the medical imaging apparatus to be tilted at the same angle as the surgical posture, and the computer uses the medical image photographing device to tilt the medical image taken from the medical imaging device at the same angle as during the actual operation. data can be obtained.

In addition, the computer may receive the patient's surgical posture from the medical staff, and may directly calculate the surgical posture to be performed based on the patient's location of the affected part, the type of surgery, and the like.

Thereafter, the computer acquires 3D modeling data generated based on the medical image data captured in the photographing posture (S204). The computer may render and generate a 3D modeling image using the 3D medical image of the object 10 . Also, the computer may receive 3D modeling data generated after being photographed in a specific posture by the medical imaging apparatus. That is, the medical image capturing apparatus (eg, a CT apparatus) may perform 3D rendering based on the medical image data to generate 3D modeling data and then transmit it to the computer.

In addition, in another embodiment, the computer generates 3D modeling data by applying a correction algorithm that changes from medical image data photographed under general photographing conditions to a physical state during a specific operation. That is, the computer generates 3D modeling data by applying changes in the body formed by the undulation model according to the body change due to gravity or the injection of carbon dioxide in the tilted state to the medical image data of the patient lying in a normal state.

For example, the correction algorithm may be calculated by learning big data formed by matching medical image data for general imaging conditions and physical conditions with body state data during surgery for specific physical conditions and surgical conditions. The body state data at the time of surgery may include image data obtained by photographing the shape or arrangement of internal organs in the body or image data photographing the body surface during actual operation.

In one embodiment, when forming a relief model during laparoscopic surgery or robotic surgery based on medical image data (eg, CT data) taken under normal conditions, the computer uses the patient's medical image data and the body at the time of surgery. By matching the surface data, it is built as big data for learning. The body surface data is taken by a medical staff before surgery.

For example, medical staff acquire body surface data by photographing the abdominal surface deformed by injecting carbon dioxide before performing laparoscopic surgery or robotic surgery. As the computer learns big data, it creates a correction criterion or correction algorithm that forms 3D modeling data in a general state into a relief model in laparoscopic or robotic surgery. Then, when the new patient's medical image data is obtained, the computer transforms the 3D modeling data generated based on the medical image data based on the correction criteria or the correction algorithm to generate the relief model.

Through this, the computer can generate 3D modeling data similar to the physical state during surgery for a specific patient and provide it to the user.

The computer acquires the virtual surgery data for the 3D modeling data from the user (S400). The user performs a simulation or rehearsal of performing a virtual surgery on the 3D modeling image. For example, the computer may display a virtual surgical tool on the 3D modeling image, and the user may control the virtual surgical tool in various ways to perform a rehearsal of performing surgery on the 3D modeling image.

The virtual surgery data is generated as a user simulates surgery on a 3D modeled patient's virtual body model. The process of acquiring virtual surgery data may be a process in which the user simulates surgery or a process in which the user rehearses under the same conditions immediately before surgery. The computer may receive virtual surgery data from the user in various ways.

In one embodiment, the computer may provide a virtual body model to the user through a virtual reality (VR) device, and may input a surgical operation for the virtual body model through a controller. The controller may have various forms capable of recognizing the movement of the user's hand. For example, since the controller is implemented as a glove type, it is possible to acquire detailed finger movements and real-time placement of hands during a user's operation.

Specifically, the computer provides a virtual body model to the HMD device worn by the user. The computer rotates or enlarges the virtual body model according to the user's operation and provides the virtual body model. Accordingly, the user performs surgery planning and simulation while closely examining the virtual body model implemented in the same way as during the scheduled surgery for the patient.

For example, depending on the body characteristics of the object, blood vessels and organs that need to be operated on may not be visible due to fat such as omentum (serosal membrane). It is possible to establish an optimal surgical plan to reach the target in need of actual surgery. The computer acquires an input for selecting a specific surgical tool by the user, and acquires an operation to perform surgery on a specific area of the virtual body model with the corresponding surgical tool through a controller.

In addition, as an embodiment, the virtual surgery data may be stored as image data in which a surgical operation performed by a user on a virtual body model is implemented. The computer may generate a simulation image based on real-time motion data acquired by the controller, a surgical tool that performed the motion, and a location in the virtual body model on which the motion is performed.

In addition, in another embodiment, the virtual surgery data may be modeling data for the virtual body model itself, the entire surgical tool and motion data used at each point in the simulation or rehearsal process. That is, the virtual surgery data may be a plurality of data sets that an external computer receives and enables to implement a simulation image.

The computer generates cue sheet data including one or more detailed surgical operations from the virtual surgical data (S600). That is, the computer not only generates an image of the virtual surgical procedure performed on the virtual body model by the user, but also divides it into one or more detailed surgical operations to generate a cue sheet indicating the surgical process.

The detailed surgical operation constituting the cue sheet data is the smallest unit of operation constituting the surgical process. Detailed surgical operation can be divided according to several criteria. For example, the detailed surgical operation may include the type of operation (eg, laparoscopic surgery, robotic surgery), the anatomical body part where the operation is performed, the surgical tools used, the number of surgical tools, the direction in which the surgical tools appear on the screen, or It may be divided based on the position, the movement of the surgical tool (eg, forward/retracted), and the like.

The division criteria and the detailed categories included in the division criteria can be directly set by the medical staff learning the actual surgical data. The computer may perform supervised learning according to the division criteria and detailed categories set by the medical staff to divide the virtual surgical data into the smallest unit of detailed surgical operation.

In addition, the division criteria and the detailed categories included in the division criteria can be extracted through the computer's surgical image learning. For example, the computer can calculate the division criteria and categories within each division criterion by deep learning (ie, unsupervised learning) on actual surgical data accumulated as big data. After that, the computer divides the virtual surgery data according to the division criteria generated through learning the actual surgical data to generate cue sheet data.

In addition, as another embodiment, virtual surgery data or actual surgery data may be segmented by identifying whether they correspond to each segmentation criterion through image recognition. That is, the computer recognizes the anatomical organ position corresponding to the division criterion, the surgical tools that appear, the number of each surgical tool, etc. on the screen of the virtual surgery data or the actual surgical data image, and performs division into detailed surgical operation units. can

In addition, in another embodiment, the computer may perform a segmentation process for generating cue sheet data based on the surgical tool movement data included in the actual surgical data or virtual surgery data. The actual surgical data may include various information input in the process of controlling the surgical robot, such as the type and number of surgical tools selected by the user, and information on the movement of each surgical tool, when the user performs robotic surgery, , the virtual surgical data may also include information on the type and number of surgical tools selected by the user during simulation of the virtual body model, and the movement of each surgical tool. Therefore, the computer can perform the division based on the information contained in each time point of the actual surgery data or virtual surgery data.

In addition, in one embodiment, virtual surgery data or actual surgical data includes various types of actions such as resection and suturing, and segmentation is performed according to the segmentation criterion. Specifically, the process of generating cue sheet data by dividing the actual surgical data (eg, actual surgical image) or virtual surgery data simulating gastric cancer surgery into detailed surgical operations will be described.

For example, gastric cancer surgery includes an action of first resecting a part or all of the stomach including the tumor and an action of resectioning the lymph nodes. In addition, various resection and connection techniques are used depending on the condition of gastric cancer. In addition, each action may be divided into a plurality of more detailed actions according to the specific location where the action is taken and the direction of movement of the surgical tool.

For example, the detailed operation of gastric cancer surgery may be first divided into an open stage, an excision stage, a connection stage, and a suturing stage.

In addition, the method of connecting the resected organs includes an extracorporeal anastomosis method in which 4-5 cm or more of the tip of the medius is cut and connected, or an in vivo anastomosis method in which the umbilicus is incised by 3 cm and resection and anastomosis are performed in the abdominal cavity, etc. can be divided in more detail according to such a specific connection method.

Furthermore, each surgical method may be divided into a plurality of detailed surgical operations according to the position and movement of the surgical tool.

Each of the divided detailed surgical operations may be given a standardized name based on the location at which the detailed surgical operation is performed and the movement path of the surgical tool.

In an embodiment, terms used for standardized names may be defined in various ways. For example, when dealing with a specific region on the lower right side of the stomach, the name of the region may be a name commonly used in the medical field, and a more comprehensive or subdivided name defined in the system according to the disclosed embodiment may be used. can be used

Accordingly, the rehearsal image may be organized as information in the form of a cue sheet in which a plurality of actions are sequentially listed based on a standardized name. Similarly, the surgical image in which the user actually performed the operation may also be divided into action units and organized into cue sheet information.

In addition, in one embodiment, the cue sheet data may be generated as code data of a specific number of digits based on a criterion for dividing into detailed surgical operations. In other words, the computer divides the virtual surgical data into standardized detailed surgical motions by applying standardized division criteria and designating detailed categories within the division criteria, and assigns standardized code values to each subcategory to distinguish each detailed surgical motion. Provides standardized code data that can be used.

The computer assigns digitalized code data to each detailed surgical operation by assigning numbers or letters sequentially from the upper category to which a specific detailed surgical operation belongs according to the division criteria application order. Through this, the computer can generate the cue sheet data in a form in which standardized code data of each detailed surgical operation is listed instead of a divided detailed surgical operation image. In addition, the user can share or deliver the simulated surgical procedure by providing only cue sheet data composed of standardized code data.

In addition, in one embodiment, the computer may give a standardized name to the standardized code (Code) of each detailed surgical operation. Through this, the user can select and confirm only the desired surgical operation (or action) part within the entire cue sheet. In addition, in this case, the user can easily grasp the progress of the operation or rehearsal by simply viewing the cue sheet in which each action is sequentially listed based on the standardized name, even if the user does not see all the rehearsal or surgery images.

The cue sheet data may be converted into surgical images using an image database for each detailed surgical operation. The image database may store images corresponding to each code data, and a plurality of images corresponding to each code data may be stored according to circumstances. For example, specific code data may be loaded with different detailed surgical operation images in the image database according to previously performed operations.

In addition, as each cue sheet data is matched with a specific virtual body model and stored, the computer sequentially applies each detailed surgical operation included in the cue sheet data to the virtual body model to reproduce a surgical simulation image.

Accordingly, the image corresponding to the cue sheet may be reproduced from the same point of view as the surgical rehearsal image, or may be reconstructed and reproduced from a different viewpoint. Alternatively, the image may be modeled in 3D, and the viewpoint and position may be adjusted according to a user's manipulation.

In addition, as in FIG. 4, in another embodiment, the computer determines whether to optimize the execution cue sheet data generated based on the user's virtual surgery data (S800); further includes.

The computer judges the adequacy of the trial cue sheet data by comparing the trial cue sheet data and the optimization cue sheet data. For example, the computer determines whether unnecessary detailed surgical operations that delay the operation time are included in the execution cue sheet data generated by the user's simulation, and detailed surgical operations that must be included before or after a specific detailed surgical operation is performed. It is determined whether or not is omitted, and it is determined whether the trial cue sheet data is properly generated to be applied to the actual patient.

The computer may perform a process of calculating the optimized cue sheet data in order to perform evaluation of the trial cue sheet data.

5 is a flowchart illustrating a method for optimizing a surgical process according to an embodiment.

The computer acquires one or more learning cue sheet data (S820). The learning cue sheet data is learning target data that is learned to calculate the optimized cue sheet data. The learning cue sheet data is cue sheet data generated based on actual surgery data (ie, actual surgery cue sheet data) or cue sheet data generated based on simulated virtual surgery data for reference (ie, virtual surgery cue sheet data for reference) may include The actual surgery cue sheet data is generated by the computer dividing the actual surgery data according to the division criteria. The virtual surgery cue sheet data for reference is not obtained in the user's surgical training process, but is generated by performing a simulation for the purpose of constructing learning target data or providing it for reference to trainees.

Thereafter, the computer performs reinforcement learning using the learning cue sheet data (S840). Reinforcement learning is a field of machine learning, in which an agent defined in an environment recognizes the current state and selects the action or action sequence that maximizes the reward among the selectable actions. Reinforcement learning can be summarized as learning how to maximize a reward based on a state transition and a reward according to the state transition.

Then, the computer calculates the optimization cue sheet data using the reinforcement learning results (S860). The optimized cue sheet data is calculated based on the reinforcement learning results based on the conditions such as the shortest surgical time, the minimum bleeding amount, the required operation group, and the required execution order that can reduce the patient's anesthesia time.

The essential operation group is a group of detailed surgical operations that must be essentially performed together in order to perform a specific detailed surgical operation. The essential sequence is a sequence of surgical operations that must be sequentially performed in the process of performing a specific surgery. For example, depending on the type of surgery or the type of surgical operation, surgical operations to appear sequentially and their order may be determined.

In addition, the computer calculates optimal cue sheet data for each situation according to the patient's physical condition, surgical site (eg, tumor tissue) conditions (eg, size, location, etc. of tumor) through reinforcement learning. To this end, the computer utilizes the patient condition, surgical site condition, etc. together with the learning cue sheet data during learning.

In one embodiment, the computer is capable of performing virtual simulation surgery on its own. For example, the computer may generate a surgical process according to the type of surgery and the type of patient based on the disclosed surgical process optimization method, and perform a virtual surgery simulation based on the generated surgical process.

The computer may obtain an optimized surgical process by evaluating the virtual surgery simulation result, and performing reinforcement learning based on the virtual surgery simulation information and the evaluation information on the result.

Even for a model trained to generate an optimal surgical process, it may be difficult to generate an optimized surgical process for each patient and type of surgery because the body structure and type of surgery are different for each patient in the case of actual surgery.

Therefore, the computer generates a surgical process based on the patient's body structure and surgery type using the learned model, performs reinforcement learning by performing virtual surgery simulation, and optimizes the surgical process according to each patient and surgery type. can create

In addition, as in FIG. 4 , in another embodiment, the computer provides surgical guide data based on specific cue sheet data according to the user's request (S1000); further includes. That is, the computer may provide the user with cue sheet data generated by the user performing a simulation or rehearsal during the operation.

The method for generating surgical simulation information according to an embodiment of the present invention described above may be implemented as a program (or application) to be executed in combination with a computer, which is hardware, and stored in a medium.

The above-described program is C, C++, JAVA, machine language, etc. that a processor (CPU) of the computer can read through a device interface of the computer in order for the computer to read the program and execute the methods implemented as a program It may include code (Code) coded in the computer language of Such code may include functional code related to functions defining functions necessary for executing the methods, etc. can do. In addition, the code may further include additional information necessary for the processor of the computer to execute the functions or code related to memory reference for which location (address address) in the internal or external memory of the computer to be referenced. have. In addition, when the processor of the computer needs to communicate with any other computer or server located remotely in order to execute the above functions, the code uses the communication module of the computer to determine how to communicate with any other computer or server remotely. It may further include a communication-related code for whether to communicate and what information or media to transmit and receive during communication.

The storage medium is not a medium that stores data for a short moment, such as a register, a cache, a memory, etc., but a medium that stores data semi-permanently and can be read by a device. Specifically, examples of the storage medium include, but are not limited to, ROM, RAM, CD-ROM, magnetic tape, floppy disk, and optical data storage device. That is, the program may be stored in various recording media on various servers accessible by the computer or in various recording media on the computer of the user. In addition, the medium may be distributed in a computer system connected by a network, and a computer readable code may be stored in a distributed manner.

The steps of a method or algorithm described in relation to an embodiment of the present invention may be implemented directly in hardware, as a software module executed by hardware, or by a combination thereof. A software module may contain random access memory (RAM), read only memory (ROM), erasable programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory, hard disk, removable disk, CD-ROM, or It may reside in any type of computer-readable recording medium well known in the art to which the present invention pertains.

As mentioned above, although embodiments of the present invention have been described with reference to the accompanying drawings, those skilled in the art to which the present invention pertains can realize that the present invention can be embodied in other specific forms without changing its technical spirit or essential features. you will be able to understand Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive.

10: medical imaging equipment
20: server
30: control unit
32: display
34: surgical robot
36: video recording unit

Claims (10)

A method performed by an apparatus comprising:
generating a three-dimensional virtual body model identical to the actual surgical situation of the patient in the ups and down state;
outputting the virtual body model based on virtual reality;
generating virtual surgery data based on a virtual surgery operation performed using the virtual body model in the virtual reality environment;
providing the generated virtual surgery data as surgical guide information during the actual operation of the patient; and
Including; dividing the virtual surgical data into a plurality of detailed surgical operations based on a specific division criterion;
The specific division criterion is,
including at least one of a surgical site included in the virtual surgical data, a type of surgical tool, the number of surgical tools, a position of the surgical tool, a direction of the surgical tool, and a movement of the surgical tool,
The plurality of detailed surgical operations, as the minimum operation unit constituting the surgical process, are sequentially applied to the virtual body model so that the virtual surgery is reproduced,
The dividing step is
A method for generating virtual reality-based surgery simulation information, which is performed according to the division criterion calculated through big data learning constructed from actual operation data of a computer and a detailed category included in the division criterion. delete According to claim 1,
The output step is
outputting the virtual body model to an external virtual reality output device connected to the device;
The generating step is
Obtaining an image of at least one of organs and blood vessels from the CT image of the patient and generating the three-dimensional virtual body model,
acquiring a hand movement corresponding to a virtual surgery simulation operation or a virtual operation planning operation of the user with respect to the virtual body model through a controller connected to the device;
Virtual reality-based surgical simulation information generating method for generating the virtual surgery data based on the obtained hand movement. delete According to claim 1,
Using the virtual surgery data to generate the execution cue sheet data including the plurality of detailed surgical operations; further comprising,
The execution cue sheet data is,
A method for generating virtual reality-based surgical simulation information, including data listing the plurality of detailed surgical operations in order. 6. The method of claim 5,
The execution cue sheet data is,
A method for generating surgical simulation information based on virtual reality, in which standardization codes for the plurality of detailed surgical operations are sequentially listed. 7. The method of claim 6,
The standardized code is
A method for generating virtual reality-based surgical simulation information in which strings corresponding to each category to which the plurality of detailed surgical operations belong are sequentially listed from a higher category. delete A computer device for performing a virtual reality-based surgical simulation information generation method, comprising:
the computer device,
Creates a three-dimensional virtual body model identical to the actual surgical situation of the patient in an ups and down state,
Outputs the virtual body model based on virtual reality,
generating virtual surgery data based on virtual surgery operations performed using the virtual body model in the environment of the virtual reality;
Providing the generated virtual surgery data as surgical guide information during the actual operation of the patient,
Based on a specific division criterion, the virtual surgical data is divided into a plurality of detailed surgical operations,
The specific division criterion is,
including at least one of a surgical site included in the virtual surgical data, a type of surgical tool, the number of surgical tools, a position of the surgical tool, a direction of the surgical tool, and a movement of the surgical tool,
The plurality of detailed surgical operations, as the minimum operation unit constituting the surgical process, are sequentially applied to the virtual body model so that the virtual surgery is reproduced,
When the computer device divides the virtual surgical data into a plurality of detailed surgical operations, according to the division criterion calculated through big data learning constructed from the actual operation data of the computer, and a detailed category included in the division criterion to perform, a computer device. A computer program stored in a computer-readable recording medium in combination with a computer as hardware, so as to perform the method of any one of claims 1, 3, and 5 to 7.

Images