KR102146672B1 - Program and method for providing feedback about result of surgery - Google Patents

Abstract

Obtaining actual surgical cue sheet data consisting of a plurality of detailed surgical operations by dividing the computer based on the actual surgical data acquired in the actual surgical procedure, obtaining reference cue sheet data for the actual surgery, and the actual operation. Disclosed is a method of providing feedback on a surgical result, comprising the step of providing feedback by comparing the surgical cue sheet data with the reference cue sheet data.

Description

Method and program to provide feedback on surgical results {PROGRAM AND METHOD FOR PROVIDING FEEDBACK ABOUT RESULT OF SURGERY}

The present invention relates to a method and program for providing feedback on surgical results.

In the process of surgery, there is a need to develop technologies that can provide information to assist doctors in surgery. In order to provide information to assist the operation, it is necessary to be able to recognize the operation.

Previously, in order to design a scenario for optimizing the surgical process, a pre-recorded medical image was referred to or a very skilled doctor consulted. However, it was difficult to judge unnecessary processes using only medical images. There was a difficult problem to get advice.

Therefore, it was difficult to use medical imaging or advice from an experienced doctor as an auxiliary purpose for optimizing the surgical process for patients to be operated on.

Therefore, the surgical process is optimized by minimizing unnecessary processes in performing surgery by using 3D medical images (for example, 3D surgical tool movements and virtual images of changes in organs caused by tool movements). And, there is a need to develop a method that can provide surgical assistance information based on this.

In addition, in recent years, deep learning has been widely used for analysis of medical images. Deep learning is defined as a set of machine learning algorithms that attempt high-level abstractions (summarizing key contents or functions in a large amount of data or complex data) through a combination of several nonlinear transformation methods. Deep learning can be seen as a branch of machine learning that teaches computers how people think in a large framework.

The problem to be solved by the present invention relates to a method and a program for providing feedback on surgical results.

The problems to be solved by the present invention are not limited to the problems mentioned above, and other problems that are not mentioned will be clearly understood by those skilled in the art from the following description.

A method for providing feedback to a surgical department according to an aspect of the present invention for solving the above-described problems is divided based on actual surgical data acquired in the actual surgical process by a computer, and is composed of a plurality of detailed surgical operations. And acquiring, acquiring reference cue sheet data for the actual surgery, and comparing the actual surgery cue sheet data with the reference cue sheet data to provide feedback.

In addition, the plurality of detailed surgical operations may include at least one of a surgical site included in the actual surgical data, a type of surgical tool, the number of surgical tools, a position of the surgical tool, a direction of the surgical tool, and a movement of the surgical tool. The actual surgical data may be divided based on one.

In addition, the plurality of detailed surgical operations may be assigned at least one of a standardized name and standardized code data, respectively.

In addition, the providing of the feedback may include obtaining search information for searching for at least one of the plurality of detailed surgical operations, at least corresponding to the search information based on the standardized name or the standardized code data It may include extracting one detailed surgical motion and providing feedback on the extracted at least one detailed surgical motion.

In addition, the reference cue sheet data may be cue sheet data optimized for the actual surgery or virtual surgery cue sheet data for reference.

In addition, the providing of the feedback may include comparing the actual surgical cue sheet data with a plurality of detailed surgical operations included in each of the reference cue sheet data, and performing unnecessary detailed surgical operations, missing detailed surgical operations, or It may include determining whether there is an incorrect detailed surgical operation.

In addition, the step of determining whether there is an incorrect detailed surgical operation may include movement of a surgical tool corresponding to the detailed surgical operation included in the reference cue sheet data, and a surgery corresponding to the detailed surgical operation included in the actual surgical cue sheet data. Comparing the movement of the tool and based on the result of the comparison of the movement of the surgical tool, the step of determining whether a detailed surgical operation included in the actual surgical cue sheet data is wrong.

In addition, the method may further include adding the actual surgical cue sheet data to the learning cue sheet data, and performing reinforcement learning on a model for obtaining optimized cue sheet data by using the learning cue sheet data.

In addition, it may further include the step of searching for at least one surgical error situation from the obtained surgical information and providing feedback on the searched surgical error situation.

In addition, obtaining information on the prognosis corresponding to each of the one or more actual surgery cue sheet data including the actual surgery cue sheet data, performing reinforcement learning based on the information on the one or more actual surgery cue sheet data and the prognosis And determining a correlation between the prognosis and at least one detailed surgical operation included in the one or more actual surgical cue sheet data based on the reinforcement learning result.

A computer program stored in a recording medium readable by a computer so as to perform a method of providing feedback on an operation result according to the disclosed embodiment by being combined with a computer as hardware according to an aspect of the present invention for solving the above-described problems Is provided.

Other specific details of the present invention are included in the detailed description and drawings.

According to the disclosed embodiment, by comparing the actual surgical process with a reference, there is an effect of providing feedback on the process and result of the surgery.

According to the disclosed embodiment, there is an effect of providing feedback by extracting a necessary part without the user having to look at all of the surgical images.

The effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the following description.

1 is a diagram showing a robotic surgery system according to the disclosed embodiment.
2 is a flowchart illustrating a method of providing feedback on an operation result according to an exemplary embodiment.
3 is a flowchart illustrating a method of calculating optimized cue sheet data according to an exemplary embodiment.
4 is a flowchart illustrating a method of obtaining feedback according to an exemplary embodiment.

Advantages and features of the present invention, and a method of achieving them will become apparent with reference to the embodiments described below in detail together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in a variety of different forms, only the present embodiments are intended to complete the disclosure of the present invention, It is provided to fully inform the technician of the scope of the present invention, and the present invention is only defined by the scope of the claims.

The terms used in the present specification are for describing exemplary embodiments and are not intended to limit the present invention. In this specification, the singular form also includes the plural form unless specifically stated in the phrase. As used in the specification, “comprises” and/or “comprising” do not exclude the presence or addition of one or more other elements other than the mentioned elements. Throughout the specification, the same reference numerals refer to the same elements, and “and/or” includes each and all combinations of one or more of the mentioned elements. Although "first", "second", and the like are used to describe various elements, it goes without saying that these elements are not limited by these terms. These terms are only used to distinguish one component from another component. Therefore, it goes without saying that the first component mentioned below may be the second component within the technical idea of the present invention.

Unless otherwise defined, all terms (including technical and scientific terms) used in the present specification may be used as meanings that can be commonly understood by those of ordinary skill in the art to which the present invention belongs. In addition, terms defined in a commonly used dictionary are not interpreted ideally or excessively unless explicitly defined specifically.

The term "unit" or "module" used in the specification refers to a hardware component such as software, FPGA or ASIC, and the "unit" or "module" performs certain roles. However, "unit" or "module" is not meant to be limited to software or hardware. The “unit” or “module” may be configured to be in an addressable storage medium, or may be configured to reproduce one or more processors. Thus, as an example, "sub" or "module" refers to components such as software components, object-oriented software components, class components, and task components, processes, functions, properties, It includes procedures, subroutines, segments of program code, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays and variables. Components and functions provided within "sub" or "module" may be combined into a smaller number of components and "sub" or "modules" or into additional components and "sub" or "modules". Can be further separated.

In the present specification, "image" may mean multi-dimensional data composed of discrete image elements (eg, pixels in a 2D image and voxels in a 3D image). For example, the image may include a medical image of an object acquired by a CT imaging apparatus.

In the present specification, the "object" may be a human or an animal, or a part or all of a human or an animal. For example, the object may include at least one of organs such as liver, heart, uterus, brain, breast, and abdomen, and blood vessels.

In the present specification, the "user" may be a medical expert, such as a doctor, a nurse, a clinical pathologist, a medical imaging expert, and the like, and may be a technician who repairs a medical device, but is not limited thereto.

In the present specification, "medical image data" is a medical image captured by a medical imaging device, and includes all medical images capable of implementing a body of an object as a 3D model. "Medical image data" may include Computed Tomography (CT) images, Magnetic Resonance Imaging (MRI), Positron Emission Tomography (PET) images, and the like.

In the present specification, the "virtual body model" refers to a model created according to an actual patient's body based on medical image data. The "virtual body model" may be generated by modeling medical image data in 3D as it is, or may be corrected to be the same as during an actual operation after modeling.

In the present specification, "virtual surgical data" means data including rehearsal or simulation actions performed on a virtual body model. "Virtual surgical data" may be image data in which rehearsal or simulation has been performed on a virtual body model in a virtual space, or data recorded on a surgical operation performed on a virtual body model.

In the present specification, "actual surgical data" refers to data acquired as an actual medical staff performs surgery. The "actual surgical data" may be image data photographing a surgical site during an actual surgical procedure, or data recorded on a surgical operation performed during an actual surgical procedure.

In the present specification, "detailed surgical operation" means the smallest unit of a surgical operation divided according to a specific criterion.

In the present specification, "Cue sheet data" refers to data that is sequentially recorded by dividing a specific surgical process into detailed surgical operations.

In the present specification, "trial cue sheet data" refers to cue sheet data obtained based on virtual surgical data performed by a user.

In the present specification, "virtual surgery cue sheet data for training" is included in the execution cue sheet data, and refers to cue sheet data generated based on virtual surgery data obtained by a user performing surgery simulation.

In the present specification, "reference virtual surgery cue sheet data" means cue sheet data for virtual surgery performed by a specific medical person to establish big data for learning or to guide a surgical process.

In the present specification, "optimized cue sheet data" refers to cue sheet data for an optimized surgical process in terms of operation time or surgical prognosis.

In the present specification, "learning cue sheet data" refers to cue sheet data used for learning for calculating optimized cue sheet data.

In the present specification, "surgical guide data" refers to data used as guide information during an actual surgery.

In the present specification, "computer" includes all various devices capable of performing arithmetic processing and providing results to a user. For example, computers are not only desktop PCs and notebooks, but also smart phones, tablet PCs, cellular phones, PCS phones, and synchronous/asynchronous systems. A mobile terminal of the International Mobile Telecommunication-2000 (IMT-2000), a Palm Personal Computer (PC), a personal digital assistant (PDA), and the like may also be applicable. In addition, when a head mounted display (HMD) device includes a computing function, the HMD device may be a computer. Also, the computer may correspond to a server that receives a request from a client and performs information processing.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a diagram showing a robotic surgery system according to the disclosed embodiment.

Referring to FIG. 1, a schematic diagram of a system capable of performing robot surgery according to the disclosed embodiment is shown.

According to FIG. 1, the robotic surgery system includes a medical imaging equipment 10, a server 20, and a control unit 30 provided in an operating room, an image capture unit 36, a display 32, and a surgical robot 34. do. Depending on the embodiment, the medical imaging equipment 10 may be omitted from the robotic surgery system according to the disclosed embodiment.

In one embodiment, robotic surgery is performed by a user controlling the surgical robot 34 using the control unit 30. In one embodiment, robotic surgery may be automatically performed by the controller 30 without user control.

The server 20 is a computing device including at least one processor and a communication unit.

The control unit 30 includes a computing device including at least one processor and a communication unit. In one embodiment, the controller 30 includes a hardware and software interface for controlling the surgical robot 34.

The image capturing unit 36 includes at least one image sensor. That is, the image capturing unit 36 includes at least one camera device and is used to photograph the surgical site. In one embodiment, the imaging unit 36 is used in combination with the surgical robot 34. For example, the imaging unit 36 may include at least one camera coupled to the surgical arm of the surgical robot 34.

In one embodiment, an image captured by the image capturing unit 36 is displayed on the display 340.

The controller 30 receives information necessary for surgery from the server 20 or generates information necessary for surgery and provides the information to the user. For example, the control unit 30 displays generated or received information necessary for surgery on the display 32.

For example, a user performs robotic surgery by controlling the movement of the surgical robot 34 by manipulating the control unit 30 while looking at the display 32.

The server 20 generates information necessary for robotic surgery using medical image data of an object (patient) previously photographed from the medical imaging equipment 10 and provides the generated information to the controller 30.

The controller 30 provides the information received from the server 20 to the user by displaying it on the display 32 or controls the surgical robot 34 using the information received from the server 20.

In one embodiment, the means that can be used in the medical imaging equipment 10 is not limited, and for example, various medical image acquisition means such as CT, X-ray, PET, MRI, etc. may be used.

Hereinafter, a method of providing feedback on an operation result will be described in detail with reference to the drawings.

2 is a flowchart illustrating a method of providing feedback on an operation result according to an exemplary embodiment.

Each of the steps shown in FIG. 2 is performed in time series by the server 20 or the control unit 30 shown in FIG. 1. In the following, for convenience of explanation, each step is described as being performed by a computer, but the execution subject of each step is not limited to a specific device, and all or part of it is to be performed by the server 20 or the control unit 30. I can.

Referring to FIG. 2, a method of providing feedback on a surgical result according to an embodiment of the present invention includes actual surgical cue sheet data consisting of a plurality of detailed surgical operations by dividing a computer based on actual surgical data acquired during an actual surgical procedure. Obtaining (S200); Obtaining reference cue sheet data for the actual surgery (S400); And comparing the actual surgical cue sheet data with the reference cue sheet data to provide feedback (S600). Hereinafter, a detailed description of each step will be described.

The computer is divided based on the actual surgical data acquired in the actual surgical procedure to obtain actual surgical cue sheet data composed of a plurality of detailed surgical operations (S200). The computer generates actual surgical cue sheet data based on a surgical image captured during a surgical procedure by a surgical robot or data obtained during a control process of the surgical robot.

The detailed surgical operation constituting the cue sheet data is the smallest unit of operation constituting the surgical process. Detailed surgical operations can be divided by several criteria. For example, the detailed operation may include the type of surgery (for example, laparoscopic surgery, robotic surgery), the anatomical body part where the surgery is performed, the surgical tools used, the number of surgical tools, the direction in which the surgical tools appear on the screen, or It can be divided based on the position, movement of the surgical tool (eg, forward/retract).

The segmentation criterion and the subcategory included in the segmentation criterion may be directly set through learning actual surgical data by medical staff. The computer can perform supervised learning according to the division criteria and subcategories set by the medical staff to divide the actual surgical data into sub-surgical operations, which are the smallest units.

In addition, the segmentation criterion and the subcategory included in the segmentation criterion may be extracted by learning a surgical image by a computer. For example, the computer may perform deep learning learning (ie, unsupervised learning) on actual surgical data accumulated as big data to calculate a segmentation criterion and a category within each segmentation criterion. After that, the computer divides the actual surgical data according to the division criteria generated through learning the actual surgical data to generate cue sheet data.

In addition, as another embodiment, the actual surgical data may be segmented by identifying whether it corresponds to each segmentation criterion through image recognition. That is, the computer recognizes the position of the anatomical organ corresponding to the segmentation criterion, the number of surgical instruments that appear, and the number of each surgical instrument on the screen in the image of the actual surgical data, and can perform the segmentation in units of detailed surgical operations.

In addition, as another embodiment, the computer may perform a segmentation process for generating cue sheet data based on the surgical tool motion data included in the actual surgical data. When the user performs robotic surgery, the actual surgical data may include various information input in the process of controlling the surgical robot, such as the type and number of surgical tools selected by the user, and information on the movement of each surgical tool. . Accordingly, the computer can perform the division based on the information included at each time point of the actual surgical data.

In addition, in an embodiment, the actual surgical data includes various types of detailed surgical operations such as resection and suture, and segmentation is performed according to the segmentation criteria. Specifically, the process of dividing the actual surgical data (for example, actual surgical image) that has actually performed gastric cancer surgery into detailed surgical operations to generate cue sheet data will be described as follows.

For example, gastric cancer surgery includes, first, a detailed surgical operation of resecting a part or all of the stomach including a tumor, and a detailed surgical operation of resecting a lymph node. In addition, various resection and connection techniques are used depending on the condition of gastric cancer. In addition, each detailed surgical operation may be divided into a plurality of more detailed detailed surgical operations according to a specific position at which the detailed surgical operation is performed and a moving direction of the surgical tool.

For example, the detailed operation of gastric cancer surgery may first be divided into an opening step, an ablation step, a connection step, and a suture step.

In addition, the method of connecting the resected organs includes an in vitro anastomosis method in which an incision and connection of 4-5 cm or more at the end of the myeongchi, or an in vivo anastomosis method in which an incision and anastomosis are performed in the abdominal cavity by incising about 3 cm in the navel, etc. are included. May be divided in more detail according to such a specific connection method.

Furthermore, each surgical method may be divided into a plurality of detailed surgical operations according to the position and movement of the surgical tool.

Each of the divided detailed surgical operations may be given a standardized name based on a location where the detailed surgical operation is performed and a pathway of the surgical tool.

In one embodiment, various terms used for standardized names may be defined. For example, when dealing with a specific area on the lower right of the stomach, the name of the area may be a name commonly used in the medical field, and a more comprehensive or subdivided name defined in the system according to the disclosed embodiment may be used. Can be used.

Accordingly, the surgical image in which the user actually performed the operation may be organized into information in the form of a cue sheet sequentially arranged based on a standardized name of a plurality of detailed surgical operations.

In addition, as an embodiment, the cue sheet data may be generated as code data of a specific number of digits according to a criterion for dividing into detailed surgical operations. In other words, the computer divides the actual surgical data into standardized detailed surgical actions by applying the standardized division criteria and designating detailed categories within the division criteria, and distinguishing each detailed surgical action by assigning a standardized code value to each detailed category. It gives standardized code data that can be done.

The computer assigns digitized code data to each detailed surgical operation by sequentially assigning numbers or letters from the upper category to which a specific detailed surgical operation belongs in the order of applying the division criteria. Through this, the computer may generate the cue sheet data in a form in which standardized code data of each detailed surgical operation is listed instead of a divided detailed surgical operation image. In addition, the user can share or transmit the actual surgical procedure by providing only cue sheet data composed of standardized code data.

In one embodiment, when the computer is a client terminal disposed in the operating room or corresponds to the control unit 30, the computer acquires standardized code data from the surgical image and transmits the acquired code data to the server 20 to perform actual surgery. Share or communicate the process.

In one embodiment, when the computer corresponds to the server 20, the surgical image is transmitted to the server 20, and the server 20 may generate cue sheet data and code data.

In addition, in an embodiment, the computer may give a standardized name to a standardized code of each detailed surgical operation. Through this, the user can select and check only the desired detailed surgical operation within the entire cue sheet. In addition, in this case, the user can easily grasp the progress of the operation by simply viewing the cue sheet sequentially arranged based on the standardized name of each detailed surgical operation, even if the user does not view all the surgical images.

The cue sheet data may be converted into a surgical image using an image database for each detailed surgical operation. In the image database, images corresponding to each code data may be stored, and a plurality of images corresponding to each code data may be stored according to circumstances. For example, specific code data may be loaded with different detailed surgical operation images in the image database according to the previously performed operation.

In addition, as each cue sheet data is matched and stored with a specific virtual body model, the computer may play back a surgical simulation image by sequentially applying each detailed surgical operation included in the cue sheet data to the virtual body model.

Accordingly, the image corresponding to the cue sheet data may be reproduced at the same point of view as the surgical image, or may be reconstructed and reproduced at a different viewpoint. Alternatively, the image is modeled in 3D, and the viewpoint and position may be adjusted according to the user's manipulation.

The computer acquires reference cue sheet data for the actual surgery (S400).

In one embodiment, the reference cue sheet data means optimized cue sheet data generated by a computer.

In another embodiment, the reference cue sheet data means virtual surgery cue sheet data for reference.

3 is a flowchart illustrating a method of calculating optimized cue sheet data according to an exemplary embodiment.

The computer acquires one or more learning cue sheet data (S420). The learning cue sheet data is learning subject data that is learned to calculate the optimized cue sheet data. The learning cue sheet data includes cue sheet data (ie, actual surgery cue sheet data) generated based on actual surgical data or cue sheet data (ie, virtual surgery cue sheet data for reference) generated based on simulated virtual surgery data for reference. Can include. The actual surgical cue sheet data is generated by a computer dividing the actual surgical data according to a division criterion. The reference virtual surgery cue sheet data is not acquired during the surgical training process of the user, but is generated by performing a simulation for the purpose of constructing the learning target data or providing it for reference to trainees.

After that, the computer performs reinforcement learning using the learning cue sheet data (S440). Reinforcement learning is an area of machine learning, in which an agent defined in an environment recognizes the current state and selects an action or sequence of actions that maximizes reward among the selectable actions. Reinforcement learning can be summarized as learning how to maximize rewards based on state transitions and rewards according to state transitions.

Thereafter, the computer calculates the optimization cue sheet data using the reinforcement learning result (S460). The optimized cue sheet data is calculated based on the results of reinforcement learning based on the condition of the shortest operation time, the minimum amount of blood loss, the required operation group, and the required order of performance to reduce the anesthesia time of the patient.

The essential operation group is a group of detailed surgical operations that must be performed together in order to perform a specific detailed surgical operation. The required procedure is a sequence of surgical operations that must be sequentially performed in the process of performing a specific operation. For example, according to the type of surgery or the type of surgery, surgical actions that should appear sequentially and their order may be determined.

In addition, the computer calculates the optimized cue sheet data for each situation according to the physical condition of the patient, the surgical site (eg, tumor tissue) condition (eg, the size and location of the tumor) through reinforcement learning. To this end, the computer utilizes patient conditions, surgical site conditions, etc. together with the learning cue sheet data during learning.

In one embodiment, the computer may perform a virtual simulation operation by itself. For example, the computer may generate a surgical process according to an operation type and a patient type based on the disclosed surgical process optimization method, and perform a virtual surgical simulation based on the generated surgical process.

The computer evaluates the virtual surgical simulation result and performs reinforcement learning based on the virtual surgical simulation information and the evaluation information for the result, thereby obtaining an optimized surgical process.

Even with a model that is trained to generate an optimal surgical process, in the case of actual surgery, since the body structure and the type of surgery are different for each patient, it may be difficult to create an optimized surgical process according to each patient and type of surgery.

Therefore, the computer generates a surgical process based on the patient's body structure and type of surgery using the learned model, performs reinforcement learning by performing a virtual surgical simulation, and performs a surgical process optimized for each patient and type of surgery. Can be generated.

The computer provides feedback by comparing the actual surgical cue sheet data with the reference cue sheet data (S600).

In one embodiment, the operation of comparing the actual surgical cue sheet data with the reference cue sheet data may be performed by a computer or controller 30 disposed in an operating room, or may be performed by the server 20.

When the comparison is performed in the server 20, the server 20 acquires the reference cue sheet data, and performs a comparison with the surgical image or cue sheet data (code data) acquired from the controller 30.

When the comparison is performed in the computer or the control unit 30 disposed in the operating room, the computer acquires the cue sheet data from the surgical image and performs the comparison with the reference cue sheet data received from the server 20.

In one embodiment, feedback may be provided through a website or application. For example, the feedback may be provided through an application installed on the doctor's mobile terminal, and when the operation is finished, a notification related to the feedback may be provided to the doctor's mobile terminal.

4 is a flowchart illustrating a method of obtaining feedback according to an exemplary embodiment.

In one embodiment, the computer may compare the types and order of detailed surgical operations included in the actual surgical cue sheet data with the types and sequences of detailed surgical operations included in the reference cue sheet data, and provide feedback on the surgical results ( S620).

For example, the computer determines whether there are missing detailed surgical actions, unnecessary detailed surgical actions, or incorrect detailed surgical actions in the detailed surgical actions included in the actual surgical cue sheet data compared to the detailed surgical actions included in the reference cue sheet data. Can be determined and provide feedback on the result.

For example, a necessary detailed surgical operation included in the reference cue sheet data may be omitted from the actual surgical cue sheet data.

In addition, unnecessary detailed surgical operations that are not included in the reference cue sheet data may be included in the actual surgical cue sheet data.

In addition, although the detailed surgical operation included in the reference cue sheet data is also included in the actual surgical cue sheet data, the specific operation may be different or incorrect.

In this case, the types of detailed surgical operations of the reference cue sheet data and the actual surgical cue sheet data are identical, but the operation may be different if specifically compared.

Therefore, the computer determines whether each detailed surgical operation has been correctly performed. A specific detailed surgical operation should be included in the entire surgical procedure, but it may not be performed normally. For example, in the case of an operation (i.e., detailed surgical operation) of holding a tissue in a specific location with a surgical tool in the form of an intercalation, it is normal to grasp the tissue deeper than the optimal depth or to a small area of the confinement to the extent that it is easy to grab and miss the tissue. It can be judged as a detailed surgical operation that is not. The computer recognizes the tissues and surgical tools in the actual surgical image through image recognition, accurately recognizes the detailed surgical operation, and performs evaluation and feedback by comparing it with the normal detailed surgical operation in the optimized cue sheet data.

In addition, the computer may analyze the actual surgical cue sheet data itself, and provide information on a record of an abnormal situation or an unexpected situation. For example, when bleeding occurs in a specific area, the computer may provide at least one of the fact that the bleeding has occurred, the location of the bleeding, and the amount of bleeding.

Further, the computer may analyze the record of detailed surgical operations included in the actual surgical cue sheet data, and determine and provide the cause of the bleeding.

Specifically, the computer analyzes the completed actual surgery cue sheet to search for a surgical error (S640). The computer may provide feedback when a surgical error situation is detected according to a preset rule.

For example, a general operation error situation and a method of providing feedback accordingly are as described below.

For example, the computer may detect foreign objects left in the patient's body and provide feedback. According to the disclosed embodiment, the computer recognizes not only the location of the surgical tool, the bleeding area, but also all objects included in the surgical image based on the acquired surgical image, and analyzes each object. The computer determines the location, number, and inflow time of objects included in the surgical image. Accordingly, when the computer determines that the introduced foreign material has not been removed from the surgical site when the operation is completed, the computer may generate a warning and provide a feedback requesting confirmation to the user. In an embodiment, the computer may request confirmation from the user even when the object introduced into the surgical site is not identified in the image. For example, if it is not confirmed that an object introduced into the surgical site is removed from the surgical site, it is possible to remain in an invisible place even if it is not included in the surgical image, so that feedback can be provided to the user.

For example, it is possible to detect a situation in a wrong patient surgery and provide feedback. According to the disclosed embodiment, the computer analyzes the surgical image in real time, and performs registration between the organs of the 3D modeled image and the actual organs. The computer tracks the location of the camera and surgical instruments in real time, determines the surgical situation, and acquires information that allows the simulator to follow the actual surgical procedure. In one embodiment, if the actual organ and the 3D modeled image are not matched in the process of matching the actual organ of the patient with the organs of the 3D modeling image, there is a possibility that the operation for the wrong patient has been performed, so the computer asks the user to confirm. Can be requested.

For example, it is possible to detect a situation on a wrong side surgery and provide feedback. In the disclosed embodiment, the computer determines the operation situation and provides a rehearsal result or a surgical guide image according to an optimal surgical method. In the process, if the actual procedure of the surgery is different from the rehearsal or optimal surgical method, the computer may request confirmation from the user because the surgery may have been performed on the wrong area or another type of surgery may have been performed.

For example, it is possible to recognize a situation of nerve damage and provide feedback. In the disclosed embodiment, the computer may provide feedback when the actual surgery is different from the rehearsal, as described above, and cut an important nerve or ganglion according to the positional relationship between the patient's organ and the surgical tool, or Warnings can be provided when approaching and risk is predicted. In addition, the computer displays the invisible blood vessels, nerves, ganglions, etc., superimposed on the surgical image using image matching and further AR or MR (Mixed Reality) technology, so that users can see important parts well after surgery. Thus, it can help to restore the surgical process.

In addition, as another embodiment, the step of adding the actual surgical cue sheet data to the learning cue sheet data used to calculate the optimized cue sheet data, and performing reinforcement learning (S660).

In an embodiment, there may be a better part of the actual surgical cue sheet data compared to the optimized cue sheet data. Accordingly, by adding actual surgical cue sheet data to the learning cue sheet data and performing reinforcement learning, a model capable of generating better optimized cue sheet data can be obtained.

In addition, in another embodiment, the computer tracks the prognosis of the patient corresponding to the respective surgical records (S680). The computer performs machine learning using each surgical record and the prognosis corresponding to each surgical record as learning data, so that it is possible to determine whether each prognosis is caused by the combination of which surgical operations.

For example, the computer can derive a detailed surgical operation or a combination of detailed surgical operations that can cause side effects even if a minor surgical operation is performed by analyzing the surgical operations of patients with specific side effects.

In an embodiment, the computer may obtain information on detailed surgical operations that bring out each prognosis through reinforcement learning. For example, the computer may perform reinforcement learning based on learning data about the prognosis that occurs when each of the operations is included or is performed in a specific order. Based on the reinforcement learning result, the computer may determine what prognosis (ie, some side effects) may occur due to a specific detailed surgical operation, a continuous detailed surgical operation, or a combination of detailed surgical operations.

Also, the computer may output and provide feedback to the user in various ways. In an embodiment, the computer may provide feedback by extracting detailed surgical operations in which a problem exists. For example, the computer can help the user to identify the problem by extracting and reproducing only images of detailed surgical operations in which a problem exists.

In addition, the computer may search for and provide detailed surgical operations included in the surgical procedure. For example, since an operation generally lasts for several hours or more, it is difficult for a user to receive feedback while checking the entire image after the operation. Accordingly, the computer provides the cue sheet data, and when the user selects one or more detailed surgical actions included in the cue sheet data, it is possible to provide feedback by extracting only the selected detailed surgical action. For example, the computer may extract and reproduce only images of selected detailed surgical operations.

According to the disclosed embodiment, each detailed surgical operation has a standardized name and standardized code data. Accordingly, the user can search for each detailed surgical operation based on the standardized name or code data, and can easily determine the operation progress by only looking at the cue sheet data.

The above-described method for providing feedback on an operation result according to an embodiment of the present invention may be implemented as a program (or application) and stored in a medium to be executed by being combined with a computer as hardware.

The above-described program is C, C++, JAVA, machine language, etc. that can be read by the computer's processor (CPU) through the device interface of the computer in order for the computer to read the program and execute the methods implemented as a program. It may include a code (Code) coded in the computer language of. Such code may include a functional code related to a function defining necessary functions for executing the methods, and a control code related to an execution procedure necessary for the processor of the computer to execute the functions according to a predetermined procedure. can do. In addition, such code may further include additional information required for the processor of the computer to execute the functions or code related to a memory reference to which location (address address) of the internal or external memory of the computer should be referenced. have. In addition, when the processor of the computer needs to communicate with any other computer or server in the remote in order to execute the functions, the code uses the communication module of the computer to determine how It may further include a communication-related code for whether to communicate, what kind of information or media should be transmitted and received during communication.

The stored medium is not a medium that stores data for a short moment, such as a register, cache, memory, etc., but a medium that stores data semi-permanently and can be read by a device. Specifically, examples of the storage medium include, but are not limited to, ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, and the like. That is, the program may be stored in various recording media on various servers to which the computer can access, or on various recording media on the user's computer. In addition, the medium may be distributed over a computer system connected through a network, and computer-readable codes may be stored in a distributed manner.

The steps of a method or algorithm described in connection with an embodiment of the present invention may be implemented directly in hardware, implemented as a software module executed by hardware, or a combination thereof. Software modules include Random Access Memory (RAM), Read Only Memory (ROM), Erasable Programmable ROM (EPROM), Electrically Erasable Programmable ROM (EEPROM), Flash Memory, hard disk, removable disk, CD-ROM, or It may reside on any type of computer-readable recording medium well known in the art to which the present invention pertains.

In the above, embodiments of the present invention have been described with reference to the accompanying drawings, but those of ordinary skill in the art to which the present invention pertains can be implemented in other specific forms without changing the technical spirit or essential features. You can understand. Therefore, the embodiments described above are illustrative in all respects, and should be understood as non-limiting.

10: Medical imaging equipment
20: server
30: control unit
32: display
34: surgical robot
36: imaging unit

Claims (14)

The computer is divided based on the actual surgical data obtained according to the actual operation of the medical staff to obtain actual surgical cue sheet data composed of a plurality of detailed surgical operations, but the detailed surgical operation is the smallest operation unit that constitutes the surgical process. Acquiring operation cue sheet data;
A computer obtaining reference cue sheet data for the actual surgery, wherein the reference cue sheet data is optimized cue sheet data or reference cue sheet data for the actual surgery, a reference cue sheet data acquisition step; And
Comparing the actual surgical cue sheet data with the reference cue sheet data and providing feedback; Including,
The reference cue sheet data is cue sheet data for virtual or actual surgery performed to establish learning big data or guide a surgical process,
The optimized cue sheet data is cue sheet data calculated as an optimized surgical process by learning one or more cue sheet data, and is calculated based on at least one of a shortest operation time, a minimum amount of blood loss, an essential operation group, and an essential performance order,
The essential action group is a group of detailed surgical actions that must be performed together in order to perform a specific detailed surgical action,
The required order of performance is a sequence of operations that must be sequentially performed in the process of performing a specific operation, a method of providing feedback on a surgical result. The method of claim 1,
The plurality of detailed surgical operations,
The actual surgical data is divided based on at least one of the surgical site included in the actual surgical data, the type of the surgical tool, the number of the surgical tools, the location of the surgical tool, the direction of the surgical tool, and the movement of the surgical tool. One, a method of providing feedback on surgical results. The method of claim 1,
The detailed surgical operation,
At least one of a standardized name and standardized code data is given, respectively, a method of providing feedback on the surgical result. The method of claim 3,
The standardized code data,
A method of providing feedback on surgical results, which is code data digitalized by sequentially assigning numbers or letters from the upper category to which a specific detailed surgical operation belongs according to the application order of the division criteria. The method of claim 1,
The step of providing the feedback,
Comparing the actual surgical cue sheet data with a plurality of detailed surgical operations included in each of the reference cue sheet data, and determining whether at least one surgical error condition is included in the actual surgical cue sheet data; Including,
The operation error situation, including unnecessary detailed surgical operation, missing detailed surgical operation, or incorrect detailed surgical operation, a method of providing feedback on a surgical result. The method of claim 5,
The step of determining whether there is an incorrect detailed surgical operation,
Comparing the movement of the surgical tool corresponding to the detailed surgical operation included in the reference cue sheet data with the movement of the surgical tool corresponding to the detailed surgical operation included in the actual surgical cue sheet data; And
Determining whether a detailed surgical operation included in the actual surgical cue sheet data is wrong based on a result of comparing the movement of the surgical tool; Containing, a method of providing feedback on the surgical results. The method of claim 1,
The step of providing the feedback,
Obtaining, by the computer, the occurrence of an event situation from the actual surgical data; And
The step of determining, by a computer, a detailed surgical operation before the event situation included in the actual surgery cue sheet data to determine the cause of the event situation; further comprising,
The event situation includes at least one of bleeding information, foreign body information, and nerve damage information,
The bleeding information includes at least one of bleeding occurrence, bleeding occurrence location, and bleeding amount,
The foreign matter information includes at least one of a location, number, and inflow time of objects inserted into the body or formed inside the body. The method of claim 1,
The step of obtaining the reference cue sheet data,
A method of providing feedback on a surgical result, characterized in that obtaining optimized cue sheet data for each situation classified according to at least one of a patient's physical condition and a surgical site condition. The method of claim 1,
Adding the actual surgery cue sheet data to the learning cue sheet data; And
Performing reinforcement learning on a model for obtaining optimized cue sheet data by using the learning cue sheet data; A method of providing feedback on the surgical result further comprising a. The method of claim 1,
Acquiring information on prognosis corresponding to each of one or more actual surgery cue sheet data including the actual surgery cue sheet data;
Performing reinforcement learning based on the at least one actual surgery cue sheet data and information on prognosis; And
Determining a correlation between a prognosis and at least one detailed operation included in the one or more actual surgery cue sheet data based on the reinforcement learning result; A method of providing feedback on the surgical result further comprising a. The method of claim 5,
Extracting and reproducing an image of one or more detailed surgical actions corresponding to a surgical error situation or an image of one or more detailed surgical actions including an event situation; further comprising, a method for providing feedback on a surgical result. delete A computer program stored in a recording medium readable by a computer to perform the method of any one of claims 1 to 11 by being combined with a computer as hardware. It is divided based on the actual surgical data acquired according to the actual operation of the medical staff to obtain actual surgical cue sheet data consisting of a plurality of detailed surgical operations,
Acquire reference cue sheet data for the actual surgery,
Provide feedback by comparing the actual surgery cue sheet data and the reference cue sheet data,
The detailed surgical operation is the smallest operation unit constituting the surgical process,
The reference cue sheet data is cue sheet data optimized for the actual surgery or virtual surgery cue sheet data for reference,
The reference virtual surgery cue sheet data is cue sheet data for virtual surgery performed by a specific medical person to establish big data for learning or to guide a surgical process,
The optimized cue sheet data is cue sheet data calculated as an optimized surgical process by learning one or more cue sheet data, and is calculated based on at least one of a shortest operation time, a minimum amount of blood loss, an essential operation group, and an essential performance order,
The essential action group is a group of detailed surgical actions that must be performed together in order to perform a specific detailed surgical action,
The essential performance order is a sequence of operations that must be sequentially performed in the process of performing a specific operation, a device for providing feedback on a surgical result.

Images