KR20210104190A - Method for analysis and recognition of medical image - Google Patents

Abstract

The present invention relates to a system, a method, and a program for analyzing and recognizing a surgical video to quickly acquire a correct recognition result. According to one embodiment of the present invention, a method for analyzing surgical by a computer comprises: a step of acquiring, by the computer, a surgical video; a step of inputting, by the computer, the surgical video to one or more surgical element recognition models; a surgical recognition result combination acquisition step of acquiring, by the computer, a surgical recognition result combination calculated from each surgical element recognition model, wherein the one or more surgical element recognition models are included in a surgical element recognition layer which is the lowest level in a surgical analysis system; and a surgical analysis result acquisition step of acquiring, by the computer, inputting the surgical recognition result combination into one or more surgical analysis models to acquire one or more analysis results, wherein the one or more surgical analysis models are included in the surgical analysis layer above the surgical element recognition layer and are selected according to a user's request.

Description

Surgical analysis device, surgical image analysis and recognition system, method and program

The present invention relates to a method for analyzing and recognizing a medical image.

Medical surgery or medical procedure (ie, medical treatment act) can be classified into open surgery, minimally invasive surgery (MIS) including laparoscopic surgery and robotic surgery, radiation therapy, and endoscopic surgery. . Laparotomy refers to an operation in which the medical staff directly sees and touches the part to be treated. Minimally invasive surgery, also called keyhole surgery, is laparoscopic surgery and robotic surgery. In laparoscopic surgery, a small hole is made in the necessary area without open surgery, a laparoscope with a special camera and surgical tools are inserted into the body to observe through a video monitor, and a laser or special instrument is used to perform micro-surgery. In addition, robotic surgery is to perform minimally invasive surgery using a surgical robot. Furthermore, radiation therapy refers to surgical treatment outside the body with radiation or laser light. In addition, the endoscopic procedure refers to a procedure performed by inserting an endoscope into a digestive tract, etc. and then inserting a tool into a passage provided in the endoscope.

In the case of such medical treatment, in many cases, an image is acquired and performed based on the image during actual execution. That is, laparotomy is performed by medical staff directly looking at the patient's organs with the naked eye, and laparoscopic surgery or endoscopic surgery is performed while looking at an image obtained through a laparoscope or an endoscope. Therefore, it is important to provide a variety of information to medical staff through images obtained during treatment.

In addition, the development of technologies capable of providing information to assist a doctor in medical surgery or a medical procedure is required. In order to provide information for assisting surgery or a procedure, it is important to recognize the operation or various surgical information and to understand the meaning of the recognized information. Therefore, it is required to develop a technology that allows a computer to recognize motions or various surgical information from an image.

In addition, recently, deep learning has been widely used in the analysis of medical images. Deep learning is defined as a set of machine learning algorithms that attempt high-level abstractions (summarizing key contents or functions in large amounts of data or complex data) through a combination of several nonlinear transformation methods. Deep learning can be viewed as a field of machine learning that teaches computers to think in a broad framework.

In order to analyze or recognize the surgical image, it is necessary to recognize various surgical elements included in the surgical image. However, it is difficult to acquire each surgical element included in the surgical image with only one learning model.

Accordingly, the present invention provides a surgical analysis device, a surgical image analysis and recognition system, and a method that can quickly obtain accurate recognition results by recognizing a plurality of surgical elements within a surgical image, respectively, through a surgical element recognition model formed in parallel. and programs.

In addition, the present invention can obtain all or selectively various analysis results required for surgical analysis through a plurality of surgical analysis models that can calculate each result using the various surgical element recognition results calculated in the surgical element recognition model. It is intended to provide a surgical analysis device, a surgical image analysis and recognition system, a method, and a program.

In addition, the present invention provides a surgical analysis device, a surgical image analysis and recognition system, and a method, including a plurality of surgical solution models that conveniently calculate service results required for medical staff based on various surgical analysis results through a plurality of surgical analysis models. and programs.

The problems to be solved by the present invention are not limited to the problems mentioned above, and other problems not mentioned will be clearly understood by those skilled in the art from the following description.

A method for analyzing surgery by a computer according to an embodiment of the present invention includes: acquiring, by a computer, a surgical image; inputting the surgical image into one or more surgical element recognition models by a computer; A computer acquires a combination of surgical recognition results calculated from each surgical element recognition model, wherein one or more surgical element recognition models are included in the surgical element recognition layer, which is the lowest level in the surgical analysis system, a surgical recognition result combination acquisition step; and the computer inputs the combination of the surgical recognition result into one or more surgical analysis models to obtain one or more analysis results, wherein the one or more surgical analysis models are included in the surgical analysis layer above the surgical element recognition layer, and the user's request It is selected according to the, surgical analysis result acquisition step; and includes.

In addition, in another embodiment, each surgery analysis model in the surgery analysis layer, based on the data required for analysis, it is characterized in that the connection relationship with one or more surgical element recognition model in the surgical element recognition layer is set.

In addition, in another embodiment, the surgical element recognition model, the organ recognition model for recognizing the organs in the surgical image; a surgical tool recognition model for recognizing a surgical tool in the surgical image and a movement of the surgical tool; and an event recognition model that recognizes an event occurring in the surgical image, wherein the event is an abnormal situation during surgery including bleeding.

In addition, in another embodiment, the surgery analysis layer, the surgical organ type recognized by the surgical element recognition layer, the surgical operation, and the blood for calculating the degree of blood loss based on a combination of surgical recognition results including an event occurring during surgery loss recognition model; and an organ damage detection model for calculating the degree of organ damage based on a combination of a surgical recognition result including a surgical stage and a surgical time recognized by the surgical element recognition layer; including, the blood loss recognition model and the organ damage detection model is used to calculate analysis results during or after surgery for each surgical procedure.

In addition, in another embodiment, when analyzing the surgical result after surgery is completed, the surgical analysis layer includes a surgical tool recognized by the surgical element recognition layer, an organ operated with the surgical tool, and the surgical tool. It further includes; a surgical tool misuse detection model that detects the use of an incorrect surgical tool based on the detailed surgical step of the entire operation being performed, and an event that occurred in the detailed surgical step.

In addition, in another embodiment, the surgery analysis layer, the optimal surgical plan calculation model for calculating the optimal surgical plan based on the combination of the surgical recognition results obtained from the surgical element recognition layer; further comprising, the optimal surgical plan The calculation model is characterized in that it calculates and provides an optimal surgical plan to be performed afterward by analyzing the previously acquired surgical image data in real time.

In addition, in another embodiment, the surgery analysis layer, the optimal surgical plan calculation model for calculating the optimal surgical plan based on the combination of the surgical recognition results obtained from the surgical element recognition layer; further comprising, the optimal surgical plan The calculation model is characterized in that it calculates and provides a surgical plan for an actual patient based on a combination of surgical recognition results for virtual surgery performed on the virtual body model of the patient before surgery.

In addition, in another embodiment, the computer inputs one or more analysis results to a specific surgical solution model, and providing a surgical product organized based on the analysis results; further includes.

In addition, in another embodiment, the surgery solution model may include: a surgery evaluation model for calculating an evaluation result for surgery based on one or more analysis results obtained in the surgery analysis layer; a chart generation model for generating a chart for surgery based on one or more analysis results obtained in the surgery analysis layer; and a surgery complexity calculation model for calculating the complexity or difficulty of surgery based on one or more analysis results obtained in the surgery analysis layer.

In addition, in another embodiment, a surgical Q&A model that calculates an answer to a question about surgery by learning one or more analysis results obtained from the surgery analysis layer; includes, wherein the operation product providing step includes a medical staff By entering a question about a specific operation in

The program for analyzing surgery by a computer according to another embodiment of the present invention may be combined with a computer that is hardware and stored in a medium to execute the method for analyzing surgery by the computer.

The surgical image analysis apparatus according to another embodiment of the present invention includes one or more surgical element recognition models for calculating a surgical recognition result as a surgical image is input, wherein the one or more surgical element recognition models are the lowest in the surgical analysis system. Which is included in the level of the surgical element recognition layer, the surgical element recognition layer; and one or more surgical analysis models for obtaining an analysis result based on a combination of surgical recognition result that is a combination of results provided by the one or more surgical element recognition models, wherein the surgical analysis layer is formed as an upper layer of the surgical element recognition layer includes ;

In addition, in another embodiment, the surgical solution providing layer, which includes one or more surgical solution models for providing a surgical product organized based on one or more analysis results obtained in the surgical analysis layer;

According to the present invention as described above, each surgical element can be accurately recognized by an individual surgical element recognition model that recognizes individual surgical elements (surgical tools, bleeding, cameras, etc.) in a surgical image. That is, it is possible to provide a recognition result with high accuracy compared to a method of recognizing various surgical elements included in a surgical image through a single recognition model.

In addition, according to the present invention, since surgical analysis is performed by combining the surgical element recognition results calculated by the surgical element recognition model, it is possible to provide accurate surgical analysis results to medical staff.

In addition, according to the present invention, since each surgical solution model automatically calculates the results required for the medical staff as one or more surgical analysis results are input, the medical staff's post-operative work can be simplified.

In addition, according to the present invention, when a new surgical element recognition is required, a new surgical analysis is required, or a new service type solution is required, a new surgical element recognition model, a surgical analysis model, or a surgical solution model in the surgical analysis system You can easily add new functions because you only need to add and establish a connection relationship with one or more models included in the upper or lower layers.

1 is a view showing a robotic surgery system according to an embodiment of the present invention.
2 is a flowchart illustrating a method of constructing surgical simulation information according to an embodiment of the present invention.
3 is a flowchart illustrating a process of reflecting texture information on an actual surgical site of an object to a virtual body model according to an embodiment of the present invention.
4 is a diagram illustrating an example of applying a method for constructing surgical simulation information according to an embodiment of the present invention.
5 is a diagram schematically showing the configuration of an apparatus 300 for constructing surgical simulation information according to an embodiment of the present invention.
6 is a schematic diagram of a system capable of performing robotic surgery according to an embodiment of the present invention.
7 is a flowchart schematically illustrating a method for generating a virtual body model using a surgical image according to an embodiment of the present invention.
8 shows an example for explaining a process of dividing a surgical image into at least one video clip (ie, a sequence) according to an embodiment of the present invention.
9 shows an example for explaining a process of detecting an organ from an image frame according to an embodiment of the present invention.
10 shows an example for explaining a process of matching at least one cluster region on a virtual body model according to an embodiment of the present invention.
11 to 13 are diagrams showing an example of applying a method for generating a virtual body model using a surgical image according to an embodiment of the present invention.
14 is a diagram schematically showing the configuration of an apparatus 600 for performing a method for generating a virtual body model using a surgical image according to an embodiment of the present invention.
15 is a schematic diagram of a system capable of performing robotic surgery according to an embodiment of the present invention.
16 is a flowchart illustrating a method of utilizing surgical information according to an embodiment of the present invention.
17 is a flowchart illustrating a method of utilizing surgical information according to another embodiment of the present invention.
18 is a diagram schematically showing the configuration of an apparatus 300 for performing a surgical information utilization method according to an embodiment of the present invention.
19 is a schematic diagram of a system capable of performing robotic surgery according to an embodiment of the present invention.
20 is a flowchart schematically illustrating a method of generating learning data based on a surgical image according to an embodiment of the present invention.
21 is a diagram illustrating a process of generating relational expression information by recognizing surgical recognition information from an image frame according to an embodiment of the present invention.
22 is a diagram illustrating relational expression information generated for a plurality of image frames in a surgical image according to an embodiment of the present invention.
23 is a diagram illustrating an example of a process of learning based on relational expression information generated for a plurality of image frames in a surgical image according to an embodiment of the present invention.
24 is a diagram schematically showing the configuration of an apparatus 400 for performing a method of generating learning data based on a surgical image according to an embodiment of the present invention.
25 is a flowchart illustrating a method of generating artificial data according to an embodiment of the present invention.
26 is a diagram for explaining a method of generating artificial data based on learning according to an embodiment of the present invention.
27 to 30 are diagrams for explaining embodiments of a method for generating artificial data based on learning according to an embodiment of the present invention.
31 is a schematic diagram of a system capable of performing robotic surgery according to an embodiment of the present invention.
32 is a diagram schematically showing the configuration of an apparatus 700 for performing a method for generating artificial data according to an embodiment of the present invention.
33 is a schematic diagram of a system capable of performing robotic surgery according to an embodiment of the present invention.
34 is a diagram schematically showing the configuration of a surgical information building system according to an embodiment of the present invention.
35 is a flowchart illustrating a method of constructing surgical information using a relay server according to an embodiment of the present invention.
36 is a diagram schematically showing the configuration of an apparatus 300 for constructing surgical information according to an embodiment of the present invention.
37 is a flowchart illustrating a method of providing a camera position based on a surgical image according to an embodiment of the present invention.
38 is a diagram illustrating an operation route for each patient for a specific operation.
39 is a view for explaining an example of a process of deriving camera position information on a standard body model from a surgical image according to an embodiment of the present invention.
40 is a diagram for explaining an example of a process of converting coordinate information of a virtual body model of a patient into coordinate information of a standard body model according to an embodiment of the present invention.
41 is a diagram for explaining a process of converting coordinate information of a standard body model into coordinate information of a virtual body model for a current surgery target according to an embodiment of the present invention.
42 and 43 are diagrams for explaining a process of deriving position information on a standard body model from a surgical image through learning according to an embodiment of the present invention.
44 is a schematic diagram of a system capable of performing robotic surgery according to an embodiment of the present invention.
45 is a diagram schematically showing the configuration of an apparatus 500 for performing a method for providing a camera position based on a surgical image according to an embodiment of the present invention.
46 is a schematic diagram of a system capable of performing robotic surgery according to an embodiment of the present invention.
47 is a flowchart schematically illustrating a method for generating a virtual body model according to an embodiment of the present invention.
48 is a flowchart illustrating a method of calculating position information of a surgical tool according to an embodiment of the present invention.
49 is a diagram schematically showing the configuration of an apparatus 300 for performing a method for calculating position information of a surgical tool according to an embodiment of the present invention.
50 is a schematic diagram of a system capable of performing robotic surgery according to an embodiment of the present invention.
51 is a flowchart illustrating a method of calculating a camera position using an actual surgical image according to an embodiment of the present invention.
52 is a flowchart illustrating a process of setting a reference position for a camera according to an embodiment of the present invention.
53 is a flowchart illustrating a process of calculating a position change amount of a camera according to an embodiment of the present invention.
54 is a diagram illustrating a position of a camera derived according to an embodiment of the present invention on a coordinate plane.
55 is a diagram schematically showing the configuration of an apparatus 300 for performing a method for calculating a camera position using an actual surgical image according to an embodiment of the present invention.
56 is a schematic diagram of a system capable of performing robotic surgery according to an embodiment of the present invention.
57 is a flowchart schematically illustrating a method for generating a virtual body model according to an embodiment of the present invention.
58 is a flowchart illustrating a method for providing location information of a surgical tool according to an embodiment of the present invention.
59 is a diagram schematically showing the configuration of an apparatus 300 for performing a method for providing position information of a surgical tool according to an embodiment of the present invention.
60 is a schematic diagram of a system capable of performing robotic surgery according to an embodiment of the present invention.
61 is a flowchart illustrating a learning-based surgical motion recognition method according to an embodiment of the present invention.
62 to 66 are diagrams for explaining a process of recognizing a surgical motion by acquiring a surgical image in the learning-based surgical motion recognition method according to an embodiment of the present invention.
67 is a flowchart illustrating a surgical image learning method according to an embodiment of the present invention.
68 is a diagram schematically showing the configuration of an apparatus 400 for performing a surgical image learning method and a learning-based surgical motion recognition method according to an embodiment of the present invention.
69 is a schematic diagram of a system capable of performing robotic surgery according to an embodiment of the present invention.
70 is a flowchart illustrating a learning-based surgical motion recognition method according to an embodiment of the present invention.
71 is a diagram illustrating a surgical image sequence.
72 is a view for explaining a method for recognizing a surgical operation through CNN-based learning according to an embodiment of the present invention.
73 is a diagram schematically showing the configuration of an apparatus 300 for performing a learning-based surgical motion recognition method according to an embodiment of the present invention.
74 is a flowchart illustrating a bleeding evaluation method using a surgical image according to an embodiment of the present invention.
75 is a view showing an example to which the bleeding evaluation method using a surgical image according to an embodiment of the present invention can be applied.
76 is a diagram illustrating an example of calculating a bleeding amount by segmenting a bleeding region in a surgical image according to an embodiment of the present invention.
77 is a diagram schematically showing the configuration of an apparatus 100 for performing a bleeding evaluation method using a surgical image according to an embodiment of the present invention.
78 is a schematic diagram of a system capable of performing robotic surgery according to an embodiment of the present invention.
79 and 80 are flowcharts illustrating a method of providing surgical information using a surgical image according to an embodiment of the present invention.
81 is an example showing a process of recognizing a surgical stage based on a surgical image according to an embodiment of the present invention.
82 is an example illustrating a process of specifying an organ region in a surgical image based on information on an organ in an organ candidate group according to an embodiment of the present invention.
83 is a diagram schematically showing the configuration of an apparatus 200 for performing a method for providing surgical information using a surgical image according to an embodiment of the present invention.
84 is a schematic diagram of a system capable of performing robotic surgery according to an embodiment of the present invention.
85 is a flowchart schematically illustrating a method of predicting an operation time based on an operation image according to an embodiment of the present invention.
86 is a view for explaining a process of generating learning data based on a surgical image according to an embodiment of the present invention.
87 is a diagram for explaining a process of performing learning based on learning data according to an embodiment of the present invention.
88 is a diagram schematically illustrating the configuration of an apparatus 200 for performing a method of predicting an operation time based on an operation image according to an embodiment of the present invention.
89 is a schematic diagram of a system capable of performing robotic surgery according to an embodiment of the present invention.
90 is a flowchart schematically illustrating a method for evaluating a degree of recognition of a surgical image according to an embodiment of the present invention.
91 is a view for explaining a process of calculating a representative recognition value for a surgical image according to an embodiment of the present invention.
92 is a view for explaining a process of calculating a representative recognition value for a surgical image according to the importance of the surgical recognition element according to an embodiment of the present invention.
93 is a diagram schematically showing the configuration of an apparatus 200 for performing a method for evaluating a recognition degree of a surgical image according to an embodiment of the present invention.
94 is a flowchart illustrating a surgical optimization method according to an embodiment of the present invention.
95 is a diagram illustrating a process of generating a gene according to an embodiment of the present invention.
96 is a diagram illustrating a process of crossing genes as an example of applying a genetic algorithm according to an embodiment of the present invention.
97 is a diagram illustrating a process of mutating a gene as an example of applying a genetic algorithm according to an embodiment of the present invention.
98 is a schematic diagram of a system capable of performing robotic surgery according to an embodiment of the present invention.
99 is a diagram schematically showing the configuration of an apparatus 400 for performing a surgical optimization method according to an embodiment of the present invention.
100 is a schematic diagram of a system capable of performing robotic surgery according to an embodiment of the present invention.
101 is a flowchart schematically illustrating a method for providing an optimized surgical tool according to an embodiment of the present invention.
102 is a flowchart schematically illustrating a method for providing an optimized surgical tool according to another embodiment of the present invention.
103 is a flowchart schematically illustrating a method for providing an optimal entry position of a surgical tool according to an embodiment of the present invention.
104 is a diagram schematically showing the configuration of an apparatus 200 for performing an optimized method for providing a surgical tool or a method for providing an optimal entry position of a surgical tool according to an embodiment of the present invention.
105 shows an embodiment for explaining a medical solution model for recognizing and solving a medical problem according to the present invention.
106 is a layer configuration diagram of a surgical analysis apparatus according to an embodiment of the present invention.
107 is a flowchart of a surgical analysis method by a computer according to an embodiment of the present invention.
108 is a flowchart of a method for analyzing surgery by a computer further including a process of providing a surgical product according to an embodiment of the present invention.

Advantages and features of the present invention and methods of achieving them will become apparent with reference to the embodiments described below in detail in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in various different forms, and only these embodiments allow the disclosure of the present invention to be complete, and those of ordinary skill in the art to which the present invention pertains. It is provided to fully understand the scope of the present invention to those skilled in the art, and the present invention is only defined by the scope of the claims.

The terminology used herein is for the purpose of describing the embodiments and is not intended to limit the present invention. As used herein, the singular also includes the plural unless specifically stated otherwise in the phrase. As used herein, “comprises” and/or “comprising” does not exclude the presence or addition of one or more other components in addition to the stated components. Like reference numerals refer to like elements throughout, and "and/or" includes each and every combination of one or more of the recited elements. Although "first", "second", etc. are used to describe various elements, these elements are not limited by these terms, of course. These terms are only used to distinguish one component from another. Accordingly, it goes without saying that the first component mentioned below may be the second component within the spirit of the present invention.

Unless otherwise defined, all terms (including technical and scientific terms) used herein will have the meaning commonly understood by those of ordinary skill in the art to which this invention belongs. In addition, terms defined in a commonly used dictionary are not to be interpreted ideally or excessively unless specifically defined explicitly.

As used herein, the term “unit” or “module” refers to a hardware component such as software, FPGA, or ASIC, and “unit” or “module” performs certain roles. However, “part” or “module” is not meant to be limited to software or hardware. A “unit” or “module” may be configured to reside on an addressable storage medium or to reproduce one or more processors. Thus, by way of example, “part” or “module” refers to components such as software components, object-oriented software components, class components and task components, processes, functions, properties, Includes procedures, subroutines, segments of program code, drivers, firmware, microcode, circuitry, data, databases, data structures, tables, arrays and variables. Components and functionality provided within “parts” or “modules” may be combined into a smaller number of components and “parts” or “modules” or as additional components and “parts” or “modules”. can be further separated.

Hereinafter, various embodiments that can be applied to the medical field are disclosed, and in particular, by utilizing the disclosed embodiments, it is possible to recognize a medical problem and function as a solution model that can solve it.

More specifically, there are surgical elements that must be recognized medically in a medical surgical procedure. In this case, the surgical element to be recognized may include a surgical stage, a body part, an event occurring during surgery, an operation time, a surgical tool, a camera, a surgical operation, and the like. In order to recognize these surgical elements, various techniques are required. Therefore, the embodiments of the present invention to be described later propose methods for recognizing these surgical elements. In addition, by utilizing the recognition of surgical elements, it is possible to finally solve various medical problems that occur in the course of medical surgery.

According to embodiments of the present invention to be described later, methods for recognizing surgical elements using medical images obtained in a medical surgery process are proposed. Furthermore, methods for recognizing surgical elements using the results of learning medical images as well as surgical elements recognized from the medical image itself are also proposed. In addition, through a method of recognizing information about other surgical elements using one surgical element, a method of recognizing individual surgical elements, a method of evaluating the recognition of surgical elements itself, an optimization method of each surgical element, and combinations thereof We propose a method to provide an optimized surgical procedure and a method to apply surgical elements and utilize them in virtual surgery.

That is, if at least one of the methods disclosed in each of the embodiments of the present invention is used in combination, a solution model for solving various medical problems may be generated. As an exemplary model for this, a recognition module for recognizing each surgical element may be configured, and a module for solving the medical problem may be configured by combining at least one or more surgical elements according to a medical problem to be solved. Such a medical model can be implemented through the embodiments of the present invention to be described later, and finally, a medical problem solving method can be derived based on the surgical element recognition technology. An exemplary model for a solution model for solving such a medical problem will be described in more detail with reference to FIGS. 105 to 108 after all embodiments of the present invention are described.

Hereinafter, various embodiments applied to the medical field using a virtual body model are disclosed.

In one embodiment, there is provided a method of constructing surgical simulation information performed by a computer, the method comprising: acquiring a pre-generated virtual body model based on medical image data of an object; acquiring actual surgical data taken during actual surgery on the object; extracting texture information about the object from the actual surgical data; and reflecting the extracted texture information to the virtual body model. A detailed description thereof will be described later with reference to FIGS. 1 to 5 .

In another embodiment, a method of generating a virtual body model using a surgical image performed by a computer, the method comprising: extracting a feature by detecting an organ from a surgical image including a plurality of image frames; matching and connecting the feature points between each of the plurality of image frames; dividing the organ into at least one cluster region by grouping regions having similar movements in the organ based on the connected feature points; and matching the at least one cluster region on a virtual body model. A detailed description thereof will be described later with reference to FIGS. 6 to 14 .

In another embodiment, there is provided a method of using operation information performed by a computer, comprising: obtaining cue sheet data including actual operation information generated in the course of an actual operation on a subject for surgery; obtaining the body information of the surgery subject from the actual surgery information included in the cue sheet data; and generating a virtual body model of the surgery subject based on the subject's body information. A detailed description thereof will be described later with reference to FIGS. 15 to 18 .

1 to 5, a method of constructing surgical simulation information according to an embodiment of the present invention will be described in detail.

In this specification, "image" may mean multidimensional data composed of discrete image elements (eg, pixels in a 2D image and voxels in a 3D image). For example, the image may include a medical image of the object obtained by the CT imaging apparatus.

As used herein, an “object” may be a human or an animal, or a part or all of a human or animal. For example, the object may include at least one of organs such as liver, heart, uterus, brain, breast, abdomen, and blood vessels.

As used herein, a “user” may be a medical professional, such as a doctor, a nurse, a clinical pathologist, or a medical imaging specialist, and may be a technician repairing a medical device, but is not limited thereto.

As used herein, "medical image data" is a medical image captured by a medical imaging device, and includes all medical images that can be implemented as a three-dimensional model of the body of an object. "Medical image data" may include computed tomography (CT) images, magnetic resonance imaging (MRI), positron emission tomography (PET) images, and the like.

As used herein, the term "virtual body model" refers to a model generated to match the actual patient's body based on medical image data. The "virtual body model" may be generated by modeling medical image data in 3D as it is, or may be corrected after modeling to be the same as during actual surgery.

As used herein, “virtual surgical data” refers to data including rehearsal or simulation actions performed on a virtual body model. “Virtual surgical data” may be image data on which rehearsal or simulation is performed on a virtual body model in a virtual space, or data recorded on a surgical operation performed on the virtual body model.

As used herein, “actual surgical data” refers to data obtained by actual medical staff performing surgery. "Actual surgical data" may be image data of a surgical site taken during an actual surgical procedure, or may be data recorded on a surgical operation performed in an actual surgical procedure.

As used herein, the term “computer” includes various devices capable of providing a result to a user by performing arithmetic processing. For example, computers include desktop PCs, notebooks (Note Books) as well as smart phones, tablet PCs, cellular phones, PCS phones (Personal Communication Service phones), synchronous/asynchronous A mobile terminal of International Mobile Telecommunication-2000 (IMT-2000), a Palm Personal Computer (PC), a Personal Digital Assistant (PDA), and the like may also be applicable. Also, when a head mounted display (HMD) device includes a computing function, the HMD device may be a computer. In addition, the computer may correspond to a server that receives a request from a client and performs information processing.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a view showing a robotic surgery system according to an embodiment of the present invention.

Referring to Figure 1, a schematic diagram of a system capable of performing robotic surgery according to an embodiment of the present invention is shown.

According to FIG. 1 , the robotic surgery system includes a medical image capturing device 10 , a server 20 and a controller 30 provided in an operating room, an image capturing unit 36 , a display 32 and a surgical robot 34 . do.

In one embodiment, the robotic surgery is performed by the user using the control unit 30 to control the surgical robot (34). In one embodiment, the robotic surgery may be performed automatically by the control unit 30 without the user's control.

The server 20 is a computing device including at least one processor and a communication unit.

The control unit 30 includes a computing device including at least one processor and a communication unit. In one embodiment, the controller 30 includes hardware and software interfaces for controlling the surgical robot 34 .

The image capturing unit 36 includes at least one image sensor. That is, the image capturing unit 36 includes at least one camera device, and is used to photograph the surgical site. In one embodiment, the imaging unit 36 is used in combination with the surgical robot (34). For example, the imaging unit 36 may include at least one camera coupled to the surgical arm of the surgical robot 34 .

In an embodiment, the image captured by the image capturing unit 36 is displayed on the display 340 .

The control unit 30 receives information necessary for surgery from the server 20 or generates information necessary for surgery and provides it to the user. For example, the controller 30 displays generated or received information necessary for surgery on the display 32 .

For example, the user performs robotic surgery by controlling the movement of the surgical robot 34 by manipulating the controller 30 while viewing the display 32 .

The server 20 generates information necessary for robotic surgery by using medical image data of an object (patient) photographed in advance from the medical imaging device 10 , and provides the generated information to the controller 30 .

The controller 30 provides the information received from the server 20 to the user by displaying it on the display 32 , or controls the surgical robot 34 using the information received from the server 20 .

In one embodiment, the means that can be used in the medical imaging equipment 10 is not limited, for example, CT, X-Ray, PET, MRI, etc., various other medical image acquisition means may be used.

Hereinafter, a method of constructing surgical simulation information will be described in detail with reference to the drawings.

2 is a flowchart illustrating a method of constructing surgical simulation information according to an embodiment of the present invention.

Each of the steps shown in FIG. 2 is performed in time series by the server 20 or the controller 30 shown in FIG. 1 . Hereinafter, each step is described as being performed by a computer for convenience of explanation, but the subject of each step is not limited to a specific device, and all or part of it may be performed by the server 20 or the control unit 30 . can

Referring to FIG. 2 , in the method for constructing surgical simulation information according to an embodiment of the present invention, the computer acquires a pre-generated virtual body model based on medical image data of the object (S100), the actual operation of the object Acquiring actual surgical data taken during surgery (S110), extracting texture information about the object from the actual surgical data (S120), and reflecting the extracted texture information to the virtual body model It includes a step (S130) of doing. Hereinafter, detailed description of each step is described.

The computer may acquire a pre-generated virtual body model based on the medical image data of the object (S100). Here, the object refers to the body of a patient, and may be a body organ or blood vessel. For example, the object may include a liver, heart, uterus, brain, breast, abdomen, and the like.

In an embodiment, the computer may acquire medical image data of the object photographed from the medical imaging equipment 10 such as CT, MRI, PET, etc., and generate a virtual body model in advance based on the acquired medical image data. have. However, the virtual body model created in advance based on medical image data accurately implements the appearance of the object (eg, organ), but features the surface characteristics of the object (eg, color, material, texture, etc.) ) is not expressed. Therefore, it is necessary to provide information reflecting the actual surface characteristics of the object in the virtual body model generated in advance based on the medical image data.

Accordingly, during the actual operation of the patient, the computer may first acquire a pre-generated virtual body model based on the medical image data. Here, the virtual body model may be 3D modeling data generated based on medical image data obtained by photographing all or part of a body organ (ie, an object) that is a target of a patient's surgical site as described above.

Thereafter, the computer may acquire actual surgical data taken during the actual operation on the object (S110). Here, the actual surgical data is data obtained as a medical staff performs an operation, and may be image data obtained by photographing an actual surgical site of the object.

In an embodiment, when minimally invasive surgery such as laparoscopic surgery or robotic surgery is performed, only a surgical tool and a camera are inserted into the patient's surgical site to perform the surgery. In this case, the computer may acquire an image captured by a camera entering the patient's body as actual surgical data. For example, the actual surgical data is the process from when the patient enters the body to the surgical site inside the body, and the process from the start to the end of the action by the surgical tool on the surgical site inside the body. It may be image data.

The computer can extract surface feature information about the object from the actual surgical data, that is, texture information such as color, material, and texture expressed on the surface of the object (S120), and obtain the extracted texture information in step S100. It can be reflected in the virtual body model (S130).

Specifically, the computer may acquire a virtual image of the object from the virtual body model, and may acquire an image of an actual surgical site of the object from the actual surgical data. Thereafter, the computer may obtain a corresponding region corresponding to the image of the actual surgical site from the virtual image, and map and reflect texture information on the actual surgical site of the object to the corresponding region. Here, the virtual image may mean an image obtained from the same viewpoint as viewed with a real camera in a virtual body model constructed in three dimensions.

In an embodiment, a camera that enters the inside of the patient's body during surgery captures all objects in the direction the camera is looking at. That is, the camera records the surgical scene from the start of the action by the surgical tool to the end of the operation on the specific surgical site of the patient. The computer may select image data at a desired point in time among image data (actual surgical data) captured by a camera that enters the patient's body during surgery, and use the selected image data to reflect it in the virtual body model.

For example, when several medical staff practice the same operation, the virtual body model must be implemented in the same state as the actual patient's condition at the start of the operation. To this end, the computer may generate a virtual body model by using the actual surgical image data obtained at the beginning among all the actual surgical data. For example, it is possible to generate a virtual body model in the initial stage of surgery by reflecting texture, fat arrangement, initial organ state, etc. from the actual surgical image data acquired in the early stage. On the other hand, if reoperation is required after performing the primary operation, the medical staff may perform a surgical simulation using a virtual body model in advance before the reoperation. In this case, the computer may construct a virtual body model by reflecting the actual surgical image data acquired in the latter half of all the actual surgical data. Thereafter, when the medical staff simulates the reoperation, virtual surgery data may be generated by performing a process of imitating the actual surgical operation in the virtual body model.

3 is a flowchart illustrating a process of reflecting texture information on an actual surgical site of an object to a virtual body model according to an embodiment of the present invention.

In one embodiment, as shown in Fig. 3, the computer may match the camera position information between the virtual image of the virtual body model and the surgical site image of the actual surgical data in order to make the rendering of the virtual body model and the actual surgical data the same. can be (S131).

For example, the position of the virtual camera of the virtual image may be matched with the position of the camera (eg, a camera included in a surgical tool such as an endoscope) of an image of an actual surgical site.

As described above, by matching camera position information between two images, positions between objects may be equally converted during mapping from one space to another in 2D or 3D.

The computer may match the virtual image and the actual surgical site image based on the location information between the cameras (S132). That is, the computer may match the part of the object included in the virtual image corresponding to the surgical site in the actual surgical site image based on the location information between the cameras.

For example, the computer may extract a portion corresponding to the object from the virtual image. In addition, the computer may adjust the enlargement, reduction, etc. to correspond to the surgical site included in the actual surgical site image of the part extracted from the virtual image. Through this adjustment, it is possible to match the object of the virtual image corresponding to the surgical site of the actual surgical site image. Accordingly, a corresponding region corresponding to the surgical site may be obtained from the object of the virtual image.

The computer may acquire location information of the corresponding area by detecting a corresponding area in the virtual image corresponding to the surgical site from the actual surgical site image through matching between the images (S133).

For example, the computer may derive the position information of the corresponding region in the virtual image based on a conversion relationship between the position information of the camera for the actual surgical site image and the position information of the camera for the virtual image. In this case, location information of the corresponding area may be calculated using a perspective value of the camera.

The computer may extract the surgical site from the actual surgical site image and acquire texture information on the surgical site (S134).

Here, the texture information includes information expressing the surface color of the object (eg, diffuse map), information expressing the surface roughness and height of the object (eg, normal map, height map), and information expressing the surface reflectivity of the object (eg, : occlusion map), etc. In addition to the standardized image-based texture information, a shader capable of generating a texture in real time through calculation may be applied. For example, applying a shader can create real-time changes to the surface of an object, such as the position of all pixels, vertices, and textures used to compose an image, hue, saturation, brightness, contrast, etc., and the normal ( Texture information can be calculated using information about normal and vertical direction) values, general surface properties such as transparency, reflectivity, and object color, and information about the type, number and direction of lighting.

The computer may map the texture information extracted from the image of the actual surgical site to the virtual image based on the location information of the corresponding region in the virtual image (S135).

For example, the computer may extract a surgical site region including texture information from an actual surgical site image, and render the extracted region mounted on the surface of the object based on location information of the corresponding region in the virtual image.

If the above-described process of FIG. 3 is repeatedly executed by moving the camera position, texture information can be extracted for the entire object and mapped to the object, so that a virtual body model identical to the actual organ of the patient can be obtained.

Meanwhile, according to an embodiment of the present invention, the computer may acquire a standardized template image of the object. The standardized template image of the object may be image data obtained by standardizing anatomical features, such as an appearance, size, and location of each object, and building a template of each object as a 3D model. In this case, the computer may acquire the standardized template image and match it with the pre-generated virtual body model obtained in step S100.

For example, the computer may extract a missing part from a virtual image including the object in the virtual body model. In addition, the computer may receive a standardized template image of an object from a database in which a standardized template image is built and stored in advance, and a defect in the virtual image may be corrected based on the received standardized template image. Through this, it is possible to provide a virtual body model more like the appearance of the real object.

Thereafter, the computer may perform a process of reflecting the texture information as described above by using the virtual body model that has been matched with the standardized template image.

Also, according to an embodiment of the present invention, the computer may reconstruct the virtual body model by reflecting the texture information in step S130.

In addition, according to an embodiment of the present invention, the computer may provide surgical information of the subject by comparing it with actual surgical data in real time during an actual operation based on the reconstructed virtual body model.

That is, since the reconstructed virtual body model is a mapping of texture information extracted from the actual surgical data of a patient undergoing actual surgery, it contains the same information as the actual surgical site of the patient. Therefore, if the operation is performed using the reconstructed virtual body model during the actual operation, the same virtual image can be obtained in terms of the actual operation image and additional information related to the operation (e.g., specifying the location of the actual operation site). It can be effective in obtaining the position, direction, movement of surgical instruments, etc.).

4 is a diagram illustrating an example of applying a method for constructing surgical simulation information according to an embodiment of the present invention.

Referring to FIG. 4 , the computer may acquire a pre-generated virtual body model 102 based on the medical image data 100 of the object ( S200 ). Also, the computer may acquire the standardized template image 112 of the object from the database 110 in which the standardized template image is built and stored in advance (S210). In addition, the computer may acquire the actual surgical data 120 of the patient's surgical site during the actual operation (S220).

Then, the computer compares the virtual body model 102 (that is, a 3D modeled virtual image including the patient's object) based on the standardized template image 112 to correct or correct a defect in the virtual body model 102 . This necessary part can be detected. Accordingly, the computer may match the missing part of the virtual body model 102 or the part requiring correction using the standardized template image 112 . Accordingly, the computer may derive a matched virtual body model (S230).

Next, the computer may detect a region corresponding to the actual surgical site from the actual surgical data 120 and extract texture information of the detected real surgical site region (S240).

Next, the computer may obtain a corresponding region corresponding to the detected real surgical site region from the matched virtual body model, and map the texture information of the real surgical site region to the obtained corresponding region ( S250 ). Accordingly, the virtual body model 130 having the same texture information as the actual surgical site of the patient can be finally obtained. That is, the finally obtained virtual body model 130 is reconstructed by reflecting the texture information of the actual surgical site in the virtual body model 102 generated in advance based on the medical image data 100 of the object.

The process of mapping the texture information on the actual surgical site of the patient to the matched virtual body model (steps S240 to S250) may be performed by applying the method of FIG. 3 .

As described above, the present invention provides the same information as the actual surgical site of the patient by constructing surgical simulation information using various information (actual surgical data, virtual body model, standardized template image) about the patient's surgical site during actual surgery. can be provided in real time during surgery.

However, in the case of a conventional 3D organ model constructed using images obtained from CT, MRI, PET, etc., although the external appearance of the organ is relatively accurate, the texture information of the surface such as the color and material of the organ is not expressed, so the actual patient There is no way to know until the organ is operated on. Therefore, conventionally, a method of mapping commonly used colors and materials to each organ, referring to illustrated books, or obtaining and mapping textures such as colors and materials from surgery images of other patients was used. In the case of the surgical simulation model constructed in this way, it is difficult to visually judge even the same patient's organs because surface features are expressed completely differently from the patient's organs in the actual surgical image.

In order to improve such a conventional surgical simulation model, as described above, in the present invention, texture information of a patient's organs extracted from image data during actual surgery is mapped in real time to a virtual body model built in advance for a specific patient to give the surgeon to provide. That is, it is possible to make the actual surgical image of the patient and the surgical simulation (ie, virtual body model) image the same in terms of visual aspect. Through this visual identity, it is effective to compare the surgical image and the surgical simulation image to the surgeon during surgery and to obtain necessary information (eg, the position of the surgical site, etc.) during surgery. In addition, since the appearance of the same organ as the actual surgical image can be reproduced by using the post-operative surgical simulation model, all procedures of surgery (eg, movement of an endoscope or surgical tool) can be expressed as if an actual operation is in progress. can Therefore, the surgical simulation model in the present invention can be utilized in real time during surgery, and can be used for reoperation later or as educational materials. For example, when a physics engine is applied to an organ in a surgical simulation model, it can be visually reproduced identically to an actual patient's organ when reoperation is attempted.

5 is a diagram schematically showing the configuration of an apparatus 300 for constructing surgical simulation information according to an embodiment of the present invention.

Referring to FIG. 5 , the processor 310 includes one or more cores (not shown) and a graphic processing unit (not shown) and/or a connection path (eg, a bus) for transmitting and receiving signals with other components. ) may be included.

The processor 310 according to an embodiment executes one or more instructions stored in the memory 320, thereby performing the surgical simulation information construction method described in relation to FIGS. 2 to 4 .

For example, the processor 310 obtains a pre-generated virtual body model based on the medical image data of the object by executing one or more instructions stored in the memory 320, and the actual surgery taken during the actual operation on the object. It is possible to acquire data, extract texture information about the object from the actual surgical data, and reflect the extracted texture information to the virtual body model.

Meanwhile, the processor 310 temporarily and/or permanently stores a signal (or data) processed inside the processor 310 . , not shown) may be further included. In addition, the processor 310 may be implemented in the form of a system on chip (SoC) including at least one of a graphic processing unit, a RAM, and a ROM.

The memory 320 may store programs (one or more instructions) for processing and controlling the processor 310 . Programs stored in the memory 320 may be divided into a plurality of modules according to functions.

The surgical simulation information construction method according to an embodiment of the present invention described above may be implemented as a program (or application) and stored in a medium to be executed in combination with a computer, which is hardware.

A method of generating a virtual body model using a surgical image according to an embodiment of the present invention will be described in detail with reference to FIGS. 6 to 14 .

6 is a schematic diagram of a system capable of performing robotic surgery according to an embodiment of the present invention.

Referring to FIG. 6 , the robotic surgery system includes a medical imaging device 10 , a server 100 , and a controller 30 , a display 32 , and a surgical robot 34 provided in the operating room. According to an embodiment, the medical imaging device 10 may be omitted from the robotic surgery system according to the disclosed embodiment.

In one embodiment, the surgical robot 34 includes a photographing device 36 and a surgical tool 38 .

In one embodiment, the robotic surgery is performed by the user using the control unit 30 to control the surgical robot (34). In one embodiment, the robotic surgery may be performed automatically by the control unit 30 without the user's control.

The server 100 is a computing device including at least one processor and a communication unit.

The control unit 30 includes a computing device including at least one processor and a communication unit. In one embodiment, the controller 30 includes hardware and software interfaces for controlling the surgical robot 34 .

The photographing device 36 includes at least one image sensor. That is, the imaging device 36 includes at least one camera device, and is used to photograph an object, that is, a surgical site. In an embodiment, the imaging device 36 includes at least one camera coupled to a surgical arm of the surgical robot 34 .

In an embodiment, the image captured by the photographing device 36 is displayed on the display 340 .

In one embodiment, the surgical robot 34 includes one or more surgical tools 38 capable of performing cutting, clipping, fixing, grabbing operations of the surgical site, and the like. The surgical tool 38 is used in combination with the surgical arm of the surgical robot 34 .

The controller 30 receives information necessary for surgery from the server 100 or generates information necessary for surgery and provides it to the user. For example, the controller 30 displays generated or received information necessary for surgery on the display 32 .

For example, the user performs robotic surgery by controlling the movement of the surgical robot 34 by manipulating the controller 30 while viewing the display 32 .

The server 100 generates information necessary for robotic surgery by using the medical image data of the object photographed in advance from the medical imaging device 10 , and provides the generated information to the controller 30 .

The controller 30 provides the information received from the server 100 to the user by displaying it on the display 32 , or controls the surgical robot 34 using the information received from the server 100 .

In one embodiment, the means that can be used in the medical imaging equipment 10 is not limited, for example, CT, X-Ray, PET, MRI, etc., various other medical image acquisition means may be used.

As described above, when performing robotic surgery, it is possible to acquire data including surgical images captured in the surgical process or various surgical information in the control process of the surgical robot. In this way, the surgical images or surgical data acquired in the surgical process can be used as learning materials or used in the surgical process of other patients. Accordingly, in the present invention, a virtual body model is created that allows a surgical simulation to be performed using a surgical image obtained during surgery, and through this, a virtual surgery that provides a more real patient's physical condition and a surgical environment similar to the actual surgical condition We want to provide simulations.

Hereinafter, for convenience of description, it will be described that a "computer" performs a method of generating learning data based on a surgical image according to an embodiment disclosed herein. “Computer” may mean the server 100 or the controller 30 of FIG. 1 , but is not limited thereto and may be used to encompass a device capable of performing computing processing. For example, the computer may be a computing device provided separately from the device illustrated in FIG. 6 .

In addition, the embodiments disclosed below may not be applied only in connection with the robotic surgery system shown in FIG. 6 , but may be applied to all kinds of embodiments in which a surgical image may be acquired and utilized in the surgical process. For example, in addition to robotic surgery, it may be applied in connection with minimally invasive surgery such as laparoscopic surgery or surgery using an endoscope.

7 is a flowchart schematically illustrating a method for generating a virtual body model using a surgical image according to an embodiment of the present invention.

Referring to FIG. 7 , the method for generating a virtual body model using a surgical image according to an embodiment of the present invention includes extracting a feature by detecting an organ from a surgical image including a plurality of image frames (S100) , a step of matching and connecting the feature points between each of the plurality of image frames (S200), grouping regions having similar movements in the organ based on the connected feature points and dividing the organ into at least one cluster region (S300) ), and matching the at least one cluster region on the virtual body model (S400). Hereinafter, detailed description of each step is described.

The computer may extract a feature by detecting an organ from a surgical image including a plurality of image frames (S100).

In one embodiment, the computer may first acquire a surgical image.

When a specific surgery (eg, gastric cancer surgery, colon cancer surgery, etc.) is performed on a patient, medical staff may directly perform an actual surgery on the patient, and use a laparoscopy or an endoscope as well as a surgical robot as described in FIG. 6 . Minimally invasive surgery can also be performed. In this case, the computer may acquire a surgical image obtained by photographing a scene including a surgical operation performed in the surgical procedure and related surgical tools, surgical sites, and the like. For example, the computer may acquire a surgical image that captures a scene including a surgical site and a surgical tool currently being operated from a camera that has entered the patient's body.

Here, the surgical image may include one or more image frames. Each image frame may represent one scene in which a surgical operation is performed including a surgical site of a patient, a surgical tool, and the like. For example, the surgical image may be composed of image frames in which a surgical operation according to time is recorded for each scene in a surgical procedure. Alternatively, the surgical image may be composed of image frames in which each surgical scene is recorded according to spatial movement, such as a position of a surgical site or a camera, during a surgical procedure.

In addition, the surgical image may be composed of all image frames including the entire surgical procedure, but may also be formed in the form of at least one video clip (ie, a sequence) divided according to a specific classification criterion.

According to an embodiment, the computer may acquire a surgical image in the form of a video clip divided into predefined sections for a specific surgery, or acquire a surgical image including all or a part of the surgical procedure, and use it as at least one video clip It can also be divided into surgical images of the form.

In an embodiment, the computer may acquire a surgical image including all or part of a surgical procedure, and may divide the surgical image into at least one video clip type surgical image according to a specific classification criterion. For example, the computer may divide the surgical image according to the lapse of time during which the operation is performed, or the surgical image may be divided according to the position or state change of the surgical site based on the surgical site during surgery. Alternatively, the computer may divide the surgical image based on the position of the camera or the movement range of the camera during the operation, and divide the surgical image based on the change (eg, replacement, etc.) of the surgical tool during the operation. You may. Alternatively, the computer may divide the surgical image based on image frames made up of a series of surgical operations in relation to the surgical operation. In addition, for each operation, a surgical stage classified according to a specific classification criterion may be predetermined. In this case, the computer may divide the surgical image based on the surgical stage.

8 shows an example for explaining a process of dividing a surgical image into at least one video clip (ie, a sequence) according to an embodiment of the present invention.

Referring to FIG. 8 , the computer may acquire a surgical image 200 including all or part of a surgical procedure. A computer can perform learning to determine whether there is an interaction between a surgical tool and an organ in a surgical image using deep learning such as a Convolutional Neural Network (CNN). For example, the computer determines whether or not contact between the surgical tool and the organ occurs from the surgical image 200 through CNN learning, and converts the surgical image 200 into a plurality of video clips ( That is, it can be divided into sequences 210 and 220 . Each of the video clips 210 and 220 may be composed of image frames including a time point when a contact between the surgical tool and an organ occurs to a time point at which the surgical tool is separated from the organ.

Referring back to FIG. 7 , the computer acquires a surgical image divided into a plurality of video clips (ie, a sequence) as described above, and applies processes to be described later for each obtained video clip (ie, a sequence) can do.

That is, the computer may acquire a surgical image (ie, a video clip) including a plurality of image frames, detect organs from the plurality of image frames, and extract feature points.

In one embodiment, the computer may detect an organ associated with the operation of the surgical tool from each of the plurality of image frames. For example, the computer may detect, from each image frame, an organ in which interaction occurs due to operation of a surgical tool, for example, an organ in contact with the surgical tool. At this time, in detecting the organ from the image frame, the computer may apply an image recognition technology (eg, semantic segmentation) using deep learning, and the surgical tool from the image frame according to the characteristics of each surgical tool and each organ. and organs. The computer may derive location information and types of recognized surgical tools and organ regions.

9 shows an example for explaining a process of detecting an organ from an image frame according to an embodiment of the present invention.

Referring to FIG. 9 , the computer may receive an image frame 300 , and detect the organ 310 and the surgical tool 320 using CNN or semantic segmentation for the received image frame 300 . In this case, the computer may derive location information of the detected organ 310 and the surgical tool 320 . The location information may be expressed as coordinate information on a two-dimensional or three-dimensional space. Also, the computer may recognize the operation information of the surgical tool 320 to recognize whether the surgical tool 320 and the organ 310 are in contact.

The computer may extract feature points for the detected organs from each of the plurality of image frames. For example, the computer may extract a feature point using an algorithm such as Scale-Invariant Feature Transform (SIFT) and Speeded Up Robust Features (SURF) for a long-term region within each image frame.

The computer may connect the feature points extracted from the plurality of image frames by matching the feature points between each of the plurality of image frames (S200).

In an embodiment, the computer may match and connect the first feature points extracted from the first image frame and the second feature points extracted from the second image frame among the plurality of image frames. In this way, for all image frames in the surgical image, a trajectory can be generated by matching feature points between each image frame. In this case, in matching the feature points between each image frame, the computer may remove an outlier. Also, the removed portion may be corrected through interpolation or the like.

The computer may divide the organ into at least one cluster region by grouping regions having similar movements in the organ based on the connected feature points ( S300 ).

In one embodiment, first, the computer may determine the degree of movement of the organ according to the operation of the surgical tool based on the connected feature points. When contact occurs between the surgical tool and the organ due to the operation of the surgical tool, a physical force is applied to the contact position within the organ, so a change (ie, movement) may occur in a specific area of the organ based on the contact position. . As such, when a change (ie, movement) of an organ occurs, the feature points extracted with respect to the organ may also change in a specific pattern. Therefore, in relation to the surgical tool, the organ may have a high degree of movement in some areas, and a small degree of movement may occur in some other areas. That is, organs may exhibit different movement responses in relation to surgical tools. Accordingly, the computer can group regions with similar movements in the organ based on the degree of movement.

For example, the computer may extract a contact position where the contact occurs between the surgical tool and the organ according to the operation of the surgical tool from each of the plurality of image frames. The computer may obtain a position change value of the organ based on the contact position, and determine the degree of movement of the organ based on the acquired position change value of the organ. For example, the computer may obtain a position change value for each feature point extracted from each image frame, and detect feature points having a similar degree of movement in an organ based on this.

Next, based on the degree of movement of the organ, the computer may group regions having similar movements in the organ and divide the region into at least one cluster region. For example, the computer may detect feature points having the same or similar position change value for the organ (ie, position change value with respect to the feature point), and group the same or similar feature points respectively. And the computer may allocate each grouped feature point to each cluster area.

In addition, in grouping regions having similar movements in an organ, the computer may apply an algorithm such as motion segmentation to each feature point extracted from each image frame. As a result of applying motion segmentation to the feature points, the computer can detect each independently moving feature point in the organ. In other words, the computer may detect feature points having similar degrees of movement within the organ. Accordingly, the computer can divide the organ into a plurality of cluster regions by grouping feature points having similar motions.

According to an embodiment, the computer may estimate the elasticity parameter for each of the divided cluster regions by grouping regions having similar movements in the organ. Each organ can have a certain range of elasticity, which is predetermined. Therefore, when a movement occurs as a physical force is applied to an organ by the operation of a surgical tool, the elasticity parameter of the organ can be inversely estimated by reflecting the degree of movement in the predetermined elasticity degree of the organ.

In an embodiment, the computer may estimate the elasticity parameter of each cluster region based on the position change value for each feature point. That is, the elasticity parameter of each cluster region may be calculated by reflecting the degree of movement of each cluster region (ie, a position change value for each feature point) in the predetermined elasticity degree of the corresponding organ.

In addition, the computer may perform learning using deep learning (eg, CNN) in the process of calculating the position change value of the organ according to the operation of the surgical tool and determining the degree of movement of the organ based on this. For example, the computer recognizes the type, position, and operation of the surgical tool in the image frame, and performs CNN learning based on the information of the surgical tool to obtain the position change value of the organ according to the operation of the surgical tool as a learning result. can Accordingly, the computer can estimate the elastic parameter of each cluster region in the organ by using the position change value of the organ obtained as a result of the learning.

The computer may match at least one cluster region divided for an organ on the virtual body model (S400).

Here, the virtual body model may be 3D modeling data generated based on medical image data (eg, medical images photographed through CT, PET, MRI, etc.) photographed in advance of the inside of the patient's body. . For example, it is modeled in conformity with the body of the subject to be operated on, and may be calibrated to the same state as the actual surgical state. Medical staff can perform rehearsals or simulations using a virtual body model that is implemented in the same way as the physical state of the subject to be operated on, and through this, they can experience the same state as during the actual operation. In addition, when performing virtual surgery using a virtual body model, virtual surgery data including rehearsal or simulation actions for the virtual body model can be obtained. For example, the virtual surgery data may be a virtual surgery image including a surgical site on which a virtual surgery has been performed on the virtual body model, or may be data recorded for a surgical operation performed on the virtual body model.

In addition, the surgical image including the plurality of image frames obtained in step S100 may be a stereoscopic 3D image, and accordingly, the surgical image may be a three-dimensional image, that is, an image having a sense of depth. Accordingly, the computer may acquire depth information (ie, a depth map) from each image frame.

As an embodiment, the computer may convert at least one cluster region into 3D coordinate information based on a depth map for each of the plurality of image frames. The computer may detect and match a point corresponding to the 3D coordinate information on the virtual body model based on the 3D coordinate information for the at least one cluster region. This will be described in detail with reference to FIG. 10 .

10 shows an example for explaining a process of matching at least one cluster region on a virtual body model according to an embodiment of the present invention.

Referring to FIG. 10 , the computer may group feature points having similar movements in an organ based on the feature points extracted from the image frame 400 and divide it into a plurality of cluster regions 410 and 420 .

The computer may match the divided plurality of cluster regions 410 and 420 to the corresponding organ 500 on the virtual body model. In this case, the computer may obtain a depth map for the plurality of cluster areas 410 and 420, and calculate coordinate information in a three-dimensional space of each feature point in the plurality of cluster areas 410 and 420 based on the depth map. .

The computer may compare the 3D coordinate information of each feature point in the plurality of cluster regions 410 and 420 with the 3D coordinate information of the virtual body model to detect pairs having the closest coordinate positions in each coordinate space. Thereafter, the computer may model the plurality of cluster regions 410 and 420 in the corresponding organ 500 on the virtual body model by matching each detected pair. For example, the computer may use an Iterative Closet Point (ICP) algorithm to match each feature point in the plurality of cluster areas 410 and 420 with the corresponding organ 500 of the virtual body model.

In addition, in matching the plurality of cluster regions 410 and 420 to the corresponding organ 500 on the virtual body model, the computer applies the elastic parameters estimated for each of the plurality of cluster regions 410 and 420 to the virtual body model. can do. As shown in FIG. 5 , the corresponding organ 500 of the virtual body model may be modeled as a plurality of divided regions 510 and 520 that are matched to correspond to each of the plurality of cluster regions 410 and 420 . Each of the plurality of divided regions 510 and 520 may have the same motion response information as that of each of the plurality of cluster regions 410 and 420 .

According to the present invention, by generating a virtual body model implemented based on a surgical image obtained in an actual surgical procedure, it is possible to give the same effect as an actual surgery during simulation through the virtual body model.

In addition, according to the present invention, by implementing a virtual body model based on the changes applied to the organ by performing the operation of the surgical tool in the actual surgical procedure, the degree of deformation or movement of the organ during actual surgery can be measured through the virtual body model. The same can be reproduced during simulation.

In addition, according to the present invention, the movement of the surgical tool is analyzed to predict how the organ will react according to the movement of the surgical tool, and the predicted organ response is expressed on the virtual body model. In addition, by using such a virtual body model, realistic training can be performed like actual surgery.

11 to 13 are diagrams showing an example of applying a method for generating a virtual body model using a surgical image according to an embodiment of the present invention. In the embodiment of FIGS. 11 to 13 , a description of an operation process overlapping with the contents described with reference to FIGS. 7 to 10 described above will be omitted.

Referring to FIG. 11 , the computer may acquire a surgical image including a surgical procedure ( S500 ). In this case, the computer may acquire a plurality of (eg, N) surgical images. For example, the plurality of surgical images may be images of surgical procedures of different patients, respectively, and each surgical procedure may be a result of different surgeries.

The computer may determine whether the surgical tool and the organ are in contact through CNN learning for each of the plurality of acquired surgical images (S502).

For each of the plurality of surgical images, the computer may divide each surgical image into a plurality of video clips (ie, sequences) based on whether the surgical tool and the organ are in contact (S504). For example, the computer divides the first surgical image from the point in time when the contact between the surgical tool and the organ to the point in time when the contact between the surgical tool and the organ is terminated into one video clip, and finally divides the first surgical image into a plurality of video clips. It can be created with (eg, M) video clips.

The computer may repeatedly perform the following processes for each video clip divided from each of a plurality of surgical images. For convenience of description, the following process will be described based on one video clip, but may be applied to all video clips.

When the acquired surgical image is a stereoscopic 3D image, the computer may calculate a depth map for an image frame in a video clip (S510).

The computer may recognize the position and operation of the surgical tool from the image frame in the video clip (S520), and store information related to the recognized surgical tool (S522). In one embodiment, the computer may recognize the surgical tool from the image frame using CNN.

The computer may extract the organ that has come into contact with the surgical tool recognized in step S520 from the image frame in the video clip (S524). In an embodiment, the computer may extract an organ that has come into contact with a surgical tool by using semantic segmentation from the image frame. Also, the computer may acquire the organ feature points extracted from each image frame in the video clip, and match the acquired feature points between each image frame to connect them.

The computer may perform motion segmentation by grouping and segmenting regions having similar movements in an organ based on the connected feature points between each image frame (S530).

The computer may match each divided region for the organ on the virtual body model, and in this case, the ICP algorithm may be used (S540).

In step S530, the computer may group and divide the feature points based on the degree of movement in the organ. For example, the degree of movement within the organ may be calculated using the change value of the position of the organ. Referring to FIG. 12 , the computer may recognize the type, position, operation, etc. of a surgical tool from each image frame (S600), and perform CNN learning based on the recognized information of the surgical tool (S610). As a result of the learning, the computer may acquire a position change value of the organ, which is the degree to which the movement of the organ is changed according to the operation of the surgical tool (S620).

In addition, the computer may generate a movement response of an organ according to the operation of a surgical tool on the virtual body model, and use this to determine the degree of movement of the organ. Referring to FIG. 13 , the computer recognizes whether the surgical tool and the organ are in contact on the virtual body model (S700), and extracts information such as the recognized contact location between the surgical tool and the organ, the type, location, and operation of the surgical tool. can be (S710). The computer may perform CNN learning based on the extracted information (S720). The computer can predict the degree of movement of the organ according to the operation of the surgical tool as a result of the learning, and this can be implemented as a learning model. Accordingly, the computer can reflect the degree of movement of the organ according to the operation of the surgical tool obtained as a result of the learning and express it on the virtual body model (S730).

Based on the degree of movement within the organ (that is, the change in the position of the organ or the movement response of the organ on the virtual body model as described above), the computer groups the feature points for the organ and calculates the elasticity parameter for each divided cluster region. can be estimated In step S540, the computer may match the organs on the virtual body model based on the elastic parameters for each cluster region.

14 is a diagram schematically showing the configuration of an apparatus 600 for performing a method for generating a virtual body model using a surgical image according to an embodiment of the present invention.

Referring to FIG. 14 , the processor 610 includes one or more cores (not shown) and a graphic processing unit (not shown) and/or a connection path (eg, a bus) for transmitting and receiving signals with other components. ) may be included.

The processor 610 according to an embodiment executes one or more instructions stored in the memory 620 , thereby performing the method of generating a virtual body model using the surgical image described with reference to FIGS. 7 to 13 .

As an example, the processor 610 detects an organ from a surgical image including a plurality of image frames by executing one or more instructions stored in the memory 620 to extract a feature point, between each of the plurality of image frames. connecting the feature points by matching, grouping regions having similar movements in the organ based on the connected feature points to divide the organ into at least one cluster region, and dividing the at least one cluster region into a virtual body model A step of matching may be performed.

Meanwhile, the processor 610 includes a random access memory (RAM) and a read-only memory (ROM) that temporarily and/or permanently store a signal (or data) processed inside the processor 610 . , not shown) may be further included. In addition, the processor 610 may be implemented in the form of a system on chip (SoC) including at least one of a graphic processing unit, a RAM, and a ROM.

The memory 620 may store programs (one or more instructions) for processing and controlling the processor 610 . Programs stored in the memory 620 may be divided into a plurality of modules according to functions.

The method for generating a virtual body model using a surgical image according to an embodiment of the present invention described above may be implemented as a program (or application) to be executed in combination with a computer, which is hardware, and stored in a medium.

6 is a schematic diagram of a system capable of performing robotic surgery according to an embodiment of the present invention.

Referring to FIG. 6 , the robotic surgery system includes a medical imaging device 10 , a server 100 , and a controller 30 , a display 32 , and a surgical robot 34 provided in the operating room. According to an embodiment, the medical imaging device 10 may be omitted from the robotic surgery system according to the disclosed embodiment.

In one embodiment, the surgical robot 34 includes a photographing device 36 and a surgical tool 38 .

In one embodiment, the robotic surgery is performed by the user using the control unit 30 to control the surgical robot (34). In one embodiment, the robotic surgery may be performed automatically by the control unit 30 without the user's control.

The server 100 is a computing device including at least one processor and a communication unit.

The control unit 30 includes a computing device including at least one processor and a communication unit. In one embodiment, the controller 30 includes hardware and software interfaces for controlling the surgical robot 34 .

The photographing device 36 includes at least one image sensor. That is, the imaging device 36 includes at least one camera device, and is used to photograph an object, that is, a surgical site. In an embodiment, the imaging device 36 includes at least one camera coupled to a surgical arm of the surgical robot 34 .

In an embodiment, the image captured by the photographing device 36 is displayed on the display 340 .

In one embodiment, the surgical robot 34 includes one or more surgical tools 38 capable of performing cutting, clipping, fixing, grabbing operations of the surgical site, and the like. The surgical tool 38 is used in combination with the surgical arm of the surgical robot 34 .

The controller 30 receives information necessary for surgery from the server 100 or generates information necessary for surgery and provides it to the user. For example, the controller 30 displays generated or received information necessary for surgery on the display 32 .

For example, the user performs robotic surgery by controlling the movement of the surgical robot 34 by manipulating the controller 30 while viewing the display 32 .

The server 100 generates information necessary for robotic surgery by using the medical image data of the object photographed in advance from the medical imaging device 10 , and provides the generated information to the controller 30 .

The controller 30 provides the information received from the server 100 to the user by displaying it on the display 32 , or controls the surgical robot 34 using the information received from the server 100 .

In one embodiment, the means that can be used in the medical imaging equipment 10 is not limited, for example, CT, X-Ray, PET, MRI, etc., various other medical image acquisition means may be used.

As described above, when performing robotic surgery, it is possible to acquire data including surgical images captured in the surgical process or various surgical information in the control process of the surgical robot. In this way, the surgical images or surgical data acquired in the surgical process can be used as learning materials or used in the surgical process of other patients. Accordingly, in the present invention, a virtual body model is created that allows a surgical simulation to be performed using a surgical image obtained during surgery, and through this, a virtual surgery that provides a more real patient's physical condition and a surgical environment similar to the actual surgical condition We want to provide simulations.

Hereinafter, for convenience of description, it will be described that a "computer" performs a method of generating learning data based on a surgical image according to an embodiment disclosed herein. “Computer” may mean the server 100 or the controller 30 of FIG. 1 , but is not limited thereto and may be used to encompass a device capable of performing computing processing. For example, the computer may be a computing device provided separately from the device illustrated in FIG. 6 .

In addition, the embodiments disclosed below may not be applied only in connection with the robotic surgery system shown in FIG. 6 , but may be applied to all kinds of embodiments in which a surgical image may be acquired and utilized in the surgical process. For example, in addition to robotic surgery, it may be applied in connection with minimally invasive surgery such as laparoscopic surgery or surgery using an endoscope.

7 is a flowchart schematically illustrating a method for generating a virtual body model using a surgical image according to an embodiment of the present invention.

Referring to FIG. 7 , the method for generating a virtual body model using a surgical image according to an embodiment of the present invention includes extracting a feature by detecting an organ from a surgical image including a plurality of image frames (S100) , a step of matching and connecting the feature points between each of the plurality of image frames (S200), grouping regions having similar movements in the organ based on the connected feature points and dividing the organ into at least one cluster region (S300) ), and matching the at least one cluster region on the virtual body model (S400). Hereinafter, detailed description of each step is described.

The computer may extract a feature by detecting an organ from a surgical image including a plurality of image frames (S100).

In one embodiment, the computer may first acquire a surgical image.

When a specific surgery (eg, gastric cancer surgery, colon cancer surgery, etc.) is performed on a patient, medical staff may directly perform an actual surgery on the patient, and use a laparoscopy or an endoscope as well as a surgical robot as described in FIG. 6 . Minimally invasive surgery can also be performed. In this case, the computer may acquire a surgical image obtained by photographing a scene including a surgical operation performed in the surgical procedure and related surgical tools, surgical sites, and the like. For example, the computer may acquire a surgical image that captures a scene including a surgical site and a surgical tool currently being operated from a camera that has entered the patient's body.

Here, the surgical image may include one or more image frames. Each image frame may represent one scene in which a surgical operation is performed including a surgical site of a patient, a surgical tool, and the like. For example, the surgical image may be composed of image frames in which a surgical operation according to time is recorded for each scene in a surgical procedure. Alternatively, the surgical image may be composed of image frames in which each surgical scene is recorded according to spatial movement, such as a position of a surgical site or a camera, during a surgical procedure.

In addition, the surgical image may be composed of all image frames including the entire surgical procedure, but may also be formed in the form of at least one video clip (ie, a sequence) divided according to a specific classification criterion.

According to an embodiment, the computer may acquire a surgical image in the form of a video clip divided into predefined sections for a specific surgery, or acquire a surgical image including all or a part of the surgical procedure, and use it as at least one video clip It can also be divided into surgical images of the form.

In an embodiment, the computer may acquire a surgical image including all or part of a surgical procedure, and may divide the surgical image into at least one video clip type surgical image according to a specific classification criterion. For example, the computer may divide the surgical image according to the lapse of time during which the operation is performed, or the surgical image may be divided according to the position or state change of the surgical site based on the surgical site during surgery. Alternatively, the computer may divide the surgical image based on the position of the camera or the movement range of the camera during the operation, and divide the surgical image based on the change (eg, replacement, etc.) of the surgical tool during the operation. You may. Alternatively, the computer may divide the surgical image based on image frames made up of a series of surgical operations in relation to the surgical operation. In addition, for each operation, a surgical stage classified according to a specific classification criterion may be predetermined. In this case, the computer may divide the surgical image based on the surgical stage.

8 shows an example for explaining a process of dividing a surgical image into at least one video clip (ie, a sequence) according to an embodiment of the present invention.

Referring to FIG. 8 , the computer may acquire a surgical image 200 including all or part of a surgical procedure. A computer can perform learning to determine whether there is an interaction between a surgical tool and an organ in a surgical image using deep learning such as a Convolutional Neural Network (CNN). For example, the computer determines whether or not contact between the surgical tool and the organ occurs from the surgical image 200 through CNN learning, and converts the surgical image 200 into a plurality of video clips ( That is, it can be divided into sequences 210 and 220 . Each of the video clips 210 and 220 may be composed of image frames including a time point when a contact between the surgical tool and an organ occurs to a time point at which the surgical tool is separated from the organ.

Referring back to FIG. 7 , the computer acquires a surgical image divided into a plurality of video clips (ie, a sequence) as described above, and applies processes to be described later for each obtained video clip (ie, a sequence) can do.

That is, the computer may acquire a surgical image (ie, a video clip) including a plurality of image frames, detect organs from the plurality of image frames, and extract feature points.

In one embodiment, the computer may detect an organ associated with the operation of the surgical tool from each of the plurality of image frames. For example, the computer may detect, from each image frame, an organ in which interaction occurs due to operation of a surgical tool, for example, an organ in contact with the surgical tool. In this case, in detecting the organ from the image frame, the computer may apply an image recognition technology (eg, semantic segmentation) using deep learning, and the surgical tool from the image frame according to the characteristics of each surgical tool and each organ. and organs. The computer may derive location information and types of recognized surgical tools and organ regions.

9 shows an example for explaining a process of detecting an organ from an image frame according to an embodiment of the present invention.

Referring to FIG. 9 , the computer may receive an image frame 300 , and detect the organ 310 and the surgical tool 320 using CNN or semantic segmentation for the received image frame 300 . In this case, the computer may derive location information of the detected organ 310 and the surgical tool 320 . The location information may be expressed as coordinate information on a two-dimensional or three-dimensional space. Also, the computer may recognize the operation information of the surgical tool 320 to recognize whether the surgical tool 320 and the organ 310 are in contact.

The computer may extract feature points for the detected organs from each of the plurality of image frames. For example, the computer may extract a feature point using an algorithm such as Scale-Invariant Feature Transform (SIFT) and Speeded Up Robust Features (SURF) for a long-term region within each image frame.

The computer may connect the feature points extracted from the plurality of image frames by matching the feature points between each of the plurality of image frames (S200).

In an embodiment, the computer may match and connect the first feature points extracted from the first image frame and the second feature points extracted from the second image frame among the plurality of image frames. In this way, for all image frames in the surgical image, a trajectory can be generated by matching feature points between each image frame. In this case, in matching the feature points between each image frame, the computer may remove an outlier. Also, the removed portion may be corrected through interpolation or the like.

The computer may divide the organ into at least one cluster region by grouping regions having similar movements in the organ based on the connected feature points ( S300 ).

In one embodiment, first, the computer may determine the degree of movement of the organ according to the operation of the surgical tool based on the connected feature points. When contact occurs between the surgical tool and the organ due to the operation of the surgical tool, a physical force is applied to the contact position within the organ, so a change (ie, movement) may occur in a specific area of the organ based on the contact position. . As such, when a change (ie, movement) of an organ occurs, the feature points extracted with respect to the organ may also change in a specific pattern. Therefore, in relation to the surgical tool, the organ may have a high degree of movement in some areas, and a small degree of movement may occur in some other areas. That is, organs may exhibit different movement responses in relation to surgical tools. Accordingly, the computer can group regions with similar movements in the organ based on the degree of movement.

For example, the computer may extract a contact position where the contact occurs between the surgical tool and the organ according to the operation of the surgical tool from each of the plurality of image frames. The computer may obtain a position change value of the organ based on the contact position, and determine the degree of movement of the organ based on the acquired position change value of the organ. For example, the computer may obtain a position change value for each feature point extracted from each image frame, and detect feature points having a similar degree of movement in an organ based on this.

Next, based on the degree of movement of the organ, the computer may group regions having similar movements in the organ and divide the region into at least one cluster region. For example, the computer may detect feature points having the same or similar position change value for the organ (ie, position change value with respect to the feature point), and group the same or similar feature points respectively. And the computer may allocate each grouped feature point to each cluster area.

In addition, in grouping regions having similar movements in an organ, the computer may apply an algorithm such as motion segmentation to each feature point extracted from each image frame. As a result of applying motion segmentation to the feature points, the computer can detect each independently moving feature point in the organ. In other words, the computer may detect feature points having similar degrees of movement within the organ. Accordingly, the computer can divide the organ into a plurality of cluster regions by grouping feature points having similar motions.

According to an embodiment, the computer may estimate the elasticity parameter for each of the divided cluster regions by grouping regions having similar movements in the organ. Each organ can have a certain range of elasticity, which is predetermined. Therefore, when a movement occurs as a physical force is applied to an organ by the operation of a surgical tool, the elasticity parameter of the organ can be inversely estimated by reflecting the degree of movement in the predetermined elasticity degree of the organ.

In an embodiment, the computer may estimate the elasticity parameter of each cluster region based on the position change value for each feature point. That is, the elasticity parameter of each cluster region may be calculated by reflecting the degree of movement of each cluster region (ie, a position change value for each feature point) in the predetermined elasticity degree of the corresponding organ.

In addition, the computer may perform learning using deep learning (eg, CNN) in the process of calculating the position change value of the organ according to the operation of the surgical tool and determining the degree of movement of the organ based on this. For example, the computer recognizes the type, position, and operation of the surgical tool in the image frame, and performs CNN learning based on the information of the surgical tool to obtain the position change value of the organ according to the operation of the surgical tool as a learning result. can Accordingly, the computer can estimate the elastic parameter of each cluster region in the organ by using the position change value of the organ obtained as a result of the learning.

The computer may match at least one cluster region divided for an organ on the virtual body model (S400).

Here, the virtual body model may be 3D modeling data generated based on medical image data (eg, medical images photographed through CT, PET, MRI, etc.) photographed in advance of the inside of the patient's body. . For example, it is modeled in conformity with the body of the subject to be operated on, and may be calibrated to the same state as the actual surgical state. Medical staff can perform rehearsals or simulations using a virtual body model that is implemented in the same way as the physical state of the subject to be operated on, and through this, they can experience the same state as during the actual operation. In addition, when performing virtual surgery using a virtual body model, virtual surgery data including rehearsal or simulation actions for the virtual body model can be obtained. For example, the virtual surgery data may be a virtual surgery image including a surgical site on which a virtual surgery has been performed on the virtual body model, or may be data recorded for a surgical operation performed on the virtual body model.

In addition, the surgical image including the plurality of image frames obtained in step S100 may be a stereoscopic 3D image, and accordingly, the surgical image may be a three-dimensional image, that is, an image having a sense of depth. Accordingly, the computer may acquire depth information (ie, a depth map) from each image frame.

As an embodiment, the computer may convert at least one cluster region into 3D coordinate information based on a depth map for each of the plurality of image frames. The computer may detect and match a point corresponding to the 3D coordinate information on the virtual body model based on the 3D coordinate information for the at least one cluster region. This will be described in detail with reference to FIG. 10 .

10 shows an example for explaining a process of matching at least one cluster region on a virtual body model according to an embodiment of the present invention.

Referring to FIG. 10 , the computer may group feature points having similar movements in an organ based on the feature points extracted from the image frame 400 and divide it into a plurality of cluster regions 410 and 420 .

The computer may match the divided plurality of cluster regions 410 and 420 to the corresponding organ 500 on the virtual body model. In this case, the computer may obtain a depth map for the plurality of cluster areas 410 and 420 , and calculate coordinate information on the three-dimensional space of each feature point in the plurality of cluster areas 410 and 420 based on the depth map. .

The computer may compare the 3D coordinate information of each feature point in the plurality of cluster regions 410 and 420 with the 3D coordinate information of the virtual body model to detect pairs having the closest coordinate positions in each coordinate space. Thereafter, the computer may model the plurality of cluster regions 410 and 420 in the corresponding organ 500 on the virtual body model by matching each detected pair. For example, the computer may use an Iterative Closet Point (ICP) algorithm to match each feature point in the plurality of cluster areas 410 and 420 with the corresponding organ 500 of the virtual body model.

In addition, in matching the plurality of cluster regions 410 and 420 to the corresponding organ 500 on the virtual body model, the computer applies the elastic parameters estimated for each of the plurality of cluster regions 410 and 420 to the virtual body model. can do. As shown in FIG. 5 , the corresponding organ 500 of the virtual body model may be modeled as a plurality of divided regions 510 and 520 that are matched to correspond to each of the plurality of cluster regions 410 and 420 . Each of the plurality of divided regions 510 and 520 may have the same motion response information as that of each of the plurality of cluster regions 410 and 420 .

According to the present invention, by generating a virtual body model implemented based on a surgical image obtained in an actual surgical procedure, it is possible to give the same effect as an actual surgery during simulation through the virtual body model.

In addition, according to the present invention, by implementing a virtual body model based on the changes applied to the organ by performing the operation of the surgical tool in the actual surgical procedure, the degree of deformation or movement of the organ during actual surgery can be measured through the virtual body model. The same can be reproduced during simulation.

In addition, according to the present invention, the movement of the surgical tool is analyzed to predict how the organ will react according to the movement of the surgical tool, and the predicted organ response is expressed on the virtual body model. In addition, by using such a virtual body model, realistic training can be performed like actual surgery.

11 to 13 are diagrams showing an example of applying a method for generating a virtual body model using a surgical image according to an embodiment of the present invention. In the embodiment of FIGS. 11 to 13 , a description of an operation process overlapping with the contents described with reference to FIGS. 7 to 10 described above will be omitted.

Referring to FIG. 11 , the computer may acquire a surgical image including a surgical procedure ( S500 ). In this case, the computer may acquire a plurality of (eg, N) surgical images. For example, the plurality of surgical images may be images of surgical procedures of different patients, respectively, and each surgical procedure may be a result of different surgeries.

The computer may determine whether the surgical tool and the organ are in contact through CNN learning for each of the plurality of acquired surgical images (S502).

For each of the plurality of surgical images, the computer may divide each surgical image into a plurality of video clips (ie, sequences) based on whether the surgical tool and the organ are in contact (S504). For example, the computer divides the first surgical image from the point in time when the contact between the surgical tool and the organ to the point in time when the contact between the surgical tool and the organ is terminated into one video clip, and finally divides the first surgical image into a plurality of video clips. It can be created with (eg, M) video clips.

The computer may repeatedly perform the following processes for each video clip divided from each of a plurality of surgical images. For convenience of description, the following process will be described based on one video clip, but may be applied to all video clips.

When the acquired surgical image is a stereoscopic 3D image, the computer may calculate a depth map for an image frame in a video clip (S510).

The computer may recognize the position and operation of the surgical tool from the image frame in the video clip (S520), and store information related to the recognized surgical tool (S522). In one embodiment, the computer may recognize the surgical tool from the image frame using CNN.

The computer may extract the organ that has come into contact with the surgical tool recognized in step S520 from the image frame in the video clip (S524). In an embodiment, the computer may extract an organ that has come into contact with a surgical tool by using semantic segmentation from the image frame. Also, the computer may acquire the organ feature points extracted from each image frame in the video clip, and match the acquired feature points between each image frame to connect them.

The computer may perform motion segmentation by grouping and segmenting regions having similar movements in an organ based on the connected feature points between each image frame (S530).

The computer may match each divided region for the organ on the virtual body model, and in this case, the ICP algorithm may be used (S540).

In step S530, the computer may group and divide the feature points based on the degree of movement in the organ. For example, the degree of movement within the organ may be calculated using the change value of the position of the organ. Referring to FIG. 12 , the computer may recognize the type, position, operation, etc. of a surgical tool from each image frame (S600), and perform CNN learning based on the recognized information of the surgical tool (S610). As a result of the learning, the computer may acquire a position change value of the organ, which is the degree to which the movement of the organ is changed according to the operation of the surgical tool (S620).

In addition, the computer may generate a movement response of an organ according to the operation of a surgical tool on the virtual body model, and use this to determine the degree of movement of the organ. Referring to FIG. 13 , the computer recognizes whether the surgical tool and the organ are in contact on the virtual body model (S700), and extracts information such as the recognized contact location between the surgical tool and the organ, the type, location, and operation of the surgical tool. can be (S710). The computer may perform CNN learning based on the extracted information (S720). The computer can predict the degree of movement of the organ according to the operation of the surgical tool as a result of the learning, and this can be implemented as a learning model. Accordingly, the computer can reflect the degree of movement of the organ according to the operation of the surgical tool obtained as a result of the learning and express it on the virtual body model (S730).

Based on the degree of movement within the organ (that is, the change in the position of the organ or the movement response of the organ on the virtual body model as described above), the computer groups the feature points for the organ and calculates the elasticity parameter for each divided cluster region. can be estimated In step S540, the computer may match the organs on the virtual body model based on the elastic parameters for each cluster region.

14 is a diagram schematically showing the configuration of an apparatus 600 for performing a method for generating a virtual body model using a surgical image according to an embodiment of the present invention.

Referring to FIG. 14 , the processor 610 includes one or more cores (not shown) and a graphic processing unit (not shown) and/or a connection path (eg, a bus) for transmitting and receiving signals with other components. ) may be included.

The processor 610 according to an embodiment executes one or more instructions stored in the memory 620 , thereby performing the method of generating a virtual body model using the surgical image described with reference to FIGS. 7 to 13 .

As an example, the processor 610 detects an organ from a surgical image including a plurality of image frames by executing one or more instructions stored in the memory 620 to extract a feature point, between each of the plurality of image frames. connecting the feature points by matching, grouping regions having similar movements in the organ based on the connected feature points to divide the organ into at least one cluster region, and dividing the at least one cluster region into a virtual body model A step of matching may be performed.

Meanwhile, the processor 610 includes a random access memory (RAM) and a read-only memory (ROM) that temporarily and/or permanently store a signal (or data) processed inside the processor 610 . , not shown) may be further included. In addition, the processor 610 may be implemented in the form of a system on chip (SoC) including at least one of a graphic processing unit, a RAM, and a ROM.

The memory 620 may store programs (one or more instructions) for processing and controlling the processor 610 . Programs stored in the memory 620 may be divided into a plurality of modules according to functions.

The method for generating a virtual body model using a surgical image according to an embodiment of the present invention described above may be implemented as a program (or application) to be executed in combination with a computer, which is hardware, and stored in a medium.

A method of utilizing surgical information according to an embodiment of the present invention will be described in detail with reference to FIGS. 15 to 18 .

15 is a schematic diagram of a system capable of performing robotic surgery according to an embodiment of the present invention.

According to FIG. 15 , the robotic surgery system includes a medical imaging device 10 , a server 100 , and a control unit 30 , a display 32 , and a surgical robot 34 provided in the operating room. According to an embodiment, the medical imaging device 10 may be omitted from the robotic surgery system according to the disclosed embodiment.

In one embodiment, the surgical robot 34 includes a photographing device 36 and a surgical tool 38 .

In one embodiment, the robotic surgery is performed by the user using the control unit 30 to control the surgical robot (34). In one embodiment, the robotic surgery may be performed automatically by the control unit 30 without the user's control.

The server 100 is a computing device including at least one processor and a communication unit.

The control unit 30 includes a computing device including at least one processor and a communication unit. In one embodiment, the controller 30 includes hardware and software interfaces for controlling the surgical robot 34 .

The photographing device 36 includes at least one image sensor. That is, the imaging device 36 includes at least one camera device, and is used to photograph an object, that is, a surgical site. In an embodiment, the imaging device 36 includes at least one camera coupled to a surgical arm of the surgical robot 34 .

In an embodiment, the image captured by the photographing device 36 is displayed on the display 340 .

In one embodiment, the surgical robot 34 includes one or more surgical tools 38 capable of performing cutting, clipping, fixing, grabbing operations of the surgical site, and the like. The surgical tool 38 is used in combination with the surgical arm of the surgical robot 34 .

The controller 30 receives information necessary for surgery from the server 100 or generates information necessary for surgery and provides it to the user. For example, the controller 30 displays generated or received information necessary for surgery on the display 32 .

For example, the user performs robotic surgery by controlling the movement of the surgical robot 34 by manipulating the controller 30 while viewing the display 32 .

The server 100 generates information necessary for robotic surgery by using the medical image data of the object photographed in advance from the medical imaging device 10 , and provides the generated information to the controller 30 .

The controller 30 provides the information received from the server 100 to the user by displaying it on the display 32 , or controls the surgical robot 34 using the information received from the server 100 .

In one embodiment, the means that can be used in the medical imaging equipment 10 is not limited, for example, CT, X-Ray, PET, MRI, etc., various other medical image acquisition means may be used.

As described above, when performing robotic surgery, it is possible to acquire data including surgical images captured in the surgical process or various surgical information in the control process of the surgical robot. As described above, in the present invention, it is intended to provide a method for estimating information on an actual operation target or a reverse estimation of the actual operation process by using surgical information obtainable in the operation process.

In general, medical image data such as CT, PET, and MRI of the patient, virtual body models created based on such medical image data, or images of the patient's actual surgery scene contain personal information, so Sharing is not possible. As such, when data including personal medical information such as medical image data or virtual body model of the patient cannot be directly obtained from the patient, there is a problem in that it is not possible to reproduce the actual surgical procedure of the patient or provide a detailed surgical procedure visually. . In order to solve this problem, the present invention intends to provide a method that can reproduce the actual surgical procedure of a patient and implement a virtual body model by acquiring cue sheet data instead of medical image data or an image of an actual surgery scene.

Hereinafter, for convenience of description, it will be described that the "computer" performs the surgical information utilization method according to the embodiment disclosed herein. “Computer” may mean the server 100 or the controller 30 of FIG. 15 , but is not limited thereto and may be used to encompass a device capable of performing computing processing. In addition, the embodiments disclosed below are not applicable only in connection with the robotic surgery system shown in FIG. 15 , and may be applied to all kinds of embodiments that can obtain surgical information and utilize it.

16 is a flowchart illustrating a method of utilizing surgical information according to an embodiment of the present invention.

Referring to FIG. 16 , the method of utilizing surgical information performed by a computer according to an embodiment of the present invention includes acquiring cue sheet data including actual surgical information generated in an actual surgery process for a surgical subject (S100) , obtaining the body information of the surgery subject from the actual surgery information included in the cue sheet data (S110), and generating a virtual body model of the surgery subject based on the body information of the surgery subject (S120). . Hereinafter, detailed description of each step is described.

The computer may acquire cue sheet data including actual surgery information generated in the course of an actual surgery for the surgical subject (S100).

In an embodiment, medical staff may directly perform an actual operation on a subject for surgery, or may perform an actual operation using a surgical robot as described in FIG. 15 . At this time, various information (ie, actual surgical information) regarding the surgical operation performed in the actual surgical procedure or the surgical tool used in connection with the surgical operation, the surgical site, etc. may be recorded. Therefore, the computer acquires the actual surgical information generated from the actual surgical process of the surgical subject, and based on this You can configure cue sheet data. For example, the computer may configure cue sheet data by acquiring actual surgical information as image data of an actual surgical procedure or data recorded on a surgical operation performed in the actual surgical procedure. In addition, as another example, when it is not possible to export the actual surgery image of the surgery subject to the outside, the method of recording the operation-related information during the actual operation in the form of a message by communicating through the server may be utilized. When performing an actual operation, medical staff may transmit and receive information related to the operation of the subject to be operated on through a server, or generate surgery-related information using a simulator or artificial intelligence device and transmit/receive the generated information through the server. At this time, the server may record the transmitted and received surgery-related information in the form of a message. Therefore, the computer can obtain the information related to surgery recorded in the server and configure it as cue sheet data.

Here, the cue sheet data may be composed of data listing actual surgical procedures in order according to time based on the minimum surgical operation unit. In one embodiment, each cue sheet data may include actual surgical information corresponding to a minimum surgical operation unit, and the actual surgical information may include surgical tool information, body part information, and the like. Surgical tool information is information about surgical tools used during actual surgery, and may include, for example, information such as types of surgical tools, number of surgical tools, movement of surgical tools (eg, forward/retracted), and directions of surgical tools. can The body part information is information on the body part related to the operation of the surgical part or surgical tool, and may include, for example, information such as the type (name) of the body part and the location of the body part. In this case, the body part may be a part or all of the body, for example, may include at least one of organs such as liver, heart, uterus, brain, breast, abdomen, and blood vessels.

The computer may acquire the body information of the surgery subject from the actual surgery information included in the cue sheet data (S110).

In one embodiment, the computer may first acquire specific body part information related to the operation information of the surgical tool from the actual surgical information included in the cue sheet data. The cue sheet data includes a record of a surgical operation, which may include operation information performed on a specific body part (eg, a specific organ, blood vessel, etc.) using a specific surgical tool. For example, in the case of cue sheet data including a cutting operation, the type of the cutting surgical tool and the cutting target body part (eg, blood vessel, liver, fat, etc.) may be recorded as actual surgical information. In this case, the computer may detect cue sheet data associated with the operation information of the surgical tool from the cue sheet data, and obtain information on a specific body part associated with the operation of the surgical tool from the detected cue sheet data. Next, the computer may extract the type and spatial information of the specific body part from the acquired specific body part information. That is, information on a body part of a subject for surgery may be acquired.

In obtaining the body information of the surgery subject in step S110, cue sheet data related to the operation information of the surgical tool may be used as described above, but body information is obtained using cue sheet data including camera information or movement information of the surgical tool. may be obtained. For example, the computer may detect data of photographing a specific body part based on view information of a camera or position information of a camera included in the cue sheet data, and acquire specific body part information from the detected data. When the surgical tool is moved, there is no contact with the body part, so the body part does not exist in the cue sheet data related to the movement information of the surgical tool. In this case, the computer may acquire the body information of the surgery subject from the cue sheet data other than the cue sheet data related to the movement information of the surgical tool.

The computer may generate a virtual body model of the surgical subject based on the surgical subject's body information (S120).

Here, the virtual body model refers to three-dimensional modeling data generated to match the body of an actual surgery target.

In one embodiment, the computer reflects the body information of the surgery subject, that is, the type and spatial information of a specific body part, in the standard body model, and based on the standard body model, a virtual body matching the actual operation body of the surgery target. A body model can be created.

Here, the standard body model may be a three-dimensional body model generated by standardizing anatomical features of the body. For example, by standardizing anatomical features such as appearance, size, and location for each body part (eg, liver, heart, uterus, brain, organs such as breast, abdomen, and blood vessels), each body part is 3D It may be a body model constructed by modeling.

In an embodiment, in reflecting the body information of the surgical subject in the standard body model, the computer maps a specific body part on the space of the standard body model using the spatial information of the specific body part, and on the space of the mapped standard body model The corresponding body part of the standard body model can be modified to conform to the shape of a specific body part. For example, the computer may map spatial information (eg, coordinate information on a three-dimensional space) of a specific body part obtained from actual surgery information of cue sheet data with coordinate information on a three-dimensional space of a standard body model. At this time, the computer compares whether the corresponding body part located on the three-dimensional space of the mapped standard body model matches the specific body part obtained from the actual surgical information, and according to the comparison result, the specific body obtained from the actual surgical information The corresponding body part on the standard body model can be modified to match the shape of the part.

In addition, if information on a specific body part of a subject for surgery cannot be obtained from the cue sheet data, the computer acquires from the cue sheet data based on the surrounding body parts located in the vicinity of the specific body part that were not obtained from the cue sheet data in the space of the standard body model It is possible to interpolate specific body parts that are not. In general, since the position and shape of a person's organs are not significantly different (eg, the positions of organs such as the stomach and liver are the same), the entire virtual body model can be completed by performing interpolation using surrounding information.

In an embodiment, in generating a virtual body model based on a standard body model, the computer may change the body information of the surgery subject and repeatedly apply it to the standard body model to generate a virtual body model, in which case the actual operation of the surgery target You can select a virtual body model that best matches the body of the city. That is, in order to complete the virtual body model closest to the physical state of the subject for surgery, the above-described steps S110 to S120 may be repeatedly performed a plurality of times. For example, the computer may adopt the virtual body model with the least error by repeatedly performing steps S110 to S120 several times by changing the initial position of the surgery subject or changing the body shape of the surgery subject. In addition, the computer uses reinforcement learning to change the location information for the entire body part (that is, organ) of the subject to be operated on or to transform the shape of the organ, so that the virtual body model that best matches the physical condition of the surgical subject at the time of the actual operation. can be derived.

In an embodiment, the computer may match a specific organ point described in the cue sheet data with a specific point on the standard body model corresponding thereto, and may accumulate a matching relationship by performing this matching process on the entire cue sheet data. And the computer can implement a virtual body model based on the accumulated results.

For example, the computer may set the basic size of the standard body model by reflecting the BMI value of the patient (ie, the subject of surgery). As described above, the cue sheet data includes surgical tool information and body part information for all surgical operations. For example, it includes surgical tool information such as the type of surgical tool, location information of the surgical tool (eg, a location in three-dimensional space), and a surgical operation performed by the surgical tool. In addition, it includes body part information such as the type of body part related to the operation of the surgical tool (eg, the name of the body organ, the detailed part of the body organ, etc.) and the location of the body part. Here, the detailed parts of the body organs may be medically divided into front, back, upper, lower, right, and left surfaces.

Therefore, the position of the surgical tool (e.g., the position in three-dimensional space), the body organ at the position, and the detailed part of the body organ can be identified from the cue sheet data. Therefore, collecting all the cue sheet data of the patient during surgery information can be obtained. For example, the computer may obtain the position of each body organ that has been contacted by the operation of a surgical tool from the cue sheet data, and may express it as points in a three-dimensional space. At this time, since the points in the three-dimensional space are displayed in the form of a cloud (cloud), the expression of the position of each body organ in the three-dimensional space is referred to as a coordinate cloud. The computer can estimate the size of each body organ from the minimum or maximum coordinate values of the coordinate cloud, and can enlarge or reduce the corresponding body organ on the standard body model based on the estimated size of each body organ. In this way, the standard body model transformed based on the information of each body organ in the coordinate cloud is referred to as the modified standard body model. Based on the deformed standard body model, the computer matches the 3D coordinates of the detailed parts of the body organs touched by the surgical tool on the cue sheet data (e.g., the front, back, top, bottom, right, left side, etc. of the detailed parts) you can check if If it is found that they do not match, the computer can rotate the deformed standard body model or adjust it to match the detailed parts of the body organs on the cue sheet data through transformation such as enlargement or reduction of a specific part. Based on the modified standard body model that has undergone these adjustments, the computer can reconfirm whether the three-dimensional coordinates are consistent with the detailed parts of the body organs. That is, the computer can transform the standard body model to match the shape and size of the patient's actual body organs as much as possible by repeatedly applying the above process, and apply this to all body organs to derive the optimal modified standard body model. can The computer can finally create a virtual body model of the patient using the optimal deformed standard body model.

In another embodiment, the computer may use the learning model in the process of implementing the virtual body model. For example, the computer may perform learning using a standard body model and data of a patient who has both a virtual body model and cue sheet data. In addition, the computer can learn the process of deriving a virtual body model by applying cue sheet data to the standard body model.

According to an embodiment of the present invention, by implementing a virtual body model of a subject for surgery, the entire process during actual surgery can be accurately reproduced, and it can also be visually played back and utilized.

In addition, virtual surgery can be performed in a virtual space through a virtual body model. For example, if it is necessary to perform reoperation after performing the primary operation, the medical staff may perform a surgical simulation using a virtual body model in advance before the reoperation. At this time, since the virtual body model is generated based on cue sheet data after the primary operation of the patient is completed, simulation can be performed in advance using this to give the same effect as performing the actual reoperation. In addition, compared to using a generalized standard body model, using a virtual body model that reflects different characteristics for each patient can give the same effect as an actual surgery, so it has high utility value as a learning model.

17 is a flowchart illustrating a method of utilizing surgical information according to another embodiment of the present invention. In the method of FIG. 17, since the overlapping process with each step of FIG. 16 is the same or similar, a detailed description thereof will be omitted.

Referring to FIG. 17 , the computer may acquire cue sheet data including actual surgery information generated in the course of an actual surgery for a surgical subject ( S200 ).

Next, the computer may acquire image information about the actual surgical procedure of the surgery subject from the actual surgery information included in the cue sheet data (S210).

Here, the image information refers to information indicating a matching relationship with an image in response to a surgical operation. For example, the computer can obtain image information about the actual surgical process of the patient from the cue sheet data in the form of describing motion information performed on a specific body part (eg, a specific organ, blood vessel, etc.) using a specific surgical tool. Alternatively, it may be obtained as information indicating a matching relationship with a predetermined specific image corresponding to the corresponding surgical operation.

Next, the computer may generate surgical image data corresponding to the actual surgical procedure of the surgical subject based on the image information (S220).

The computer can visually reproduce the actual surgical procedure of the patient through the surgical image data. Therefore, although it is difficult for medical staff to understand the actual surgical process only with cue sheet data, it is easy to understand the entire actual surgical process by using the virtual body model and surgical image data of the surgery subject as described above.

In general, it is not possible to share medical images such as CT and PET of the subject of surgery or images of actual surgery scenes. Therefore, there is a problem in that it is difficult to utilize data for surgery in the prior art. However, according to the present invention, even if there is no image of a medical image or an actual surgery scene, if only cue sheet data is secured, the patient's virtual body model and actual surgery image data can be reproduced conversely, so more sufficient surgical data can be secured. there are advantages to

18 is a diagram schematically showing the configuration of an apparatus 300 for performing a surgical information utilization method according to an embodiment of the present invention.

Referring to FIG. 18 , the processor 310 includes one or more cores (not shown) and a graphic processing unit (not shown) and/or a connection path (eg, a bus) for transmitting and receiving signals with other components. ) may be included.

The processor 310 according to an embodiment executes one or more instructions stored in the memory 320, thereby performing the surgical information utilization method described with reference to FIGS. 16 and 17 .

For example, the processor 310 executes one or more instructions stored in the memory 320 to obtain cue sheet data including actual surgery information generated in an actual surgery process for a surgical subject, and the actual surgery included in the cue sheet data. It is possible to obtain the body information of the surgery target from the information, and create a virtual body model of the surgery target based on the body information of the surgery target.

In addition, the processor 310 obtains cue sheet data including actual surgery information generated in the course of an actual surgery for a surgical subject by executing one or more instructions stored in the memory 320, and from the actual surgery information included in the cue sheet data. It is possible to obtain image information on the actual surgical procedure of the subject to be operated on, and generate surgical image data corresponding to the actual surgical procedure of the subject for surgery based on the image information.

Meanwhile, the processor 310 temporarily and/or permanently stores a signal (or data) processed inside the processor 310 . , not shown) may be further included. In addition, the processor 310 may be implemented in the form of a system on chip (SoC) including at least one of a graphic processing unit, a RAM, and a ROM.

The memory 320 may store programs (one or more instructions) for processing and controlling the processor 310 . Programs stored in the memory 320 may be divided into a plurality of modules according to functions.

The surgical information utilization method according to an embodiment of the present invention described above may be implemented as a program (or application) to be executed in combination with a computer, which is hardware, and stored in a medium.

Hereinafter, various embodiments applied to the medical field are disclosed by generating learning data based on a medical image (eg, a surgical image) and performing learning using the learning data.

In one embodiment, there is provided a method for generating learning data based on a surgical image performed by a computer, the method comprising: obtaining a surgical image including a plurality of image frames; recognizing surgery recognition information from each of the plurality of image frames; and generating, for each of the plurality of image frames, Relational Representation information indicating a relationship between surgical elements included in the surgical recognition information based on the surgical recognition information. A detailed description thereof will be described later with reference to FIGS. 19 to 24 .

In another embodiment, there is provided a method for generating artificial data performed by a computer, the method comprising: obtaining a background image and an object image; and generating artificial data by combining the background image and the object image, wherein the background image includes a specific area inside the body photographed by an endoscope camera, and the object image is a surgical tool or blood It includes, wherein the artificial data is generated to approximate the actual surgical image based on the arrangement relationship between the surgical tool or blood in the object image, and a specific region inside the body in the background image. A detailed description thereof will be described later with reference to FIGS. 25 to 32 .

In another embodiment, in the relay server for constructing surgical information, but relaying communication between clients, receiving the actual surgery information generated based on the actual surgery image from the first client, and receiving the actual surgery information Transmits to a second client, receives virtual surgery information generated by performing a virtual surgery based on the actual surgery information from the second client, and transmits the virtual surgery information to the first client. A detailed description thereof will be described later with reference to FIGS. 33 to 36 .

A method of generating learning data based on a surgical image according to an embodiment of the present invention will be described in detail with reference to FIGS. 19 to 24 .

19 is a schematic diagram of a system capable of performing robotic surgery according to an embodiment of the present invention.

Referring to FIG. 19 , the robotic surgery system includes a medical imaging device 10 , a server 100 , and a controller 30 , a display 32 , and a surgical robot 34 provided in the operating room. According to an embodiment, the medical imaging device 10 may be omitted from the robotic surgery system according to the disclosed embodiment.

In one embodiment, the surgical robot 34 includes a photographing device 36 and a surgical tool 38 .

In one embodiment, the robotic surgery is performed by the user using the control unit 30 to control the surgical robot (34). In one embodiment, the robotic surgery may be performed automatically by the control unit 30 without the user's control.

The server 100 is a computing device including at least one processor and a communication unit.

The control unit 30 includes a computing device including at least one processor and a communication unit. In one embodiment, the controller 30 includes hardware and software interfaces for controlling the surgical robot 34 .

The photographing device 36 includes at least one image sensor. That is, the imaging device 36 includes at least one camera device, and is used to photograph an object, that is, a surgical site. In an embodiment, the imaging device 36 includes at least one camera coupled to a surgical arm of the surgical robot 34 .

In an embodiment, the image captured by the photographing device 36 is displayed on the display 340 .

In one embodiment, the surgical robot 34 includes one or more surgical tools 38 capable of performing cutting, clipping, fixing, grabbing operations of the surgical site, and the like. The surgical tool 38 is used in combination with the surgical arm of the surgical robot 34 .

The controller 30 receives information necessary for surgery from the server 100 or generates information necessary for surgery and provides it to the user. For example, the controller 30 displays generated or received information necessary for surgery on the display 32 .

For example, the user performs robotic surgery by controlling the movement of the surgical robot 34 by manipulating the controller 30 while viewing the display 32 .

The server 100 generates information necessary for robotic surgery by using the medical image data of the object photographed in advance from the medical imaging device 10 , and provides the generated information to the controller 30 .

The controller 30 provides the information received from the server 100 to the user by displaying it on the display 32 , or controls the surgical robot 34 using the information received from the server 100 .

In one embodiment, the means that can be used in the medical imaging equipment 10 is not limited, for example, CT, X-Ray, PET, MRI, etc., various other medical image acquisition means may be used.

As described above, when performing robotic surgery, it is possible to acquire data including surgical images captured in the surgical process or various surgical information in the control process of the surgical robot. Therefore, in the present invention, a learning model capable of inferring various information related to surgery by enabling the surgical image or surgical data that can be obtained in the surgical procedure to be used as a learning data, and by performing learning based on the learning data We want to provide you with a way to build it.

Hereinafter, for convenience of description, it will be described that a "computer" performs a method of generating learning data based on a surgical image according to an embodiment disclosed herein. “Computer” may mean the server 100 or the controller 30 of FIG. 19 , but is not limited thereto and may be used to encompass a device capable of performing computing processing. For example, the computer may be a computing device provided separately from the device shown in FIG. 19 .

In addition, the embodiments disclosed below may not be applied only in connection with the robotic surgery system shown in FIG. 19, but may be applied to all kinds of embodiments in which a surgical image may be acquired and utilized in the surgical process. For example, in addition to robotic surgery, it may be applied in connection with minimally invasive surgery such as laparoscopic surgery or surgery using an endoscope.

20 is a flowchart schematically illustrating a method of generating learning data based on a surgical image according to an embodiment of the present invention.

Referring to FIG. 20 , the method for generating learning data based on a surgical image according to an embodiment of the present invention includes: acquiring a surgical image including a plurality of image frames (S100), each of the plurality of image frames Recognizing the surgical recognition information from (S200), and for each of the plurality of image frames, a relational expression representing a relationship between surgical elements included in the surgical recognition information based on the surgical recognition information. Representation) generating information (S300) may be included. Hereinafter, detailed description of each step is described.

The computer may acquire a surgical image including a plurality of image frames (S100).

When a specific surgery (eg, gastric cancer surgery, colon cancer surgery, etc.) is performed on a patient, medical staff may directly perform an actual operation on the patient, and use a laparoscopy or an endoscope as well as a surgical robot as described in FIG. 19 . Minimally invasive surgery can also be performed.

In this case, the computer may acquire a surgical image obtained by photographing a scene including a surgical operation performed in the surgical procedure and related surgical tools, surgical sites, and the like. In an embodiment, the computer may acquire a surgical image that captures a scene including a surgical site and a surgical tool currently being operated from a camera that has entered the patient's body.

The surgical image may include one or more image frames. Each image frame may represent one scene in which a surgical operation is performed, including a surgical site of a subject to be operated, a surgical tool, and the like. For example, the surgical image may be composed of image frames in which a surgical operation according to time is recorded for each scene in a surgical procedure. Alternatively, the surgical image may be composed of image frames in which each surgical scene is recorded according to spatial movement, such as a position of a surgical site or a camera, during a surgical procedure.

In addition, the surgical image may be composed of all image frames including the entire surgical procedure, but may also be formed in the form of at least one video clip divided according to a specific classification criterion.

According to an embodiment, the computer may acquire a surgical image in the form of a video clip, acquire a surgical image including all or part of a surgical procedure, and divide it into at least one surgical image in the form of a video clip.

In an embodiment, the computer may acquire a surgical image including all or part of a surgical procedure, and may divide the surgical image into at least one video clip type surgical image according to a specific classification criterion. For example, the computer may divide the surgical image according to the lapse of time during which the operation is performed, or the surgical image may be divided according to the position or state change of the surgical site based on the surgical site during surgery. Alternatively, the computer may divide the surgical image based on the position of the camera or the movement range of the camera during the operation, and divide the surgical image based on the change (eg, replacement, etc.) of the surgical tool during the operation. You may. Alternatively, the computer may divide the surgical image based on image frames made up of a series of surgical operations in relation to the surgical operation. In addition, for each operation, a surgical stage classified according to a specific classification criterion may be predetermined. In this case, the computer may divide the surgical image based on the surgical stage.

The computer may recognize the surgical recognition information from each of a plurality of image frames in the acquired surgical image (S200).

As described above, each image frame in the surgical image includes surgery-related information. For example, it includes various types of information related to surgery, such as surgical tools, surgical operations, surgical sites (ie, body parts such as organs, blood vessels, and tissues), and whether or not bleeding occurs. Therefore, in the present invention, each element that can be recognized as surgery-related information from an image frame is defined as a surgical element. The surgical element may be a concept corresponding to an object, but in the present invention, it refers to a broad concept encompassing not only the object but also information such as motion, function, and state related to the object.

In one embodiment, the computer may extract each of the surgical recognition information determined as a surgical element from each of the plurality of image frames. Here, the surgical recognition information refers to information on the surgical elements recognized from the image frame, for example, surgical tools, surgical motions, body parts, bleeding status, surgical steps, surgery time (eg, remaining surgery time, surgery time required, etc.), and a camera. Information (eg, position, angle, direction, movement, etc. of the camera) may include at least one surgical element.

In addition, the computer may derive the position information of the surgical element in the surgical recognition information extracted from each of the plurality of image frames. The position information of the surgical element may be information on the area where the surgical element is located on the image frame. For example, it is possible to calculate the position information of the surgical element based on the coordinate information on the two-dimensional space in the image frame or the coordinate information on the three-dimensional space.

The computer may generate Relational Representation information indicating the relationship between the surgical elements included in each surgical recognition information based on the recognized surgical recognition information for each of the plurality of image frames (S300).

In one embodiment, the computer uses a plurality of image frames (eg, first to n-th image frames) based on the plurality of surgical elements extracted from the first image frame and positional information on the plurality of surgical elements, the first image It can be determined whether there is a correlation between a plurality of surgical elements within the frame. Next, the computer may generate relational expression information for the first image frame based on the correlation between the plurality of surgical elements. The computer generates relational expression information not only for the first image frame, but also for the second to n-th image frames in the surgical image in the same manner.

In determining the correlation between the plurality of surgical elements, the computer may determine whether a correlation between the plurality of surgical elements exists based on relationship information on the predefined surgical elements. Here, the predefined relational information on the surgical element may be information set based on at least one of the type of the surgical element, the position information of the surgical element, the state information of the surgical element, and the operation information of the surgical element.

As described above, a surgical tool, a surgical operation, a body part, whether bleeding is present, an operation stage, an operation time, camera information, and the like can be defined as surgical elements. Each defined surgical element may further include additional information such as location information, state information, and operation information according to the type of the surgical element. For example, the type of surgical tool may be determined according to the surgical site, and operation state information such as open/close, energy presence, contact, etc. may be determined according to each surgical tool. have. Bleeding status may have color information based on the bleeding point. That is, the computer may preset each surgical element and additional information determined according to the type of each surgical element as relational information. Accordingly, the computer may recognize each surgical element from the first image frame, and grasp relationship information for each recognized surgical element.

For example, when the computer extracts the surgical tool and the surgical site from the first image frame, the computer recognizes the relationship information on the specific surgical tool and the corresponding specific surgical site based on the predefined relationship information to determine the correlation between the two surgical elements. can be considered to exist.

Alternatively, when the computer extracts the surgical tool from the first image frame, it is determined that the correlation between the surgical tool and the surgical operation exists by identifying the relationship information on the specific surgical tool and the corresponding operation based on the predefined relationship information. can judge

In addition, when a plurality of surgical elements having the same or similar position information are extracted from the first image frame, the computer may determine that a correlation between the plurality of surgical elements exists. For example, when the computer extracts a body organ and a surgical tool from the first image frame and recognizes that the extracted body organ and the surgical tool exist at the same location, it is determined that a positional correlation between the two surgical elements exists. can judge The computer may determine that a positional correlation exists when the surgical tool is applying a specific motion to the body organ, or when the surgical tool and the body organ are in contact with each other.

The computer may generate relational expression information for the first image frame based on whether a correlation between the plurality of surgical elements recognized from the first image frame exists.

In an embodiment, the computer may generate relational expression information by mapping each surgical element and information on each surgical element. For example, relational expression information may be generated in the form of a matrix. Each surgical element is arranged to correspond to a row and column, but information derived based on the correlation between each surgical element may be represented as an entry of a matrix.

The computer may generate relational expression information for each of the second to nth image frames in the surgical image in the same manner as the first image frame as described above.

21 is a diagram illustrating a process of generating relational expression information by recognizing surgical recognition information from an image frame according to an embodiment of the present invention. As an example, FIG. 21 shows a process of recognizing surgery recognition information from a first image frame 200 in a surgical image, and generating it as relational expression information 300 .

In one embodiment, the computer may recognize each surgical element by applying an image recognition technology using deep learning from the first image frame 200 . In recognizing each surgical element, the computer may apply various individual recognition techniques according to the characteristics of each surgical element. For example, the computer extracts feature information (eg, a feature map) from the first image frame 200, and uses the texture feature information to express color or texture to a body part (eg, Organs such as liver, stomach, blood vessels) or bleeding can be recognized. Alternatively, the computer may recognize a surgical tool or a surgical operation using characteristic information about the shape. Alternatively, the computer may recognize the surgical operation by using the positional characteristic information between the surgical elements. Alternatively, the computer may recognize camera information (eg, movement information such as angle and movement of the camera), operation steps, operation operations, operation time, and the like by using characteristic information between a plurality of image frames.

As described above, as a result of the computer recognizing the surgical recognition information for the first image frame 200 based on the characteristic information of each surgical element, as shown in FIG. Operational state, gauze, and surgical site were recognized as surgical elements.

Also, the computer may derive position information for each recognized surgical element from the first image frame 200 . In one embodiment, the computer may divide the first image frame 200 into at least one region, and calculate the position information of each surgical element based on the region in which each surgical element exists among the divided regions. In this case, the computer may divide the space in the first image frame 200 into at least one region based on two-dimensional or three-dimensional coordinate information, and a specific coordinate value (eg, the center point of each region) for each divided region. ) or by assigning an index value, it can be used as location information of each area.

For example, as shown in FIG. 21 , the computer divides the first image frame 200 into nine regions, and assigns index values 0 to 8 to each divided region to be used as location information. . The computer determines that the surgical element 'gauze' recognized from the first image frame 200 is located in the index 0, 1 area among the divided areas, and then, based on the index 0, 1 position information on the surgical element 'gauze' can be calculated. For other surgical elements recognized from the first image frame 200 (eg, two surgical tools, a surgical site), position information of each surgical element may be calculated in the same manner as above.

The computer may generate the relational expression information 300 based on the position information of each surgical element and each surgical element recognized from the first image frame 200 . In one embodiment, the computer may configure the relational expression information 300 in the form of a matrix. Each surgical element may be arranged in each row and each column, and relationship information between the corresponding surgical elements may be indicated in each component value of the matrix.

For example, as shown in FIG. 21 , the relational expression information 300 of the first image frame 200 may be expressed as a matrix. As an example, each row and each column may be arranged from the surgical element a to the surgical element s. At this time, surgical element a is the surgical stage, surgical elements b to h are surgical tools, surgical elements i to l are body parts, surgical elements m to q are surgical operations, surgical element r is bleeding, and surgical element s is camera information, etc. can be defined as In addition, the i-th column of the i-th row of the matrix may indicate position information of the corresponding surgical element (eg, a surgical element, b surgical element, etc.). The j-th column of the i-th row of the matrix may represent relationship information between the surgical element arranged in the i-th row and the surgical element arranged in the j-th column. For example, in the row corresponding to the surgical element b of the matrix and the column corresponding to the surgical element m, a specific surgical tool (surgical tool defined as surgical element b, such as 'Harmonics') and a specific surgical operation (defined as surgical element m) Information on the surgical operation performed, eg, open/grap) can be displayed. Depending on the surgical element, other specific values may be assigned to matrix components to indicate relational information. For example, if an event related to a surgical element occurs or contains such an event throughout the image frame, such as camera information or a surgical stage, the matrix component for the surgical element contains the position information or relationship information of the surgical element instead of You can also provide information related to the event.

The computer may repeatedly perform the same process as described with reference to FIG. 21 for each image frame in the surgical image. That is, the computer may recognize the surgical element for each image frame in the surgical image, and generate relational expression information based on the recognized surgical element.

22 is a diagram illustrating relational expression information generated for a plurality of image frames in a surgical image according to an embodiment of the present invention.

Referring to FIG. 22 , the computer may generate relational expression information for each of a plurality of image frames in a surgical image. In this case, since each relational expression information simultaneously expresses relational information between a plurality of surgical elements, it may have tensor information. As an embodiment, the computer may generate each relational expression information generated from a plurality of image frames in the form of a tensor. Also, the computer may express a relationship between the plurality of image frames by grouping the plurality of image frames based on the temporal size of the tensor. For example, when relational expression information needs to be generated based on a plurality of image frames, such as camera information or a surgical stage, the computer groups a specific number of image frames to generate each relational expression information in a tensor form, and camera information Alternatively, information related to the operation stage may be expressed as a tensor by correlating it with a relationship between image frames.

According to an embodiment of the present invention, as described above, relational expression information is generated for each image frame in the surgical image, and learning data can be constructed based on each of the generated relational expression information. In addition, learning may be performed to derive a specific learning result based on the learning data constructed in this way.

23 is a diagram illustrating an example of a process of learning based on relational expression information generated for a plurality of image frames in a surgical image according to an embodiment of the present invention.

Referring to FIG. 23 , the computer may generate relational expression information for each image frame in the surgical image, and build learning data based on the generated relational expression information (S400).

In one embodiment, in constructing learning data by generating relational expression information for each image frame in a surgical image, labeling learning may be performed by a medical staff, or labeling learning may be performed automatically by a computer. In addition, when learning is automatically performed by a computer, semi-supervised learning may be performed. For example, the computer applies various individual recognition technologies (eg, body organ recognition, surgical tool recognition, bleeding recognition, surgical motion recognition, camera movement recognition, etc.) according to the characteristics of each surgical element as described above to label each image frame. information can be extracted and relational expression information can be created based on it.

The computer may derive a specific learning result by performing learning based on the learning data (S500).

Here, the specific learning result refers to a learning result that can be obtained as an output value of the specific learning model by building a specific learning model by performing learning using the learning data generated according to an embodiment of the present invention as an input value. say

For example, the computer may determine as a learning result to derive relational expression information according to an embodiment of the present invention. That is, the computer may perform learning on a newly input image frame based on pre-established learning data (relational expression information), and obtain relational expression information on the newly input image frame as a result of the learning.

Alternatively, the computer performs learning by using the learning data generated according to the embodiment of the present invention as an input value in order to obtain various learning results, such as recognizing the surgical process through the surgical image or recognizing the meaning of the surgical operation. can

According to an embodiment, the computer may use the image frames together with the learning data generated according to the embodiment of the present invention as input values for deriving a specific learning result. In one embodiment, the computer may perform learning using a convolutional neural network (CNN) on a plurality of image frames in the surgical image, and obtain feature information for each image frame (S410). In addition, the computer may acquire learning data by generating relational expression information for a plurality of image frames in the surgical image (S400). The computer finally performs learning based on the characteristic information and relational expression information for each image frame, and can obtain a specific output value as a result of the learning (S500).

According to the present invention, more meaningful information can be obtained through an image frame by defining various surgical elements that can be recognized from an image frame, and identifying information about each surgical element itself as well as relationship information between the surgical elements. .

In addition, according to the present invention, it does not generate learning data for building one learning model, but provides base data for building various learning models. In addition, by providing such data for base learning, the learning data is increased, so that more improved learning results can be obtained.

In addition, according to the present invention, various information necessary for surgery can be flexibly defined according to the learning purpose through relational expression information, and thus effective learning can be performed.

24 is a diagram schematically showing the configuration of an apparatus 400 for performing a method of generating learning data based on a surgical image according to an embodiment of the present invention.

Referring to FIG. 24 , the processor 410 includes one or more cores (core, not shown) and a graphic processing unit (not shown) and/or a connection path (eg, a bus) for transmitting and receiving signals with other components. ) may be included.

The processor 410 according to an embodiment performs a method of generating learning data based on the surgical image described in relation to FIGS. 20 to 23 by executing one or more instructions stored in the memory 420 .

In one example, the processor 410 acquires a surgical image including a plurality of image frames by executing one or more instructions stored in the memory 420, recognizing the operation recognition information from each of the plurality of image frames, and For each of the plurality of image frames, a step of generating Relational Representation information indicating a relationship between surgical elements included in the surgical recognition information based on the surgical recognition information may be performed.

Meanwhile, the processor 410 includes a random access memory (RAM) and a read-only memory (ROM) that temporarily and/or permanently store a signal (or data) processed inside the processor 410 . , not shown) may be further included. Also, the processor 210 may be implemented in the form of a system on chip (SoC) including at least one of a graphic processing unit, a RAM, and a ROM.

The memory 420 may store programs (one or more instructions) for processing and controlling the processor 410 . Programs stored in the memory 420 may be divided into a plurality of modules according to functions.

The method for generating learning data based on the surgical image according to an embodiment of the present invention described above may be implemented as a program (or application) to be executed in combination with a computer, which is hardware, and stored in a medium.

A method of generating artificial data according to an embodiment of the present invention will be described in detail with reference to FIGS. 25 to 32 .

25 is a flowchart illustrating a method of generating artificial data according to an embodiment of the present invention.

Although the method of FIG. 25 is described as being performed by a computer for convenience of description, the subject performing each step is not limited to a specific device and may be used to encompass devices capable of performing computing processing. That is, in the present embodiment, the computer may refer to a device capable of performing the artificial data generation method according to the embodiment of the present invention.

Referring to FIG. 25 , the method for generating artificial data according to an embodiment of the present invention includes the steps of obtaining a background image and an object image (S100), and generating artificial data by combining the background image and the object image (S110). ) may be included. Hereinafter, detailed description of each step is described.

The computer may acquire a background image and an object image (S100).

The background image and the object image may be a part or the entire image area included in the actual surgical image. The actual surgical image may be data obtained as an actual medical staff performs an operation. For example, medical staff may perform surgery directly on a patient, or may perform minimally invasive surgery using a surgical robot, a laparoscope, an endoscope, or the like. At this time, it is possible to obtain a variety of information about the surgical operation performed in the surgical process, surgical tools related to the surgical operation, surgical site, and the like. For example, an actual surgical image obtained by photographing organs, surgical instruments, etc. inside the body with a focus on the surgical site in the surgical process may be acquired, or data recorded on the surgical operation performed in the surgical process may be acquired.

In one embodiment, the background image may include a specific area inside the body photographed by the endoscope camera. For example, it may be image data obtained by photographing a specific region including organs (eg, liver, heart, uterus, brain, breast, abdomen, etc.), blood vessels, and tissues inside the body.

As an embodiment, the object image may be image data including a surgical tool or blood (including bleeding in which blood comes out of a blood vessel due to blood vessel damage).

According to an embodiment, the computer may acquire an object image or a background image from an actual surgical image including an object and a background (ie, a background including a specific area inside the body). Alternatively, the computer may acquire an object image from an actual surgical image containing only an object, or may acquire a background image from an actual surgical image containing only a background. Thereafter, the computer may generate various artificial data using an object image or a background image obtained from an actual surgical image. A specific embodiment thereof will be described with reference to FIGS. 27 to 30 .

Also, according to an embodiment, the computer may acquire the actual surgical image in units of frames, or may acquire it in units of sequences, which is a set of consecutive frames. For example, when generating artificial data for an object including a surgical tool, the computer acquires an actual surgical image in units of one frame and extracts an object image including a surgical tool from it to generate artificial data. have. Alternatively, when generating artificial data for an object containing blood or bleeding, the computer acquires an actual surgical image in a sequence unit consisting of a plurality of frames, and extracts an object image containing blood or bleeding from the artificial data. can create In this case, the computer inputs the actual surgical image including the object image in sequence units (ie, consecutive frames) to the generator to be described later, and accordingly, the generator generates artificial data in sequence units (ie, consecutive frames). You can create and print it. In the case of an object such as blood or bleeding, it is difficult to grasp the meaning of the object by looking at only one frame image. Accordingly, in the present invention, artificial data may be generated in units of frames or sequences according to types of objects. In addition, according to the present invention, even when artificial data corresponding to a specific surgical operation (eg, cutting, grabbing, etc.) is generated, an object image is acquired in sequence units in the same way as when artificial data including blood or bleeding is generated. Artificial data can be created.

Also, in the present invention, artificial data may be generated by acquiring an actual surgical image in a frame unit or a sequence unit as an object changes. In one embodiment, when the computer acquires an actual surgical image in which a change occurs in a specific object, for example, a real surgical image performing a specific surgical operation according to the movement of the surgical tool or the movement of the surgical tool is acquired in successive frames. In this case, the computer may continuously extract object images including a specific object (eg, surgical tool) from successive frames to generate artificial data reflecting the change in movement of the specific object (eg, surgical tool). In this case, the artificial data may be generated as continuous frames according to the change of a specific object. In addition, the computer may extract a background image from the actual surgical image in response to a change in movement of a specific object (eg, surgical tool). For example, the computer continuously extracts the extracted object image and the background image according to the change in the movement of a specific object (eg, surgical tool), and sequentially combines the continuous object image and the background image to generate continuous artificial data. You may. Accordingly, artificial data closer to reality can be generated by extracting and reflecting the background image in response to the change of the object.

The computer may generate artificial data by combining the background image and the object image (S110).

In one embodiment, the computer is based on a disposition relationship between an object in the object image (eg, surgical tool, blood, bleeding, etc.) and a specific area inside the body in the background image (eg, internal organs, blood vessels, tissues, etc.) Thus, artificial data can be generated by combining an object image and a background image to approximate the actual surgical image.

The arrangement relationship may be information derived based on information such as a form, location, direction or camera shooting direction in which the background and objects in each image are arranged, and the type and number of backgrounds and objects in each image.

In generating artificial data, a computer may generate artificial data through learning using a generator and a discriminator. This will be described in detail with reference to FIG. 26 .

26 is a diagram for explaining a method of generating artificial data based on learning according to an embodiment of the present invention.

In one embodiment, in generating artificial data by combining the object and the background based on the arrangement relationship between the object in the object image and the background in the background image, the computer allows the artificial data to be generated closer to the actual surgical image. learning can be performed. In this case, the computer may perform learning using the generator 200 and the discriminator 210 . For example, the generator 200 and the discriminator 210 may operate through competition with each other using a generative adversarial network (GAN)-based learning method.

Referring to FIG. 26 , the generator 200 learns a generative model using an actual surgical image, and generates input data (actual surgical image) 220 based on the learned generative model as artificial data close to reality. It can be created with (230).

In one embodiment, the generator 200 may generate artificial data by combining a background image and an object image obtained from each actual surgical image using a generative model.

The discriminator 210 trains a discriminative model to discriminate whether the artificial data 230 generated by the generator 200 is real data or artificial data. In this case, the discriminator 210 may train the discrimination model to discriminate whether the artificial data 230 is authentic or not based on the actual surgical image 240 corresponding to the artificial data. The discriminator 210 may re-train the generative model based on the learned discrimination model.

In an embodiment, the discriminator 210 may determine whether the artificial data 230 generated by the generator 200 is real data or artificial data by using the discrimination model. When it is determined that the discriminator 210 is artificial data rather than real, that is, when the generator 200 fails to deceive the discriminator 210 , the generator 200 fails to deceive the discriminator 210 . The generative model can be retrained in the direction of reducing . The generator 200 may generate improved artificial data through re-learning of the generative model. Conversely, when the discriminator 210 determines that it is real data, that is, when the discriminator 210 is deceived by the generator 200, the discriminator 210 selects the discrimination model in the direction of reducing the incorrect answer rate (error). can be re-learned. As this process is repeated in the generator 200 and the discriminator 210 , artificial data close to reality can be created.

Therefore, when artificial data approximating an actual surgical image is generated through mutual learning between the generator 200 and the discriminator 210, the computer can configure a learning data set for learning a learning model for surgical image recognition based on this. .

Surgical images can be acquired only through the actual surgical procedure, and due to their characteristics, they often contain only limited scenes. In actual surgery, surgery is performed on a specific surgical site, and since the operation is performed using a surgical tool determined according to the surgical site or purpose of surgery, it is difficult to secure surgical images including various surgical tools or surgical sites. . Therefore, when learning is performed using these surgical images, it is difficult to construct sufficient learning data using only the actual surgical images obtained during the actual surgery. Therefore, in the present invention, by generating artificial data with little difference from the actual surgical image, it can be provided as learning data of a learning model that performs learning using the surgical image.

27 to 30 are diagrams for explaining embodiments of a method for generating artificial data based on learning according to an embodiment of the present invention.

Various embodiments in which a computer generates artificial data by acquiring a background image and an object image will be described with reference to FIGS. 27 to 30 .

27 is a first embodiment of generating artificial data using first and second actual surgical images including an object and a background.

Referring to FIG. 27 , the computer may acquire a first real surgical image 300 including an object and a background, and a second real surgical image 310 including an object and a background. At this time, the computer may extract an object image or a background image included in each of the first real surgical image 300 and the second real surgical image 310, and combine them to generate artificial data.

In one embodiment, the computer may extract the background image in the first actual surgical image 300, and extract the object image in the second actual surgery image (310). The computer may generate the artificial data 330 close to the actual surgical image by using the extracted background image and the object image as a generation model (ie, the generator) 320 . For example, the computer uses the background in the background image (ie, organs, blood vessels, tissues, etc. existing in a specific area inside the body) and the object in the object image (ie, surgical tools) based on the learning in the generative model 320 . , blood, bleeding, etc.) can be combined with the background image and the object image based on the arrangement relationship between them. In this case, the computer may provide the object segmentation 311 in the second actual surgery image 310 together with the second actual surgery image 310 as a correct answer to the generation model 320 . Since the generation model 320 is learned through the object segmentation 311 that is the correct answer, it is possible to generate artificial data closer to the actual surgical image.

The computer may provide the artificial data 330 generated by the generation model 320 to the discrimination model (ie, the discriminator) 350 . The differential model 350 may determine the authenticity of the artificial data 330 based on the actual surgical image 340 corresponding to the artificial data 330 . In this case, the computer may additionally provide the object segmentation 331 in the artificial data 330 and the object segmentation 341 in the actual surgical image 340 for learning the differential model 350 .

The computer may retrain the generative model 320 or the differential model 350 according to the determination result of the differential model 350 . For example, when it is determined that the artificial data 330 is not real but artificial data as a result of the determination of the differential model 350 , the computer may re-train the generative model 320 . Conversely, when it is determined that the artificial data 330 is real data, the computer may train the discrimination model 350 again.

In another embodiment, the computer may generate artificial data by extracting the object image in the first real surgical image 300 and extracting the background image in the second real surgical image 310 . Since a detailed process for this is similar to that described with reference to FIG. 3 , a description thereof will be omitted in this embodiment.

According to the above-described embodiment, artificial data is generated by using actual surgical images including different objects, which are operated on rare objects (eg, surgical tools, blood or bleeding) that do not appear well during surgery. Images can be accumulated. In addition, when manually labeling is performed, expensive segmentation data can be replaced with artificial data, thereby saving costs.

28 is a second embodiment of generating artificial data using a first real surgical image including only an object and a second real surgical image including both an object and a background.

Referring to FIG. 28 , the computer may acquire a first actual surgery image 400 including only an object, and a second actual operation image 410 including both an object and a background. In this case, the computer may extract an object image from the first actual surgical image 400 , and extract a background image from the second actual surgical image 410 . The computer may generate the artificial data 430 close to the actual surgical image by using the extracted background image and the object image by using the generation model (ie, the generator) 420 .

For example, the computer uses the background in the background image (ie, organs, blood vessels, tissues, etc. existing in a specific area inside the body) and the object in the object image (ie, surgical tools) based on the learning in the generative model 420 . , blood, bleeding, etc.) can be combined with the background image and the object image based on the arrangement relationship between them.

The computer may provide the artificial data 430 generated by the generation model 420 to the discrimination model (ie, the discriminator) 450 . The differential model 450 may determine the authenticity of the artificial data 430 based on the actual surgical image 440 corresponding to the artificial data 430 . In this case, the computer may additionally provide the object segmentation 431 in the artificial data 430 and the object segmentation 441 in the actual surgical image 440 for learning the differential model 450 .

The computer may retrain the generative model 420 or the differential model 450 according to the determination result of the differential model 450 . For example, when it is determined that the artificial data 430 is not real but artificial data as a result of the determination of the differential model 450 , the computer may re-train the generative model 320 . Conversely, when it is determined that the artificial data 430 is real data, the computer may train the discrimination model 450 again.

According to the above-described embodiment, flexibility for artificial data generation can be increased, and surgical images for various objects can be accumulated. For example, it is possible to secure images of various types of surgical tools, surgical images in which positions, directions, and arrangements of surgical tools are set differently.

29 is a third embodiment of generating artificial data using a first real surgical image including only a background and a second real surgical image including both an object and a background.

Referring to FIG. 29 , the computer may acquire a first actual surgical image 500 including only a background and a second actual surgical image 510 including both an object and a background. In this case, the computer may extract a background image from the first actual surgical image 500 and extract an object image from the second actual surgical image 510 . The computer may generate the artificial data 530 close to the actual surgical image by using the extracted background image and the object image as a generating model (ie, the generator) 520 .

For example, the computer uses the background in the background image (ie, organs, blood vessels, tissues, etc. existing in a specific area inside the body) and the object in the object image (ie, surgical tools) based on the learning in the generative model 520 . , blood, bleeding, etc.) can be combined with the background image and the object image based on the arrangement relationship between them.

In this case, the computer may provide the object segmentation 511 in the second actual surgery image 510 together with the second actual surgery image 510 as a correct answer to the generation model 520 . Since the generation model 520 is learned through the object segmentation 511 that is the correct answer, it is possible to generate artificial data closer to the actual surgical image.

The computer may provide the artificial data 530 generated by the generation model 520 to the discrimination model (ie, the discriminator) 550 . The differential model 550 may determine the authenticity of the artificial data 530 based on the actual surgical image 540 corresponding to the artificial data 530 . In this case, the computer may additionally provide the object segmentation 531 in the artificial data 530 and the object segmentation 541 in the actual surgical image 540 for learning the differential model 550 .

The computer may retrain the generative model 520 or the differential model 550 according to the determination result of the differential model 550 . For example, when it is determined that the artificial data 530 is not real but artificial data as a result of the determination of the differential model 550 , the computer may re-train the generative model 520 . Conversely, when it is determined that the artificial data 530 is real data, the computer may train the discrimination model 550 again.

According to the above-described embodiment, flexibility for artificial data generation can be increased, and images of various types of internal organs, blood vessels, tissues, etc. operated on a specific object (eg, surgical tool) can be secured. For example, it is possible to secure a surgical image in which different body parts are operated for one surgical tool.

30 is a fourth embodiment of generating artificial data using a first real surgical image including only a background and a second real surgical image including only an object.

Referring to FIG. 30 , the computer may acquire a first real surgical image 600 including only a background and a second real surgical image 610 including only an object. In this case, the computer may extract a background image from the first actual surgical image 600 , and extract an object image from the second actual surgical image 610 . The computer may generate the artificial data 630 close to the actual surgical image by using the extracted background image and the object image as a generating model (ie, the generator) 620 .

For example, the computer uses the background in the background image (ie, organs, blood vessels, tissues, etc. existing in a specific area inside the body) and the object in the object image (ie, surgical tools) based on the learning in the generative model 620 . , blood, bleeding, etc.) can be combined with the background image and the object image based on the arrangement relationship between them.

The computer may provide the artificial data 630 generated by the generation model 620 to the discrimination model (ie, the discriminator) 650 . The differential model 650 may determine the authenticity of the artificial data 630 based on the actual surgical image 640 corresponding to the artificial data 630 . In this case, the computer may additionally provide the object segmentation 631 in the artificial data 630 and the object segmentation 641 in the actual surgical image 640 for learning the differential model 650 .

The computer may retrain the generative model 620 or the differential model 650 according to the determination result of the differential model 650 . For example, when it is determined that the artificial data 630 is not real but artificial data as a result of the determination of the differential model 650 , the computer may re-train the generation model 620 . Conversely, when it is determined that the artificial data 630 is real data, the computer may re-train the discrimination model 650 .

According to the above-described embodiment, it is possible to acquire a surgical image including various objects and various internal body spaces. For example, various types of surgical images can be obtained by freely arranging necessary surgical tools or information on necessary internal organs, blood vessels, tissues, and the like.

In the embodiment of Figures 29 and 30 described above, artificial data is generated by acquiring an actual surgical image (background image) including only the background. At this time, the actual surgical image including only the background does not include objects such as surgical tools, and only includes a background image. In this case, the computer may generate a background image including only the background area by acquiring image frames photographed at a specific location inside the body by the endoscope camera, and removing object images such as surgical tools from the image frames. Alternatively, the computer may use the image frame acquired by the endoscope entering the body during surgery using the endoscope. In this case, it is possible to obtain an image frame having only a background image that does not include an object such as a surgical tool. Alternatively, the computer may extract an image frame including only a background from an image frame acquired by a camera that enters the body during minimally invasive surgery such as laparoscopy or robotic surgery. In the case of minimally invasive surgery, the camera and surgical tools are also moved from the first area to the second area when the surgical operation in the first area is completed and the operation is performed by moving to the second area. At this time, there is a restriction that the surgical tool does not move when the camera moves due to the nature of the operation. Therefore, when moving from the first area to the second area, the camera moves first and then the surgical tool moves, so that the camera moving to the second area first can acquire an image frame without the surgical tool.

The computer may construct the learning data by using the artificial data generated from the above-described embodiments. For example, artificial data can be used as learning data for learning a learning model for image recognition. Therefore, according to an embodiment of the present invention, by providing artificially generated surgical images together with actual surgical images to artificial intelligence, deep learning, machine learning, etc. that require a large amount of learning data, sufficient learning effect can be obtained.

31 is a schematic diagram of a system capable of performing robotic surgery according to an embodiment of the present invention.

Referring to FIG. 31 , the robotic surgery system includes a medical imaging device 10 , a server 100 , and a controller 30 , a display 32 , and a surgical robot 34 provided in the operating room. According to an embodiment, the medical imaging device 10 may be omitted from the robotic surgery system according to the disclosed embodiment.

In one embodiment, the surgical robot 34 includes a photographing device 36 and a surgical tool 38 .

In one embodiment, the robotic surgery is performed by the user using the control unit 30 to control the surgical robot (34). In one embodiment, the robotic surgery may be performed automatically by the control unit 30 without the user's control.

The server 100 is a computing device including at least one processor and a communication unit.

The control unit 30 includes a computing device including at least one processor and a communication unit. In one embodiment, the controller 30 includes hardware and software interfaces for controlling the surgical robot 34 .

The photographing device 36 includes at least one image sensor. That is, the imaging device 36 includes at least one camera device, and is used to photograph an object, that is, a surgical site. In an embodiment, the imaging device 36 includes at least one camera coupled to a surgical arm of the surgical robot 34 .

In an embodiment, the image captured by the photographing device 36 is displayed on the display 340 .

In one embodiment, the surgical robot 34 includes one or more surgical tools 38 capable of performing cutting, clipping, fixing, grabbing operations of the surgical site, and the like. The surgical tool 38 is used in combination with the surgical arm of the surgical robot 34 .

The controller 30 receives information necessary for surgery from the server 100 or generates information necessary for surgery and provides it to the user. For example, the controller 30 displays generated or received information necessary for surgery on the display 32 .

For example, the user performs robotic surgery by controlling the movement of the surgical robot 34 by manipulating the controller 30 while viewing the display 32 .

The server 100 generates information necessary for robotic surgery by using the medical image data of the object photographed in advance from the medical imaging device 10 , and provides the generated information to the controller 30 .

The controller 30 provides the information received from the server 100 to the user by displaying it on the display 32 , or controls the surgical robot 34 using the information received from the server 100 .

In one embodiment, the means that can be used in the medical imaging equipment 10 is not limited, for example, CT, X-Ray, PET, MRI, etc., various other medical image acquisition means may be used.

When performing robotic surgery as described above, it is possible to acquire data including surgical images captured in the surgical process or various surgical information in the control process of the surgical robot. In one embodiment, in the present invention, artificial data may be generated by acquiring the object image or background image as described above based on the surgical information (ie, surgical image) obtained in the robotic surgery process. Of course, in generating artificial data according to an embodiment of the present invention, utilizing the surgical image obtained in the robotic surgery process is only one example, and the present invention is not limited thereto. In another embodiment, minimally invasive surgery using a laparoscope, an endoscope, or a surgical image obtained when medical staff directly performs an operation on a patient may also be utilized to generate artificial data.

32 is a diagram schematically showing the configuration of an apparatus 700 for performing a method for generating artificial data according to an embodiment of the present invention.

Referring to FIG. 32 , the processor 710 includes one or more cores (not shown) and a graphic processing unit (not shown) and/or a connection path (eg, a bus) for transmitting and receiving signals with other components. ) may be included.

The processor 710 according to an embodiment executes one or more instructions stored in the memory 720 to perform the artificial data generation method described with reference to FIGS. 25 to 30 .

For example, the processor 710 may obtain a background image and an object image by executing one or more instructions stored in the memory 720 , and may generate artificial data by combining the background image and the object image.

Meanwhile, the processor 710 includes a random access memory (RAM) and a read-only memory (ROM) that temporarily and/or permanently store signals (or data) processed inside the processor 710 . , not shown) may be further included. In addition, the processor 710 may be implemented in the form of a system on chip (SoC) including at least one of a graphic processing unit, a RAM, and a ROM.

The memory 720 may store programs (one or more instructions) for processing and controlling the processor 710 . Programs stored in the memory 720 may be divided into a plurality of modules according to functions.

The artificial data generation method according to an embodiment of the present invention described above may be implemented as a program (or application) to be executed in combination with a computer, which is hardware, and stored in a medium.

With reference to FIGS. 33 to 36, a relay server for constructing surgical information according to an embodiment of the present invention and a method for constructing surgical information performed by the relay server will be described in detail.

33 is a schematic diagram of a system capable of performing robotic surgery according to an embodiment of the present invention.

Referring to FIG. 33 , the robotic surgery system includes a medical imaging device 10 , a server 100 , and a controller 30 , a display 32 , and a surgical robot 34 provided in the operating room. According to an embodiment, the medical imaging device 10 may be omitted from the robotic surgery system according to the disclosed embodiment.

In one embodiment, the surgical robot 34 includes a photographing device 36 and a surgical tool 38 .

In one embodiment, the robotic surgery is performed by the user using the control unit 30 to control the surgical robot (34). In one embodiment, the robotic surgery may be performed automatically by the control unit 30 without the user's control.

The server 100 is a computing device including at least one processor and a communication unit.

The control unit 30 includes a computing device including at least one processor and a communication unit. In one embodiment, the controller 30 includes hardware and software interfaces for controlling the surgical robot 34 .

The photographing device 36 includes at least one image sensor. That is, the imaging device 36 includes at least one camera device, and is used to photograph an object, that is, a surgical site. In an embodiment, the imaging device 36 includes at least one camera coupled to a surgical arm of the surgical robot 34 .

In an embodiment, the image captured by the photographing device 36 is displayed on the display 340 .

In one embodiment, the surgical robot 34 includes one or more surgical tools 38 capable of performing cutting, clipping, fixing, grabbing operations of the surgical site, and the like. The surgical tool 38 is used in combination with the surgical arm of the surgical robot 34 .

The controller 30 receives information necessary for surgery from the server 100 or generates information necessary for surgery and provides it to the user. For example, the controller 30 displays generated or received information necessary for surgery on the display 32 .

For example, the user performs robotic surgery by controlling the movement of the surgical robot 34 by manipulating the controller 30 while viewing the display 32 .

The server 100 generates information necessary for robotic surgery by using the medical image data of the object photographed in advance from the medical imaging device 10 , and provides the generated information to the controller 30 .

The controller 30 provides the information received from the server 100 to the user by displaying it on the display 32 , or controls the surgical robot 34 using the information received from the server 100 .

In one embodiment, the means that can be used in the medical imaging equipment 10 is not limited, for example, CT, X-Ray, PET, MRI, etc., various other medical image acquisition means may be used.

In the disclosed embodiment, the surgical image acquired by the imaging device 36 is transmitted to the controller 30 .

Hereinafter, a process of constructing surgical information using a relay server that relays information to and from clients will be described. The relay server may mean the server 100 of FIG. 33 , or may be constructed separately from this and operate in conjunction with the server 100 of FIG. 33 .

Recently, surgery can be performed in various ways, such as when a medical staff directly performs an operation in an operating room, using a surgical robot or a simulator, or remotely guides an operation. Accordingly, in order to provide a more effective and optimized surgery to the patient during surgery, there is a need for a method for connecting various devices used during surgery and transmitting/receiving information therebetween. Accordingly, the present invention intends to provide a method for connecting devices using a relay server and collecting and utilizing information through this.

34 is a diagram schematically showing the configuration of a surgical information building system according to an embodiment of the present invention.

Referring to FIG. 34 , the surgical information building system according to an embodiment of the present invention may include a relay server 200 and a plurality of clients. As an embodiment, in FIG. 2 , it is described as including the first client 210 , the second client 220 , and the third client 230 , but this is only an example and the present invention is not limited thereto. . Of course, it can be configured to include fewer or more clients than shown in FIG. 34 .

The relay server 200 is a device that relays communication between a plurality of clients, and may perform, for example, a function such as a chat server. For example, the relay server 200 may relay surgery by connecting a robotic surgery system using a surgical robot, a virtual surgery system using a simulator, and a remote system for guiding surgery remotely during surgery.

Also, the relay server 200 may construct learning data by receiving various surgical information from a plurality of clients. According to an embodiment, the relay server 200 may perform learning based on the learning data and calculate an optimized surgical process, or may provide the learning data to an external server. In this case, the external server may calculate an optimized surgical process by performing learning based on the learning data provided from the relay server 200 .

The first client 210 may be a device for generating actual surgery information by acquiring an actual surgery image. In one embodiment, the first client 210 may be a device provided in an operating room (surgery site). For example, it may correspond to the surgical robot 34 or the control unit 30 shown in FIG. 1 , or may be provided in the operating room as a separate device and connected to the surgical robot 34 or the control unit 30 .

The first client 210 may access the relay server 200 and transmit actual surgical information generated during surgery to the relay server 200 . Also, the first client 210 may receive various information generated by other clients from the relay server 200 .

Here, the actual surgical image may refer to data obtained as an actual medical staff performs an operation. For example, it may be an image of an actual surgical scene actually performed by the surgical robot 34 . That is, the actual surgical image is data recorded about the surgical site and operation in the actual surgical process.

The actual operation information is information obtained from the actual operation image, and may include, for example, information about a patient's surgical site, information about a surgical tool, and information about a camera. In this case, the surgical site refers to a body part of a patient on which surgery is being performed, and may be, for example, an organ or blood vessel. The surgical tool may be a tool used during surgery, for example, may be a tool necessary for performing robotic surgery or consumables used during surgery. The camera is a device for photographing an actual surgical scene, for example, may be installed in an operating room to photograph a surgical scene, or may be provided with an endoscope that enters the patient's body during robotic surgery to photograph the inside of the body.

For example, the first client 210 acquires an actual surgical image from a camera or an external device included in the surgical robot, recognizes a surgical site, a surgical tool, and a camera from the acquired actual surgical image, and based on this, the actual surgical information can create That is, the first client 210 may generate the type and location information of the surgical site, the type, location, direction, and movement information of the surgical tool, and information on the location and movement direction of the camera from the actual surgical image. Thereafter, the first client 210 may transmit the generated actual surgery information to the relay server 200 and provide it to other clients connected to the relay server 200 .

The second client 220 may be a device that generates virtual surgery information by performing a virtual surgery. In one embodiment, the second client 220 may be a device provided in an operating room (surgery site) or a remote location away from it, for example, rehearsal or simulation for a virtual body model modeled identically to the surgical subject (ie, patient). It may be a simulator capable of performing an action.

Here, virtual surgery refers to a virtual surgery in which rehearsal or simulation is performed on a virtual body model that is implemented in the same way as the actual body state based on the patient's medical image data (eg, CT, MRI, PET image, etc.) . The virtual surgery information is information obtained based on image data on which rehearsal or simulation is performed on a virtual body model in a virtual space, and may include simulation data. For example, the simulation data may include information about a surgical site of a subject to be operated on, information about a surgical tool, and information about a camera. That is, the virtual surgery information may include all information included in the data recorded for the surgical operation performed on the virtual body model.

In addition, the virtual body model may be 3D modeling data generated based on medical image data taken in advance of the inside of the patient's body. For example, it is modeled in conformity with the body of the subject to be operated on, and may be calibrated to the same state as the actual surgical state.

The second client 220 may access the relay server 200 and transmit virtual surgery information generated by performing the virtual surgery to the relay server 200 . Also, the second client 220 may receive various information generated by other clients from the relay server 200 .

For example, the second client 220 receives the actual surgery information generated by the first client 210 from the relay server 200, and performs virtual surgery based on the received actual surgery information to receive the virtual surgery information. can create

In the case of virtual surgery, it is important to provide the same virtual surgical state as the actual surgical state for the surgical subject. Accordingly, in the present invention, the actual operation information (eg, the position, direction, movement information of the surgical tool, etc.) obtained by performing the actual operation on the first client 210 through the relay server 200 is transferred to the second client 220 . Since it can be transmitted in real time, the second client 220 can provide more accurate virtual surgery information reflecting the actual surgery information to the surgical staff in real time. Therefore, since the surgical medical staff can use both the information of the first client 210 and the second client 220 through the relay server 200 , they can effectively perform the actual surgery on the surgery target.

The third client 230 may be a device for generating surgical guide information corresponding to an actual surgical image. In one embodiment, the third client 230 may be a device provided in an operating room (surgery site) or a remote location away from it, and for example, when a specific event occurs during surgery, it may generate surgical guide information in response thereto. Alternatively, another medical team may access the third client 230 at the request of a medical team performing surgery in the operating room (surgery site). In this case, by performing communication between the third clients 230 through the relay server 200, other medical staff can provide information about the operation in real time to the medical staff performing the operation in the operating room (surgery site).

The third client 230 accesses the relay server 200 and transmits the surgical guide information generated by it to the relay server 200 , and can receive various information generated by other clients from the relay server 200 . .

For example, the third client 230 may monitor the surgical situation as it receives information (ie, actual surgery information and virtual surgery information) provided by other clients 210 and 220 from the relay server 200 . . In addition, the third client 230 may recognize whether a specific event occurs during the operation based on the monitoring, and generate operation guide information corresponding thereto. According to an embodiment, the third client 230 may automatically recognize whether a specific event occurs, and generate predetermined surgical guide information according to the specific event. Alternatively, the medical staff may monitor the actual surgical situation through the third client 230 to recognize that a specific event has occurred, and input corresponding surgical guide information.

The relay server 200 according to the embodiment of the present invention can additionally connect clients necessary for surgery in addition to the first, second, and third clients 210, 220, 230 described above, and transmit and receive information between these clients. can play a role in controlling it.

On the other hand, when surgery information is directly transmitted and received between clients without going through the relay server 200, surgery information is shared only between clients that communicate directly, so that other clients cannot obtain information about the surgical procedure. . Accordingly, there is a problem in that it is difficult for other clients to intervene in the middle of the operation, and the operation process cannot be checked in real time.

In order to solve this problem, in the embodiment of the present invention, as described above, the relay server 200 plays a role in relaying communication between clients, so that an intermediate intervention is performed through the relay server 200 during the actual operation. can In addition, the actual surgical procedure can be monitored in real time through the relay server 200 . In addition, since the relay server 200 can obtain surgical information from all clients, it is possible to derive information about the entire surgical procedure. Accordingly, the relay server 200 may record the entire surgical process in order through this surgical information to form history information for the corresponding surgery.

35 is a flowchart illustrating a method of constructing surgical information using a relay server according to an embodiment of the present invention. The method of FIG. 35 may be performed in the surgical information building system shown in FIG. 34 .

According to an embodiment, first, the relay server 200 may create a group of at least one client necessary for surgery. For example, the relay server 200 may create a group of clients necessary for actual surgery based on a specific patient, or may directly receive a group request signal from a client and create a group in response thereto. . Alternatively, when receiving an access request from a client, the relay server 200 may automatically determine whether the client belongs to a corresponding group and process the client to access its own group.

Thereafter, the relay server 200 enables transmission and reception of information between clients in the group as the operation proceeds. A detailed operation process for this will be described below.

Referring to FIG. 35 , the first client 210 may acquire an actual surgical image and generate actual surgical information corresponding to the acquired actual surgical image (S100).

In one embodiment, the first client 210 acquires an actual surgical image from a camera or an external device included in the surgical robot, recognizes a surgical site, a surgical tool, and a camera from the acquired actual surgical image, and performs an actual operation based on this. information can be generated. Actual surgical information includes, as described above, surgical site information (eg, type and location information of the surgical site), surgical tool information (eg, surgical tool type, location, direction, and movement information), camera information (eg, camera) location and motion information), and the like.

The first client 210 may transmit the actual operation information to the relay server 200 (S105), and the relay server 200 may store the actual operation information received from the first client 210 in a database ( S110).

Also, the relay server 200 may transmit the actual operation information to at least one client (eg, the second client 220 and/or the third client 230) in the same group (S115).

The second client 220 may perform virtual surgery to generate virtual surgery information (S120).

In one embodiment, the second client 220 receives the actual surgery information generated by the first client 210 through the relay server 200, and performs virtual surgery based on the received actual surgery information to perform virtual surgery. information can be generated.

The second client 220 may transmit virtual surgery information to the relay server 200 (S125), and the relay server 200 may store the virtual surgery information received from the second client 220 in a database ( S130).

Also, the relay server 200 may transmit the virtual surgery information to at least one client (eg, the first client 210 and/or the third client 230) in the same group (S135).

The third client 230 may monitor the surgical situation based on the surgical information for the subject to be operated on received through the relay server 200, and may also generate surgical guide information based on the monitoring result (S140).

In one embodiment, the third client 230 generates a specific event during surgery based on the actual surgery information of the first client 210 and the virtual surgery information of the second client 220 received from the relay server 200 . It is possible to recognize whether or not to do so, and to generate surgical guide information corresponding thereto. In another embodiment, the third client 230 obtains the actual surgical image through the relay server 200 and provides it to the medical staff, so that the medical staff can directly generate surgery guide information corresponding to the actual surgical image. In this case, the third client 230 may receive surgical guide information from medical staff and transmit it to the relay server 200 .

The third client 230 may transmit the surgery guide information to the relay server 200 (S145), and the relay server 200 may store the surgery guide information received from the third client 230 in a database (S145). S150).

Also, the relay server 200 may transmit the surgical guide information to at least one client (eg, the first client 210 and/or the second client 220) in the same group (S155).

As described in steps S100 to S155 described above, the relay server 200 serves as a relay to transmit and receive information between clients belonging to a group as an operation is performed on a subject for surgery, and thus to store the transmitted and received information in a database can In addition, the relay server 200 may perform learning to construct learning data and provide an optimized surgical process by using various surgical information about the surgery target stored in the database.

The relay server 200 may generate cue sheet data for a surgical procedure of a subject for surgery based on various surgical information received from the client (S160).

In an embodiment, the relay server 200 stores at least one of actual surgery information of the first client 210 , virtual surgery information of the second client 220 , and surgery guide information of the third client 230 stored in the database. It is possible to generate cue sheet data for the surgical procedure of the surgical subject based on the .

Here, the cue sheet data may refer to data recorded in order by dividing a specific surgical procedure into detailed surgical operations. The detailed surgical operation may be the minimum operation unit constituting the surgical process, and may be divided according to several criteria. For example, detailed surgical motions include the type of surgery (eg, laparoscopic surgery, robotic surgery, etc.), the anatomical body part where the surgery is performed, the surgical tools used, the number of surgical tools, and the direction or location of the surgical tools on the screen. , can be divided based on the movement of the surgical tool (eg, forward/retracted, etc.). Since the relay server 200 can grasp information on criteria for dividing detailed surgical motions through actual surgery information, virtual surgery information, surgery guide information, etc. stored in the database, cue sheet data corresponding to the actual surgical procedure of the surgery target can be built accurately. According to an embodiment of the present invention, the relay server 200 can form a history of the entire surgical procedure only by generating cue sheet data. Also, according to an embodiment of the present invention, the relay server 200 may perform step S165.

The relay server 200 may perform learning based on the cue sheet data to calculate the optimized cue sheet data corresponding to the surgical procedure of the surgery subject (S165).

In one embodiment, the relay server 200 may configure actual surgery information, virtual surgery information, surgery guide information, etc. stored in the database as learning data, or configure the cue sheet data itself as learning data. Thereafter, the relay server 200 may perform learning based on the learning data and calculate an optimized surgical process for the surgery target. Here, the learning method may use a machine learning method such as supervised learning, unsupervised learning, reinforcement learning, for example, a deep learning learning method may be used.

The relay server 200 may store the optimized cue sheet data in a database, and may provide the optimized cue sheet data to at least one client in the group. Accordingly, at least one client may use the received optimized cue sheet data during surgery of a subject to be operated on.

In this embodiment, the relay server 200 performs learning based on the learning data to calculate the optimized cue sheet data, but according to the embodiment, the relay server 200 stores real surgical information stored in the database, virtual surgery Information, surgery guide information, etc. (ie, learning data) may be provided to an external server, and learning may be performed on the external server based on the learning data.

In addition, in an embodiment of the present invention, a virtual body model of a subject for surgery may be implemented based on the cue sheet data for the subject for surgery generated in step S160 or/and step S165, or conversely, surgery performed on the subject for surgery You can even create images.

According to an embodiment of the present invention, surgical information can be collected from multiple clients in real time during an actual operation, so that surgical staff can effectively perform a surgical procedure. In addition, according to an embodiment of the present invention, since surgery can be performed through more various surgical information and a surgical system compared to a case where surgery is performed only by a surgical staff, it is possible to perform a more accurate operation.

According to an embodiment of the present invention, since operation information can be transmitted and received through a relay server rather than transmitted and received by each client, the communication load can be reduced. In addition, since each client only needs to communicate through the relay server, even if the number of clients increases, mutual communication is easily possible and an additional load is not generated. Therefore, even if the number of clients increases, the performance is not affected.

According to an embodiment of the present invention, since all surgical information can be built through a relay server, it can be utilized as learning data in the future.

36 is a diagram schematically showing the configuration of an apparatus 300 for constructing surgical information according to an embodiment of the present invention.

Referring to FIG. 36 , the processor 310 includes one or more cores (not shown) and a graphic processing unit (not shown) and/or a connection path (eg, a bus) for transmitting and receiving signals with other components. ) may be included.

The processor 310 according to an embodiment executes one or more instructions stored in the memory 320, thereby performing the surgical information construction method described in relation to FIGS. 34 and 35 .

For example, the processor 310 receives the actual surgery information for the actual surgery image from the first client by executing one or more instructions stored in the memory 320, transmits the actual surgery information to the second client, and the second It is possible to receive the virtual surgery information generated based on the actual operation information from the client, and store the actual operation information and the virtual operation information.

Meanwhile, the processor 310 temporarily and/or permanently stores a signal (or data) processed inside the processor 310 . , not shown) may be further included. In addition, the processor 310 may be implemented in the form of a system on chip (SoC) including at least one of a graphic processing unit, a RAM, and a ROM.

The memory 320 may store programs (one or more instructions) for processing and controlling the processor 310 . Programs stored in the memory 320 may be divided into a plurality of modules according to functions.

The method for constructing surgical information according to an embodiment of the present invention described above may be implemented as a program (or application) to be executed in combination with a computer, which is hardware, and stored in a medium.

Hereinafter, embodiments of recognizing various information such as the position and direction of surgical tools or cameras existing inside the patient's body during minimally invasive surgery, the state of contact with internal organs of the body, etc., and providing the recognized information to the user are disclosed. do.

In one embodiment, there is provided a method for providing a camera position based on a surgical image performed by a computer, the method comprising: acquiring a surgical image photographed as the camera enters the body and moves a surgical path; deriving camera position information on a standard body model based on the surgical image; and providing camera location information on the virtual body model of the current surgery target based on the camera location information on the standard body model. A detailed description thereof will be described later with reference to FIGS. 37 to 45 .

In another embodiment, as a method for calculating position information of a surgical tool performed by a computer, based on sensing information obtained from a sensing device attached to a surgical tool to be inserted into the internal space of the surgical subject, calculating a position of a first point of the surgical tool; Through the virtual body model generated in accordance with the physical condition of the surgery subject, the surgical tool for the internal space of the surgical tool by reflecting the characteristic information of the surgical tool based on the position of the first point of the surgical tool calculating a second point position; and providing positional information of the surgical tool in an actual internal body space of the surgical subject based on the position of the second point of the surgical tool with respect to the virtual body model, wherein the position of the first point of the surgical tool is a position for a specific point located on the external space of the body in the surgical tool, and the second point position of the surgical tool is a position for a specific point located on the internal space of the body in the surgical tool characterized. A detailed description thereof will be described later with reference to FIGS. 46 to 49 .

In another embodiment, as a method of calculating a camera position using an actual surgical image performed by a computer, the reference position for the camera is obtained by obtaining a reference object from an actual surgical image captured by a camera entering the body of a subject for surgery. setting; calculating a position change amount of the camera as the camera moves; and calculating a current position of the camera based on an amount of change in the position of the camera with respect to the reference position. A detailed description thereof will be described later with reference to FIGS. 50 to 55 .

In another embodiment, there is provided a method of providing position information of a surgical tool, performed by a computer, comprising: obtaining coordinate information of a surgical robot including a surgical tool based on a reference point of a subject to be operated; matching the coordinate information of the virtual body model generated in accordance with the physical condition of the subject to be operated with the coordinate information of the surgical robot; and calculating the position of the surgical tool in the virtual body model corresponding to the position of the surgical tool obtained from the coordinate information of the surgical robot. A detailed description thereof will be described later with reference to FIGS. 56 to 59 .

37 to 45, a method for providing a camera position based on a surgical image according to an embodiment of the present invention will be described in detail.

When performing a medical operation, it is possible to acquire data including a surgical image captured in the surgical process or various surgical information generated in the surgical process. For example, in the case of minimally invasive surgery such as laparoscopic surgery or robotic surgery, a camera enters the inside of the patient's body together with a surgical tool, and microsurgery is performed while observing the surgical image captured by the camera. Accordingly, in the present invention, it is intended to propose a method that can utilize the surgical information (ie, surgical image) that can be obtained during the operation as described above. Hereinafter, as a method proposed by the present invention, a method of providing actual position information of a camera in an operation currently being performed on a specific patient based on a surgical image will be described.

In addition, hereinafter, for convenience of description, it will be described that a "computer" performs the embodiments disclosed in this specification. “Computer” may be used to encompass a device capable of performing computing processing. For example, a "computer" is a variety of devices that can perform arithmetic processing and provide a result to a user. phone), PCS phone (Personal Communication Service phone), synchronous/asynchronous IMT-2000 (International Mobile Telecommunication-2000) mobile terminal, Palm PC (Palm Personal Computer), Personal Digital Assistant (PDA; Personal Digital Assistant) ) may also apply. Also, when a head mounted display (HMD) device includes a computing function, the HMD device may be a computer. In addition, the computer may correspond to a server that receives a request from a client and performs information processing.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

37 is a flowchart illustrating a method of providing a camera position based on a surgical image according to an embodiment of the present invention.

Referring to FIG. 37 , the method for providing a camera position based on a surgical image performed by a computer according to an embodiment of the present invention includes the steps of acquiring a surgical image photographed as the camera moves into the body and moves the surgical path ( S100), deriving camera location information on the standard body model based on the surgical image (S110), and providing camera location information on the virtual body model of the current surgery target based on the camera location information on the standard body model It may include a step (S120) of doing. Hereinafter, detailed description of each step is described.

The computer may acquire a surgical image taken as the camera enters the inside of the body and moves on the surgical path (S100).

In one embodiment, the computer may acquire an actual surgery image as the actual operation is performed. For example, the computer may acquire an actual surgical image photographed when minimally invasive surgery such as robotic surgery or laparoscopic surgery is performed, or may be data obtained as an actual medical staff performs an operation.

In this case, the surgical image may be a surgical image obtained by performing surgery on at least one patient. In general, in the case of a surgical operation, the same type of operation proceeds with a similar surgical procedure even if the doctor or patient is different. Therefore, if the operation is the same, the surgical images acquired for several arbitrary patients include similar surgical motions or surgical routes. Accordingly, the computer may acquire a surgical image from at least one patient and use it as learning data for learning. In addition, the computer can learn by using the learning data in a standard body model to be described later. A detailed process for this will be described later.

38 is a diagram illustrating an operation route for each patient for a specific operation. Although the three-dimensional modeled body model (eg, virtual body model) for each patient shown in (a), (b), (c) of FIG. 38 is different from each other, the surgical path 200, 210, 220 through which surgery is performed ) are similar to each other. Therefore, when surgery is performed along this surgical route, it is possible to obtain a surgical image including a similar surgical route for each patient shown in (a), (b), and (c).

Referring back to FIG. 37 , the computer may derive camera location information on the standard body model based on the surgical image ( S110 ).

Here, the standard body model may be a three-dimensional body model generated by standardizing anatomical features of the body. For example, by standardizing anatomical features such as appearance, size, and location for each body part (eg, liver, heart, uterus, brain, organs such as breast, abdomen, and blood vessels), each body part is 3D It may be a body model constructed by modeling. In addition, the standard body model may be a body model generated by standardizing the anatomical characteristics of each body according to various conditions such as gender, age, external condition, etc., and may be implemented as at least one standard body model according to these various conditions. have.

In one embodiment, the computer may designate a surgical image corresponding to at least one reference point on the surgical path from the surgical image. The reference point may be at least one point satisfying a predetermined condition among points at which the camera enters the body and moves along the surgical path. For example, a point on the surgical route at which the camera does not move for a predetermined time may be determined as a reference point. Alternatively, a point at which the camera can be located on the surgical path may be determined as a reference point based on a predetermined time interval. Next, the computer may use the surgical image corresponding to at least one reference point to map to the corresponding point on the standard body model. Then, the computer may calculate coordinate information for a corresponding point on the standard body model, and derive it as camera position information on the standard body model. In this case, the camera position information may refer to coordinate information on a three-dimensional space in a standard body model that is viewed from the same viewpoint as a surgical image corresponding to at least one reference point.

39 is a view for explaining an example of a process of deriving camera position information on a standard body model from a surgical image according to an embodiment of the present invention. Referring to FIG. 39 , the computer may acquire a surgical image 300 , and may detect surgical images 310 and 320 corresponding to at least one reference point in the acquired surgical image 300 . As an embodiment, a point at which the camera does not move on the surgical path inside the patient's body (for example, it may be a case where surgery is being performed in a specific area) may be designated as a reference point. In this case, the computer may recognize, as a reference point, a point corresponding to a surgical image at a point in time at which camera movement does not occur among the surgical images 300 , that is, surgical images in which the same background image continuously appears. Therefore, the computer converts the surgical images 310 and 320 of the last time among the surgical images at the time when the camera does not move, that is, the surgical images 310 and 320 at the time when the movement of the camera occurs. It can be detected from the image 300 . Thereafter, the computer may map the points corresponding to the detected surgical images 310 and 320 on the surgical path 330 of the standard body model, and calculate the location information of the mapped points as coordinate information of the standard body model. can For example, when the standard body model is 3D modeled data, it may be calculated as coordinate information for 3D space. The embodiment of FIG. 39 may be implemented through learning. For example, the learning method may use a machine learning method such as supervised learning, unsupervised learning, reinforcement learning, for example, a deep learning-based convolutional neural network (CNN) may be used. Alternatively, a person (eg, a doctor, etc.) may perform learning by detecting a surgical image corresponding to a reference point and labeling a corresponding point on the standard body model.

In another embodiment of step S110, the computer may designate a surgical image corresponding to at least one reference point on the surgical path from the surgical image. In addition, the computer may acquire a virtual body model of the patient corresponding to the surgical image, and map the surgical image corresponding to at least one reference point on the virtual body model of the patient. This may proceed similarly to the process of FIG. 39 described above. Thereafter, the computer may convert and derive the point mapped on the patient's virtual body model into camera position information on the standard body model. At this time, the computer may perform a process of transforming the coordinate information of the virtual body model into the coordinate information of the standard body model, for example, using a transformation matrix (Transformation Matrix) to match the coordinate information between each other.

40 is a diagram for explaining an example of a process of converting coordinate information of a virtual body model of a patient into coordinate information of a standard body model according to an embodiment of the present invention. The coordinate information of the standard body model is derived by transforming the virtual body model for each patient shown in (a), (b), (c) of FIG. 40 by using the transformation matrix (T _A , T _B , T _{C ).} can do. The computer can finally calculate coordinate information transformed into the standard body model space from the surgical image obtained for each patient shown in (a), (b), (c) of FIG. 40 .

In the embodiment of the above-described step S110, the computer may derive the position information on the standard body model through learning by configuring the surgical image as learning data. Such a learning process will be described with reference to FIGS. 42 and 43 .

Referring back to FIG. 37 , the computer may provide camera location information on the virtual body model of the current surgery target based on the camera location information on the standard body model ( S120 ).

Here, the current surgery target may be different from the patient corresponding to the surgical image obtained in step S100 described above, and may mean a patient to be currently operated on. In addition, a surgical image obtained in advance for the current surgery target may be used.

In one embodiment, the computer may convert coordinate information on the standard body model into coordinate information on the virtual body model of the current surgery target. In addition, the computer may calculate a corresponding position corresponding to the camera position information on the standard body model from the coordinate information converted on the virtual body model of the current surgery target. That is, the standard body model and the corresponding camera position between the virtual body model of the current surgery subject may have the same camera view point, and thus, finally, the camera view point of the surgical image obtained in step S100 and the virtual body model of the current surgery subject. The camera viewpoint may be mapped identically. In other words, the computer may determine the position information of the current camera on the virtual body model of the current surgery subject through mapping with the standard body model based on the surgical image acquired in step S100.

In performing the transformation between the standard body model and the virtual body model, the transformation may be performed on the entire body or transformation may be performed for each organ inside the body. In addition, during the transformation, a linear transformation may be applied or a non-linear transformation may be applied. For example, the computer may extract a reference position on the standard body model, and extract a corresponding reference position on the virtual body model corresponding to the extracted reference position. And by applying a transformation matrix by matching reference positions corresponding to each other, coordinate information on a standard body model can be converted into coordinate information on a virtual body model. Then, the computer may derive the camera position information from the converted coordinate information on the virtual body model. Here, the reference position may be determined by extracting a specific organ inside the body and determining the position as the reference position, or by extracting each organ inside the body and determining the reference position of each organ (eg, the center point of the organ) as the reference position. have. For example, when transformation is performed on the entire body, transformation may be performed by extracting a specific organ as a reference and applying a transformation matrix based on the corresponding position. Alternatively, when transformation is performed for each organ, the transformation may be performed by extracting a reference position for each organ and applying a transformation matrix based on each extracted reference position.

41 is a diagram for explaining a process of converting coordinate information of a standard body model into coordinate information of a virtual body model for a current surgery target according to an embodiment of the present invention. In one embodiment, as shown in FIG. 5 , the computer may extract each organ by applying segmentation to the standard body model, and may calculate a reference position for each organ. In addition, the computer may extract each organ by applying segmentation to the virtual body model of the current surgery subject, and calculate a reference position for each organ. The computer can apply the transformation matrix (T) by matching the reference positions of the corresponding organs between the standard body model and the virtual body model. In this case, the transformation matrix may be linearly transformed or non-linearly transformed according to an embodiment. The computer can transform the coordinate information between the standard body model and the virtual body model by applying the transformation matrix. Therefore, finally, the computer can derive the camera position information converted into the coordinate information on the virtual body model.

42 and 43 are diagrams for explaining a process of deriving position information on a standard body model from a surgical image through learning according to an embodiment of the present invention.

Referring to FIG. 42 , the computer receives the surgical image 400 as an input and outputs the camera position information (x, y, z, r _x , r _y , r _z ) in the standard body model space. Learning can be performed using the CNN method. For example, the computer may acquire a surgical image 400 for a new patient as an input value. In this case, the surgical image 400 may be an image corresponding to a specific location inside the body of the new patient. Accordingly, the computer can search the position information (x, y, z, r _x , r _y , r _z ) on the standard body model through the CNN learning model 410 for the input surgical image 400 . . In the case of such a learning model, an existing patient's surgical image and location information on a standard body model corresponding to the surgical image may be used as a learning dataset.

In addition, as the above-described embodiment of FIG. 40 , even when the computer derives location information on the standard body model through the virtual body model of the patient from the surgical image, the above-described CNN method learning can be performed. For example, the computer receives the surgical image 400 as an input and outputs the camera position information (x, y, z, r _x , r _y , r _z ) in the virtual body model space of the patient as an output. Learning can be performed using the CNN method. In addition, the computer applies the transformation matrix between the patient's virtual body model and the standard model, and converts the camera position information (x, y, z, r _x , r _y , r _z ) in the patient's virtual body model space to the standard body model. It can be transformed into a position on the image, and this process can also be applied to CNN learning. In the case of such a learning model, the existing patient's surgical image, location information on the patient's virtual body model, and the transformation matrix between the standard body model and the virtual body model can be used as the learning dataset, and between the standard body model and the virtual body model. The position on the virtual body model and the position on the standard body model through the transformation matrix can be constructed as the final training dataset.

In addition, in an embodiment of the present invention, training may be performed in the same manner as in FIG. 42 and a test process for training may be performed. Referring to FIG. 43 , the computer may receive the surgical image 400 as an input through the CNN 410 and output coordinate information 420 in the standard body model space. In addition, the computer may convert the coordinate information 420 on the output standard body model space into coordinate information 430 for the virtual body model of the patient using the transformation matrix T. That is, the computer may determine whether the value to be predicted is accurately inferred from the training data by performing the training and testing process.

On the other hand, the virtual body model in the present invention may be 3D modeling data generated based on medical image data photographed in advance of the inside of the patient's body. For example, it is modeled in conformity with the body of the subject to be operated on, and may be calibrated to the same state as the actual surgical state.

In an embodiment of the present invention, the computer may calculate the camera position information in the real body of the current surgery subject based on the camera position information on the virtual body model derived in step S120 during surgery on the current surgery subject. For example, when the computer receives a current surgical image obtained by photographing a current surgical scene during an actual operation on a current surgical subject, the computer performs the above-described steps S100 to S120 to correspond to the viewpoint of the received current surgical image. Camera position information on the virtual body model can be calculated. Since the virtual body model is modeled to match the body of the current surgery subject and implemented in the same state as during the actual operation, the computer detects the current camera position information on the virtual body model to detect the camera within the actual body of the current surgery subject. It is possible to calculate the information on the location of the point.

In addition, in an embodiment of the present invention, the computer may guide the movement path of the camera on the surgical path of the current surgery target based on the camera position information on the virtual body model derived in step S120. For example, the computer may recognize that the operation of the first area on the operation path of the current operation target is finished. Then, the computer may recognize that the operation is proceeding to the second area, which is the next surgical area of the first area, and accordingly guide the location information on the second area. For example, when the current surgical image is output through the screen, information indicating the movement path of the camera (eg, the movement time point and movement direction of the camera) on the screen of the current operation image may be output together.

At this time, in recognizing that the surgery in the first area is finished and the surgery is in the second area, the computer may recognize the end of the surgery in the corresponding area (point) by image recognition, and as auxiliary information other than the surgical image. You can also use cue sheet data. The cue sheet data is composed of data that lists actual surgical procedures in order according to time based on the minimum surgical operation unit, and may include operation information corresponding to the minimum surgical operation unit. Therefore, the computer can recognize the surgical operation information of the corresponding surgical area from the cue sheet data, and recognize the end of the surgical operation based on this. In addition, since the cue sheet date consists of a hierarchical structure classified from the upper surgical operation to the lower surgical operation, even if the same surgical operation appears several times, by using the cue sheet data, it is possible to identify where the corresponding surgical operation belongs on the hierarchical structure. It is possible to recognize the end of the surgical operation relatively accurately.

In addition, in an embodiment of the present invention, the computer may display the movement path of the camera on the virtual body model. For example, a movement path that moves step by step from the first operation area to the second operation area on the operation path of the current operation target may be displayed on the virtual body model. In this case, the computer can determine the moving time of the camera by recognizing the actions immediately before moving to the next surgical area (surgery stage) from the surgical image during surgery. In addition, as described above, the computer may recognize the action immediately before moving from the cue sheet data to the next surgical area (surgery stage) to determine the time of movement of the camera.

In addition, in an embodiment of the present invention, when performing minimally invasive surgery such as laparoscopic surgery or robotic surgery, a function related to the movement path of the camera may be added to the laparoscope or the surgical robot. For example, when the surgical operation in the first surgical area on the current surgical path of the patient is terminated, the computer may provide a function of setting the camera to automatically move the position of the camera to the next surgical area (surgical stage).

According to the present invention, it is possible to efficiently provide camera coordinate information in a three-dimensional space with only a surgical image, even if a separate device is not provided to determine the position of the camera during actual surgery.

In addition, during surgery, it is important to provide a camera position in the area where the actual surgery is being performed. Therefore, in the present invention, by deriving the camera position information on the point corresponding to the reference position on the surgical route, it is possible to provide a point where the camera should be located in advance before surgery.

According to the present invention, since it is only necessary to build the learning model for the standard body model without having to individually build the learning model for each patient, the process of building the learning model and performing the learning can be performed efficiently.

44 is a schematic diagram of a system capable of performing robotic surgery according to an embodiment of the present invention.

Referring to FIG. 44 , the robotic surgery system includes a medical imaging device 10 , a server 100 , and a controller 30 , a display 32 , and a surgical robot 34 provided in the operating room. According to an embodiment, the medical imaging device 10 may be omitted from the robotic surgery system according to the disclosed embodiment.

In one embodiment, the surgical robot 34 includes a photographing device 36 and a surgical tool 38 .

In one embodiment, the robotic surgery is performed by the user using the control unit 30 to control the surgical robot (34). In one embodiment, the robotic surgery may be performed automatically by the control unit 30 without the user's control.

The server 100 is a computing device including at least one processor and a communication unit.

The control unit 30 includes a computing device including at least one processor and a communication unit. In one embodiment, the controller 30 includes hardware and software interfaces for controlling the surgical robot 34 .

The photographing device 36 includes at least one image sensor. That is, the imaging device 36 includes at least one camera device, and is used to photograph an object, that is, a surgical site. In an embodiment, the imaging device 36 includes at least one camera coupled to a surgical arm of the surgical robot 34 .

In an embodiment, the image captured by the photographing device 36 is displayed on the display 340 .

In one embodiment, the surgical robot 34 includes one or more surgical tools 38 capable of performing cutting, clipping, fixing, grabbing operations of the surgical site, and the like. The surgical tool 38 is used in combination with the surgical arm of the surgical robot 34 .

The controller 30 receives information necessary for surgery from the server 100 or generates information necessary for surgery and provides it to the user. For example, the controller 30 displays generated or received information necessary for surgery on the display 32 .

For example, the user performs robotic surgery by controlling the movement of the surgical robot 34 by manipulating the controller 30 while viewing the display 32 .

The server 100 generates information necessary for robotic surgery by using the medical image data of the object photographed in advance from the medical imaging device 10 , and provides the generated information to the controller 30 .

The controller 30 provides the information received from the server 100 to the user by displaying it on the display 32 , or controls the surgical robot 34 using the information received from the server 100 .

In one embodiment, the means that can be used in the medical imaging equipment 10 is not limited, for example, CT, X-Ray, PET, MRI, etc., various other medical image acquisition means may be used.

As described above, when performing robotic surgery, it is possible to acquire data including surgical images captured in the surgical process or various surgical information in the control process of the surgical robot. Accordingly, in the present invention, surgical information (ie, surgical image) is acquired in the surgical process as described above, and based on this, it is possible to provide actual position information of the camera in a surgery currently being performed for a specific patient. Since this has been described in detail through the embodiments of FIGS. 37 to 43 , a description thereof will be omitted herein. Of course, the embodiments of FIGS. 37 to 43 described above are not applicable only in connection with the robotic surgery system shown in FIG. 44, and can be implemented in various forms in all kinds of embodiments to which the present invention can be applied. .

45 is a diagram schematically showing the configuration of an apparatus 500 for performing a method for providing a camera position based on a surgical image according to an embodiment of the present invention.

Referring to FIG. 45 , the processor 510 includes one or more cores (not shown) and a graphic processing unit (not shown) and/or a connection path (eg, a bus) for transmitting and receiving signals with other components. ) may be included.

The processor 510 according to an embodiment executes one or more instructions stored in the memory 520, thereby performing the method of providing a camera position based on the surgical image described in relation to FIGS. 37 to 43 .

For example, the processor 510 executes one or more instructions stored in the memory 520 to obtain a surgical image taken as the camera enters the inside of the body and moves the surgical path, and based on the surgical image, the standard body It is possible to derive camera position information on the model, and provide camera position information on the virtual body model of the current surgery target based on the camera position information on the standard body model.

Meanwhile, the processor 510 includes a random access memory (RAM) and a read-only memory (ROM) that temporarily and/or permanently store signals (or data) processed inside the processor 510 . , not shown) may be further included. In addition, the processor 510 may be implemented in the form of a system on chip (SoC) including at least one of a graphic processing unit, a RAM, and a ROM.

The memory 520 may store programs (one or more instructions) for processing and controlling the processor 510 . Programs stored in the memory 520 may be divided into a plurality of modules according to functions.

The method for providing a camera position based on a surgical image according to an embodiment of the present invention described above may be implemented as a program (or application) and stored in a medium in order to be executed in combination with a computer, which is hardware.

46 to 49 , a method for calculating position information of a surgical tool according to an embodiment of the present invention will be described in detail.

In this specification, "image" may mean multidimensional data composed of discrete image elements (eg, pixels in a 2D image and voxels in a 3D image). For example, the image may include a medical image of the object obtained by the CT imaging apparatus.

As used herein, an “object” may be a person or an animal, or a part or all of a person or an animal. For example, the object may include at least one of organs such as liver, heart, uterus, brain, breast, abdomen, and blood vessels. In particular, in the present specification, the internal body object may include a body part existing inside the body, an object introduced from the outside, an object created by itself, and the like. The body part may be an object existing inside the body, such as organs, blood vessels, bones, tendons, nervous tissue, and the like. The object introduced from the outside may be consumables necessary for surgery, such as gauze and clips. An object created by itself may include bleeding occurring in a body part.

As used herein, a “user” may be a medical professional, such as a doctor, a nurse, a clinical pathologist, or a medical imaging specialist, and may be a technician repairing a medical device, but is not limited thereto.

As used herein, "medical image data" is a medical image captured by a medical imaging device, and includes all medical images that can be implemented as a three-dimensional model of the body of an object. "Medical image data" may include computed tomography (CT) images, magnetic resonance imaging (MRI), positron emission tomography (PET) images, and the like.

As used herein, the term "virtual body model" refers to a model generated to match the actual patient's body based on medical image data. The "virtual body model" may be generated by modeling medical image data in 3D as it is, or may be corrected after modeling to be the same as during actual surgery.

As used herein, “virtual surgical data” refers to data including rehearsal or simulation actions performed on a virtual body model. “Virtual surgical data” may be image data on which rehearsal or simulation is performed on a virtual body model in a virtual space, or data recorded on a surgical operation performed on the virtual body model.

As used herein, “actual surgical data” refers to data obtained by actual medical staff performing surgery. "Actual surgical data" may be image data of a surgical site taken during an actual surgical procedure, or may be data recorded on a surgical operation performed in an actual surgical procedure.

As used herein, the term “computer” includes various devices capable of providing a result to a user by performing arithmetic processing. For example, computers include desktop PCs, notebooks (Note Books) as well as smart phones, tablet PCs, cellular phones, PCS phones (Personal Communication Service phones), synchronous/asynchronous A mobile terminal of International Mobile Telecommunication-2000 (IMT-2000), a Palm Personal Computer (PC), a Personal Digital Assistant (PDA), and the like may also be applicable. Also, when a head mounted display (HMD) device includes a computing function, the HMD device may be a computer. In addition, the computer may correspond to a server that receives a request from a client and performs information processing.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

46 is a schematic diagram of a system capable of performing robotic surgery according to an embodiment of the present invention.

Referring to FIG. 46 , the robotic surgery system includes a medical imaging device 10 , a server 100 , and a controller 30 , a display 32 , and a surgical robot 34 provided in the operating room. According to an embodiment, the medical imaging device 10 may be omitted from the robotic surgery system according to the disclosed embodiment.

In one embodiment, the surgical robot 34 includes a photographing device 36 and a surgical tool 38 .

In one embodiment, the robotic surgery is performed by the user using the control unit 30 to control the surgical robot (34). In one embodiment, the robotic surgery may be performed automatically by the control unit 30 without the user's control.

The server 100 is a computing device including at least one processor and a communication unit.

The control unit 30 includes a computing device including at least one processor and a communication unit. In one embodiment, the controller 30 includes hardware and software interfaces for controlling the surgical robot 34 .

The photographing device 36 includes at least one image sensor. That is, the imaging device 36 includes at least one camera device, and is used to photograph an object, that is, a surgical site. In an embodiment, the imaging device 36 includes at least one camera coupled to a surgical arm of the surgical robot 34 .

In an embodiment, the image captured by the photographing device 36 is displayed on the display 32 .

In one embodiment, the surgical robot 34 includes one or more surgical tools 38 capable of performing cutting, clipping, fixing, grabbing operations of the surgical site, and the like. The surgical tool 38 is used in combination with the surgical arm of the surgical robot 34 .

The controller 30 receives information necessary for surgery from the server 100 or generates information necessary for surgery and provides it to the user. For example, the controller 30 displays generated or received information necessary for surgery on the display 32 .

For example, the user performs robotic surgery by controlling the movement of the surgical robot 34 by manipulating the controller 30 while viewing the display 32 .

The server 100 generates information necessary for robotic surgery by using the medical image data of the object photographed in advance from the medical imaging device 10 , and provides the generated information to the controller 30 .

The controller 30 provides the information received from the server 100 to the user by displaying it on the display 32 , or controls the surgical robot 34 using the information received from the server 100 .

In one embodiment, the means that can be used in the medical imaging equipment 10 is not limited, for example, CT, X-Ray, PET, MRI, etc., various other medical image acquisition means may be used.

In the disclosed embodiment, the surgical image acquired by the imaging device 36 is transmitted to the controller 30 .

Hereinafter, when performing minimally invasive surgery, such as laparoscopic surgery and robotic surgery, information that can be obtained from the external space of the surgical subject is applied to the virtual body model, so that the internal space of the surgical subject in real time during the actual operation. A method of providing accurate location information of a surgical tool positioned on the upper surface will be described. First, the virtual body model will be described in detail.

47 is a flowchart schematically illustrating a method for generating a virtual body model according to an embodiment of the present invention.

Each of the steps shown in FIG. 47 is time-series performed by the server 100 or the controller 30 shown in FIG. 46 . Hereinafter, each step is described as being performed by a computer for convenience of explanation, but the subject performing each step is not limited to a specific device, and all or part of it may be performed by the server 100 or the control unit 30 . can

The computer may acquire the medical image data of the surgery subject taken by CT, MRI, PET, etc. (S100).

In one embodiment, it is important that the virtual body model provides the same virtual surgical state as the actual surgical state. Accordingly, the computer may calculate the photographing posture of the subject for surgery based on the surgical posture determined based on the location of the affected area or the type of surgery. Since the surgical posture may vary depending on the patient's location of the affected area (that is, the surgical site), the type of surgery, etc. Medical image data can be obtained.

The computer may perform 3D modeling based on the medical image data of the surgical subject to generate a virtual body model to match the surgical subject's body (S110). That is, the virtual body model may be 3D modeling data generated by performing 3D rendering on medical image data captured by a medical imaging apparatus such as CT, MRI, or PET.

For example, when using CT image data obtained as a specific Hounsfield Unit (HU) value by applying the same imaging posture as the surgical posture to a specific patient, the color (ie, gray scale), the density of each point can be calculated to generate 3D modeling data of the patient's body.

In addition, the computer may generate 3D modeling data by applying a correction algorithm that changes the image data taken under general shooting conditions to a body state at the time of a specific operation. For example, the computer can generate 3D modeling data by applying changes in the body due to gravity in a tilted state to medical image data of a patient lying down in a normal state or changes in the body formed as a relief model according to carbon dioxide injection. . In this case, the correction algorithm may be calculated by learning the learning data formed by matching the medical image data for general imaging conditions and physical conditions with the body state data during surgery for specific physical conditions and surgical conditions. The body state data during surgery may include image data obtained by photographing the shape or arrangement of internal organs in the body or image data obtained by photographing the body surface during actual surgery.

In one embodiment, when forming the relief model during laparoscopic surgery or robotic surgery, the computer may match the patient's medical image data with the body surface data during surgery to build the learning data to be learned. The body surface data may be taken by a medical staff before surgery. For example, the medical staff may acquire body surface data by photographing the abdominal surface deformed by injecting carbon dioxide before performing laparoscopic surgery or robotic surgery. As the computer learns the learning data, it can generate a correction algorithm (eg, a relief forming algorithm) that forms 3D modeling data in a general state into a relief model during laparoscopic surgery or robotic surgery. Thereafter, when the new patient's medical image data is obtained, the computer may generate a relief model by transforming the 3D modeling data generated based on the medical image data through a correction algorithm.

Through this, the computer can implement a 3D modeled virtual body model identical to the physical state during actual surgery for a specific patient. Accordingly, the medical staff can perform a simulation or rehearsal that provides a virtual surgery through the virtual body model. In addition, it is possible to provide a virtual image (ie, a 3D modeling image) through a virtual body model together with an actual surgical image during an actual operation. Accordingly, more accurate surgical information can be provided by reflecting and displaying information on body parts that are not displayed or difficult to identify in the actual surgical image on the virtual body model.

In general, in the case of minimally invasive surgery, such as laparoscopic surgery or robotic surgery, surgery is performed by inserting a surgical tool and a camera into the body of a patient (ie, a surgical target), that is, a surgical site. In this case, the operation is performed by grasping the state of the surgical site or the movement of the surgical tool through the surgical image captured by the camera that has entered the patient's body. However, since the field of view of the camera is limited, it is impossible to grasp information about surgical tools or objects inside the body located in an area outside the field of view of the camera. In particular, during surgery such as minimally invasive surgery, there is a restriction that the surgical tool does not move when the camera moves. For example, when the first surgical site is operated by the surgical robot, the camera photographs the first surgical site existing within the field of view of the camera and the surgical tool positioned therein. After that, when the operation needs to be performed by moving from the first surgical site to the second surgical site, the surgical robot moves the camera to the second surgical site, which is the target point, and then the surgical tool is moved. At this time, only the second surgical site is photographed in the field of view of the camera moved to the second surgical site, and the surgical tool is not photographed by the camera. Therefore, since location information about the surgical tool or information about the inside of the body according to the movement of the surgical tool cannot be grasped through the surgical image, the medical staff performing the operation cannot be provided with accurate surgical information.

In order to solve this problem, the present invention proposes a method that can provide accurate information of the position of the surgical tool and the object inside the body of the subject to be operated by applying a virtual body model during minimally invasive surgery such as laparoscopic surgery or robotic surgery. do.

48 is a flowchart illustrating a method of calculating position information of a surgical tool according to an embodiment of the present invention.

Each of the steps shown in FIG. 48 is time-series performed by the server 100 or the controller 30 shown in FIG. 46 . Hereinafter, each step is described as being performed by a computer for convenience of explanation, but the subject performing each step is not limited to a specific device, and all or part of it may be performed by the server 100 or the control unit 30 . can

In one embodiment, the present invention can be applied to the case of minimally invasive surgery such as laparoscopic surgery or robotic surgery. In this case, surgery is performed by inserting a surgical tool into the internal space of the patient's body. In one embodiment, the surgical tool may be configured to include an operating unit that is inserted into the body to perform a surgical operation, and an arm coupled to the operating unit and extending to the outside of the body. For example, the operation unit is a part capable of performing an operation such as approaching the surgical site and grabbing a target, cutting, moving, or suturing, and may be composed of various instruments according to the purpose of the surgical operation. The arm is connected to the operating unit to control the movement of the operating unit.

In actual minimally invasive surgery, the medical staff who performs the operation acquires the surgical image of the operating part of the surgical tool with a camera inserted into the body of the patient, and through this, the position or movement of the operating part of the surgical tool is identified. perform surgery However, as described above, since the field of view is limited according to the position or direction of the camera, information about the surgical tool inside the body is also limitedly obtained. Therefore, in the present invention, more accurate surgical information (i.e., location information of surgical tools).

Referring to FIG. 48 , the method for calculating position information of a surgical tool according to an embodiment of the present invention includes a computer based on sensing information obtained from a sensing device attached to a surgical tool to be inserted into an internal space of a subject for surgery. Calculating the position of the first point of the surgical tool with respect to the external space of the subject (S200), the surgical tool based on the position of the first point of the surgical tool through the virtual body model created in accordance with the physical condition of the subject to be operated on calculating the position of the second point of the surgical tool with respect to the internal space of the body of the surgery subject by reflecting the characteristic information of (S210), and the actual body of the surgery subject based on the position of the second point of the surgical tool with respect to the virtual body model It may include a step (S220) of providing location information of the surgical tool in the internal space. Hereinafter, detailed description of each step is described.

The computer may calculate the position of the first point of the surgical tool in the external space of the surgical subject based on the sensing information obtained from the sensing device attached to the surgical tool to be inserted into the internal space of the surgical subject (S200).

For convenience of explanation, the position of the first point of the surgical tool refers to spatial location information in which a specific point (eg, one end) of the arm visible in the external space from the inside of the body to the outside of the body to be operated is located. . In distinction from this, the position of the second point of the surgical tool refers to spatial location information in which a specific point (eg, the distal end) of the operating part inserted into the body of a subject to be operated is located.

In an embodiment, the sensing device may be installed on the arm of the surgical tool, and may transmit/receive sensing signals to sense position information, motion information, and the like of the surgical tool. For example, the sensing device may use an infrared measuring sensor, an electromagnetic sensor, an inertial measurement unit (IMU), a motion measuring sensor, or the like. If an infrared measuring sensor is installed on the arm of the surgical tool, the computer can determine the position of the surgical tool by acquiring a sensing signal from the infrared measuring sensor installed on the arm of the surgical tool through the infrared camera installed on the top of the operating table or on the surgical robot. have. Alternatively, when the electromagnetic sensor is installed in the arm of the surgical tool, the computer generates electromagnetic waves in the surgical space during surgery to obtain position information of the electromagnetic sensor to determine the location of the surgical tool. In this way, it is possible to obtain position information of the surgical tool (ie, the arm) in the space outside the body of the subject to be operated by using the sensing device installed on the arm of the surgical tool.

In addition, the computer may calculate the position of the first point of the surgical tool by using the reference position on the body surface of the surgical subject together with the sensing information obtained by the sensing device attached to the surgical tool. Here, the reference position refers to at least one specified point on the body surface of the surgery subject. For example, a central or representative point on the body may be determined as a reference position, or an insertion point into which a surgical tool is inserted during minimally invasive surgery may be determined as a reference position. As another example, a reference position may be acquired based on information sensed therefrom by attaching a position sensing device to at least one specified point on the external surface of the body of the subject to be operated on. The reference position may be expressed as coordinate information on a space outside the body (operation space).

In one embodiment, the computer acquires at least one reference position specified on the body surface of the subject to be operated on, and based on the obtained at least one reference position, sensing information of the surgical tool (ie, the operation acquired by the sensing device) The position of the first point of the surgical tool (ie, the arm) in the space outside the body can be derived using the position information of the tool. That is, the computer may calculate the coordinate information of the point at which the surgical tool (ie, the arm) is located on the external space of the body relative to the reference position.

The computer reflects the characteristic information of the surgical tool based on the location of the first point of the surgical tool through the virtual body model created in accordance with the physical condition of the patient to be operated on, and the second point of the surgical tool in the internal space of the patient's body The position can be calculated (S210).

As described above, the virtual body model is 3D modeling data generated based on the patient's medical image data, and is implemented in the same way as the physical state of the surgery target during the actual operation, so that AR (Augmented Reality) or MR (Mixed Reality) Using technology, it can be provided to medical staff together with actual surgical images during actual surgery.

The characteristic information of the surgical tool is information determined according to the type of surgical tool, the type of operation, etc. That is, the computer specifies the length of the specified surgical tool, the degree of freedom of the joint (eg, rotation angle, direction, etc.) when the surgical tool to be used is specified. ), unique information such as a degree of movement may be acquired as characteristic information.

In one embodiment, the computer may acquire characteristic information including at least one of length information and motion information of the surgical tool, and reflect the acquired characteristic information of the surgical tool in the virtual body model. At this time, the computer can derive the position of the second point of the surgical tool from the virtual body model by reflecting the characteristic information of the surgical tool in the virtual body model based on the position of the first point of the surgical tool calculated in step S200. . In other words, if the computer can know the length information and movement information for each of the operating unit and the arm, it can determine the positional relationship from the arm located in the external space of the body to the operating unit located in the internal space of the body. For example, the computer knows the length of the arm and the operating part, and grasps motion information such as the angle and the degree of inclination between the two that occurs according to the movement of the arm and the operating part, so that from the position of the first point of the surgical tool (that is, the arm) It is possible to relatively calculate the position of the second point of the surgical tool (ie, the operation unit). Here, in the motion information, as the movement between the first point and the second point occurs, the position of each of the first point and the second point is changed, and the position between the first point and the second point for grasping the positional relationship is changed. It may be information indicating an angle, a degree of inclination, and the like.

According to an embodiment, the motion information of the surgical tool may be grasped by installing an inertial measuring sensor or a motion measuring sensor in each of the arm and the operating unit. In this case, since movement information of the surgical tool can be sensed from each of the outside and the inside of the body, it can be more effective to calculate the positional relationship between the arm and the operating unit.

In addition, in acquiring the characteristic information of the surgical tool, when the surgical tool includes at least one joint for connecting the first point and the second point (for example, the surgical tool includes at least one operating part and at least one When configured including the arms of the joint, the computer uses the number of joints and joint driving information (eg, degree of freedom of each joint, degree of movement, etc.) as characteristic information. It can also be obtained in advance from the surgical robot. In this case, the computer may calculate the position of the second point from the position of the first point of the surgical tool by additionally reflecting the joint characteristic information in the virtual body model.

The computer may provide the position information of the surgical tool in the actual internal body space of the surgery subject based on the position of the second point of the surgical tool with respect to the virtual body model (S220).

In one embodiment, the computer provides the same environment as during actual surgery through a virtual image by reflecting the surgical tool on the virtual body model based on the first point position of the surgical tool (ie, the operating unit) calculated in step S210 can do. Accordingly, the medical staff can acquire the actual position of the surgical tool located inside the body of the subject to be operated on.

As described above, due to the limited field of view of the camera or the restrictions between the camera and the surgical tool during minimally invasive surgery, there may be cases where the surgical tool is not visible in the surgical image acquired by the camera inserted into the body of the patient. have. In this case, according to the present invention, since it is possible to provide the exact actual position of the surgical tool through the virtual body model without relying only on the surgical image, it is possible to induce the medical staff to perform more effective surgery.

In addition, according to the present invention, it is possible to derive the coordinate information in real space from the position sensing information of the surgical tool even without directly acquiring the coordinate system information of the surgical robot when performing surgery in connection with the surgical robot.

In addition, the computer can derive the positional relationship between the surgical tool (that is, the distal end of the operating part) and the internal body object by reflecting the surgical tool on the virtual body model, and determine the movement of the surgical tool based on this positional relationship or It is also possible to determine whether there is a collision between a tool and an object inside the body, and provide additional information about this. As described above, the internal body object may include a body part existing inside the body, an object introduced from the outside, an object created by itself, and the like. The body part may be an object existing inside the body, such as organs, blood vessels, bones, tendons, nervous tissue, and the like. The object introduced from the outside may be consumables necessary for surgery, such as gauze and clips. An object created by itself may include bleeding occurring in a body part.

49 is a diagram schematically showing the configuration of an apparatus 300 for performing a method for calculating position information of a surgical tool according to an embodiment of the present invention.

Referring to FIG. 49 , the processor 310 includes one or more cores (not shown) and a graphic processing unit (not shown) and/or a connection path (eg, a bus) for transmitting and receiving signals with other components. ) may be included.

The processor 310 according to an embodiment executes one or more instructions stored in the memory 320, and the method for generating the virtual body model described in relation to FIGS. 47 to 48 and the method for calculating the position information of the surgical tool carry out

For example, the processor 310 executes one or more instructions stored in the memory 320, and based on sensing information obtained from a sensing device attached to a surgical tool that is inserted into the internal space of the patient's body, the outside of the body of the surgery subject. The position of the first point of the surgical tool in the space is calculated, and through the virtual body model created in accordance with the physical condition of the subject to be operated, the surgical tool's characteristic information is reflected based on the position of the first point of the surgical tool to be operated. Calculate the position of the second point of the surgical tool with respect to the internal space of have.

Meanwhile, the processor 310 temporarily and/or permanently stores a signal (or data) processed inside the processor 310 . , not shown) may be further included. In addition, the processor 310 may be implemented in the form of a system on chip (SoC) including at least one of a graphic processing unit, a RAM, and a ROM.

The memory 320 may store programs (one or more instructions) for processing and controlling the processor 310 . Programs stored in the memory 320 may be divided into a plurality of modules according to functions.

The method for calculating the position information of the surgical tool according to an embodiment of the present invention described above may be implemented as a program (or application) to be executed in combination with a computer, which is hardware, and stored in a medium.

A method of calculating a camera position using an actual surgical image according to an embodiment of the present invention will be described in detail with reference to FIGS. 50 to 55 .

In this specification, "image" may mean multidimensional data composed of discrete image elements (eg, pixels in a 2D image and voxels in a 3D image). For example, the image may include a medical image of the object obtained by the CT imaging apparatus.

As used herein, an “object” may be a person or an animal, or a part or all of a person or an animal. For example, the object may include at least one of organs such as liver, heart, uterus, brain, breast, abdomen, and blood vessels.

As used herein, a “user” may be a medical professional, such as a doctor, a nurse, a clinical pathologist, or a medical imaging specialist, and may be a technician repairing a medical device, but is not limited thereto.

As used herein, "medical image data" is a medical image captured by a medical imaging device, and includes all medical images that can be implemented as a three-dimensional model of the body of an object. "Medical image data" may include computed tomography (CT) images, magnetic resonance imaging (MRI), positron emission tomography (PET) images, and the like.

As used herein, the term "virtual body model" refers to a model generated to match the actual patient's body based on medical image data. The "virtual body model" may be generated by modeling medical image data in 3D as it is, or may be corrected after modeling to be the same as during actual surgery.

As used herein, “virtual surgical data” refers to data including rehearsal or simulation actions performed on a virtual body model. “Virtual surgical data” may be image data on which rehearsal or simulation is performed on a virtual body model in a virtual space, or data recorded on a surgical operation performed on the virtual body model.

As used herein, “actual surgical data” refers to data obtained by actual medical staff performing surgery. "Actual surgical data" may be image data of a surgical site taken during an actual surgical procedure, or may be data recorded on a surgical operation performed in an actual surgical procedure.

As used herein, the term “computer” includes various devices capable of providing a result to a user by performing arithmetic processing. For example, computers include desktop PCs, notebooks (Note Books) as well as smart phones, tablet PCs, cellular phones, PCS phones (Personal Communication Service phones), synchronous/asynchronous A mobile terminal of International Mobile Telecommunication-2000 (IMT-2000), a Palm Personal Computer (PC), a Personal Digital Assistant (PDA), and the like may also be applicable. Also, when a head mounted display (HMD) device includes a computing function, the HMD device may be a computer. In addition, the computer may correspond to a server that receives a request from a client and performs information processing.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

50 is a schematic diagram of a system capable of performing robotic surgery according to an embodiment of the present invention.

Referring to FIG. 50 , the robotic surgery system includes a medical imaging device 10 , a server 100 , and a controller 30 , a display 32 , and a surgical robot 34 provided in the operating room. According to an embodiment, the medical imaging device 10 may be omitted from the robotic surgery system according to the disclosed embodiment.

In one embodiment, the surgical robot 34 includes a photographing device 36 and a surgical tool 38 .

In one embodiment, the robotic surgery is performed by the user using the control unit 30 to control the surgical robot (34). In one embodiment, the robotic surgery may be performed automatically by the control unit 30 without the user's control.

The server 100 is a computing device including at least one processor and a communication unit.

The control unit 30 includes a computing device including at least one processor and a communication unit. In one embodiment, the controller 30 includes hardware and software interfaces for controlling the surgical robot 34 .

The photographing device 36 includes at least one image sensor. That is, the imaging device 36 includes at least one camera device, and is used to photograph an object, that is, a surgical site. In an embodiment, the imaging device 36 includes at least one camera coupled to a surgical arm of the surgical robot 34 .

In an embodiment, the image captured by the photographing device 36 is displayed on the display 32 .

In one embodiment, the surgical robot 34 includes one or more surgical tools 38 capable of performing cutting, clipping, fixing, grabbing operations of the surgical site, and the like. The surgical tool 38 is used in combination with the surgical arm of the surgical robot 34 .

The controller 30 receives information necessary for surgery from the server 100 or generates information necessary for surgery and provides it to the user. For example, the controller 30 displays generated or received information necessary for surgery on the display 32 .

For example, the user performs robotic surgery by controlling the movement of the surgical robot 34 by manipulating the controller 30 while viewing the display 32 .

The server 100 generates information necessary for robotic surgery by using the medical image data of the object photographed in advance from the medical imaging device 10 , and provides the generated information to the controller 30 .

The controller 30 provides the information received from the server 100 to the user by displaying it on the display 32 , or controls the surgical robot 34 using the information received from the server 100 .

In one embodiment, the means that can be used in the medical imaging equipment 10 is not limited, for example, CT, X-Ray, PET, MRI, etc., various other medical image acquisition means may be used.

In the disclosed embodiment, the surgical image acquired by the imaging device 36 is transmitted to the controller 30 .

Hereinafter, a method of calculating a camera position using an actual surgical image will be described in detail with reference to the drawings.

51 is a flowchart illustrating a method of calculating a camera position using an actual surgical image according to an embodiment of the present invention.

Each of the steps shown in FIG. 51 is time-series performed by the server 100 or the controller 30 shown in FIG. 50 . Hereinafter, each step is described as being performed by a computer for convenience of explanation, but the subject of each step is not limited to a specific device, and all or part of it may be performed by the server 20 or the control unit 30 . can

Referring to FIG. 51 , in the method of calculating a camera position using an actual surgical image according to an embodiment of the present invention, a computer acquires a reference object from an actual surgical image captured by a camera that enters the inside of the body of a subject to be operated. Setting a reference position for the camera (S100), calculating the position change amount of the camera as the camera moves (S200), the current position of the camera based on the position change amount of the camera with respect to the reference position It may include the step of calculating (S300). Hereinafter, detailed description of each step is described.

The computer may obtain a reference object from an actual surgical image captured by a camera entering the body of the surgery subject and set a reference position for the camera (S100).

The actual surgical image refers to data obtained as actual medical staff perform surgery, and may be, for example, an image of an actual surgical scene actually performed by the surgical robot 34 . In addition, the actual surgical image may be a stereoscopic 3D image, and thus the actual surgical image may be a three-dimensional image, that is, an image having a sense of depth. Therefore, it is possible to accurately grasp the position of the surgical tool in the three-dimensional space through the depth map of the actual surgical image. In addition, the camera may be a device capable of capturing a 3D image, for example, a stereo camera. Accordingly, the actual surgical image may be taken as a 3D image by a stereo camera.

According to the embodiment, the depth information of the actual surgical image does not need to have a depth value for all pixels in the entire screen area, and it is sufficient if the depth value can be obtained for a minimum reference point that can sufficiently express geometrical features. For example, if there is no reference 3D data model (eg virtual body model), depth values may be required for all pixels, but if there is already a 3D data model (eg virtual body model), depth values for all pixels It doesn't have to have a value.

In one embodiment, when performing minimally invasive surgery such as laparoscopic surgery or robotic surgery, only a surgical tool and a camera are inserted into the surgical site of the subject (patient) to perform the surgery. In this case, the computer may acquire an image captured by a camera entering the patient's body as an actual surgical image. For example, the actual surgical image captures the process from when the patient enters the inside of the body to the surgical site inside the body, and the process from the start to the end of the action with the surgical tool on the surgical site inside the body. It may be image data.

As described above, the object (subject) may be an organ located inside the body of a subject for surgery, or a part other than the organ. In an embodiment of the present invention, the reference object may be a specific organ or a specific part or region that satisfies a predetermined condition among objects located inside the body of a subject to be operated on. That is, the reference object is easy to detect features from the image, exists in a fixed position inside the body, has little or no movement during surgery, does not change shape, is not affected by surgical tools, and is a medical image. It is possible to use an organ or a specific internal part that satisfies at least one of conditions, such as being able to be acquired even by data (eg, data photographed by CT, PET, etc.). For example, a portion that moves very little during surgery, such as a liver or abdominal disease, or a portion that can be obtained from medical image data, such as the stomach, esophagus, and gallbladder, may be determined as a reference object.

52 is a flowchart illustrating a process of setting a reference position for a camera according to an embodiment of the present invention. 3 is a detailed description of the operation process of step S100.

Referring to FIG. 52 , the computer may acquire a virtual image including a reference object from the virtual body model of the surgery subject ( S110 ).

Here, the virtual body model may be 3D modeling data generated based on medical image data obtained by photographing the inside of the patient's body in advance as described above. In addition, the virtual image may be an image of a viewpoint having the same information as that captured by a real camera in a virtual body model constructed in 3D.

In one embodiment, a camera that enters the inside of the patient's body during surgery captures all objects in the direction the camera is looking at. At this time, the computer may select an image of a desired viewpoint, that is, an image of a viewpoint including the reference object, from among the actual surgical images captured by such a camera. Thereafter, the computer may acquire a virtual image in the virtual body model corresponding to the actual surgical image of the viewpoint including the reference object.

The computer may match the reference object included in the virtual image with the reference object included in the actual surgical image (S120).

In an embodiment, registration between two images may be performed by extracting and comparing features of a reference object. For example, a feature of a reference object may be extracted by obtaining a depth map or applying an image segmentation technique of deep learning.

The computer may obtain a depth map for the reference object from the actual surgical image, and compare this with depth map information for the reference object obtained from the virtual image to determine whether the reference object between the two images is the same object. The depth map can be used to compare surface information of objects (organs). Depending on the object, it may be difficult to identify only the surface features. In this case, the computer may apply the image segmentation technique of deep learning to obtain a segmentation image for the reference object and perform registration between the two images. According to an embodiment, the computer may perform registration between two images by using the segmentation image information in combination with the depth map information. For example, when an object (organ) has a smooth surface feature, it is difficult to identify the object through the depth map, so image segmentation of deep learning can be applied together. In addition, when an object (organ) has a three-dimensional structure, image segmentation is applied to the surface features of the object because image segmentation of deep learning is performed in two dimensions, and a depth map can be applied to the three-dimensional structure.

In addition, since the virtual body model is generated including information on the type of object and location information for each object, this information can be configured as an image map. Accordingly, the computer may detect an object region expected to be matched in the virtual image based on the characteristic information of the reference object obtained from the actual surgical image. The computer may obtain depth information and segmentation information centering on the detected object area in the virtual image, and may configure it as a map image. The computer compares the map image information of the object obtained from the virtual image and the characteristic information (depth information and segmentation image information) of the reference object obtained from the actual surgical image to determine whether the reference object between the two images matches.

Meanwhile, in performing the registration between two images, if the registration is performed at a random position, it may take a long time to perform the registration. However, due to the characteristics of the surgical image, the position of the camera does not change abruptly, so there is not much change in the image. Therefore, the computer may acquire an actual surgical image for a predetermined point of view of the camera, acquire a virtual image of a virtual body model corresponding to this point of view, and perform registration between the two images. In one embodiment, since the surgical procedure is constant, the computer can recognize the position of a surgical tool or an organ (object) for the surgical procedure through deep learning, and through this, an actual surgical image can be obtained for a predetermined time point of the camera. . That is, the computer can find the reference position of the camera by recognizing the position of the reference object through deep learning and acquiring the actual surgical image at the recognized time point.

In addition, according to an embodiment, the computer first recognizes the position of the reference object through deep learning to acquire the actual surgical image of the viewpoint including the reference object, and then corresponds to the actual surgical image of the viewpoint including the reference object It is also possible to match the virtual image of the virtual body model at the point of view. At this time, the registration between the actual surgical image and the virtual image may be applied to the image segmentation technique of the depth map and / and deep learning as described above. In the case of this embodiment, first, the image of the viewpoint including the reference object in the actual surgical image is quickly found, the area is narrowed, and the position of the reference object can be accurately found through the registration between the actual operation image and the virtual image of the corresponding viewpoint.

The computer may obtain the position of the reference object from the virtual body model based on the matching result of the reference object between the two images, and set the reference position for the camera based on this (S130).

As described above, since the virtual body model is generated including the location information of the object, the location of the reference object photographed by the camera can be derived from the virtual body model. Accordingly, the computer may determine the reference position of the camera based on the position of the reference object.

Referring back to FIG. 51 , the computer may calculate the amount of change in the position of the camera as the camera moves ( S200 ).

Since the camera is inserted into the body of the subject to be operated and the operation process is photographed while moving, the position of the camera may be continuously changed. Therefore, after setting the reference position of the camera, it is possible to calculate the position of the camera during surgery in real time by calculating the amount of change according to the movement of the camera.

53 is a flowchart illustrating a process of calculating a position change amount of a camera according to an embodiment of the present invention. 53 is a detailed description of the operation process of step S200.

Referring to FIG. 53 , the computer may acquire a first image and a second image in which the movement of the camera is detected from the actual surgical image (S210). That is, the computer detects an image in which the camera moves from among the actual surgical images taken by the camera after setting the reference position.

To this end, it is necessary to detect an optical flow related to camera movement in an image and filter out optical flows that are not related to camera movement, for example, movement of a surgical tool, but it is very difficult to distinguish it from a general image. However, there is a limitation that the surgical tool does not move when the camera moves during surgery. Therefore, using these restrictions, only the image in which the camera moves can be effectively detected from the actual surgical image. That is, in the characteristic of an image with only camera movement, all parts of the image move in a certain direction. On the other hand, in the image in which the surgical tools and organs are also moved, the direction of movement is different for each part in the image. Therefore, using these features, an image with only camera movement can be obtained from the actual surgical image. In one embodiment, the computer may detect the first image and the second image in which the camera movement occurs from the actual surgical image through a technique such as deep learning by applying the characteristics of the surgical image as described above.

In this embodiment, it is considered that there is a restriction that the surgical tool does not move when the camera moves as described above during surgery, but this is only described as an example and the present invention is not limited thereto. According to the embodiment, a case in which both the camera and the surgical tool move may be considered, and even in this case, if the deep learning technique is applied, the first image and the second image in which the camera moves are detected from the actual surgical image. can do.

Next, the computer may detect a change between the images by matching the first image and the second image ( S220 ).

In an embodiment, the computer may extract and match feature points of each of the first image and the second image. For example, the computer may extract each feature point from each of the first image and the second image using an algorithm such as scale-invariant feature transform (SIFT). In this case, in addition to SIFT, various feature point extraction algorithms may be used. In addition, the computer may match the feature points of the first image with the feature points of the second image, and detect a change between the matched feature points. In this case, a feature point matching algorithm may be used, for example, brute force, fast approximate nearest neighbor (FLANN) search, or the like may be used. For example, the computer may construct a point cloud by calculating a depth map for each of the first image and the second image. In this case, the corresponding points in the point cloud for each of the two images may also be matched through feature point matching between the first image and the second image. Here, since feature point matching is performed in 2D coordinates, it is converted into 3D coordinates using depth maps for each of the first image and the second image, and matching in the point cloud between the two images based on the converted 3D coordinates. can Through the matching between the two images, it is possible to detect that the corresponding points of the point cloud for each of the two images move in the same direction. Accordingly, the computer can know the amount of change in the position of the camera based on the change in the position of the feature point (corresponding point) between the two images.

According to an embodiment, the computer may obtain a depth map through matching of a stereo image when constructing a point cloud for the first image and the second image, but this is only an example, and the first image and the A point cloud for the second image may be configured and matched. For example, in a limited situation, a depth map may be obtained using a single image. Since the size of the surgical tool is accurately known and the size of the surgical tool does not change during surgery, the characteristics of the camera can be kept constant. In this case, a depth value for a point of interest (feature point) may be obtained using a single image.

Next, the computer may calculate the amount of change in the position of the camera based on the change between images ( S230 ).

As an embodiment, as described above, since the computer can detect the amount of change in the position of the feature point between the two images, the amount of change can be estimated as the amount of change according to the movement of the camera.

Meanwhile, in calculating the amount of change in the position of the camera according to the embodiment, the computer may calculate the amount of change in the position of the camera based on the surgical tool, depending on whether the surgical tool is included in the actual surgical image.

In an embodiment, the computer may determine whether a surgical tool is included in the actual surgical image obtained as the camera moves. For example, the computer may determine whether a surgical tool is included in the first image acquired at the start of the movement of the camera and the second image acquired at the end of the movement of the camera. In this case, the computer may determine whether the surgical tool is present from the first image and the second image using the image recognition method.

When the surgical tool in the first image and the second image are included according to the determination result, the computer compares the surgical tool information in the first image with the surgical tool information in the second image, and the position of the camera based on the comparison result The amount of change can be calculated relatively. As described above, since there is a restriction that the surgical tool does not move when the camera moves during surgery, the surgical tool does not move in the images (first image and second image) acquired from the start of movement of the camera to the end of movement. and is fixed Therefore, if a surgical tool is included in the first image and the second image, the corresponding surgical tool is changed only by the movement of the camera. That is, the computer derives changes in the position, size, direction, etc. of the surgical tools included in each of the first image and the second image, and calculates the relative change in the position of the camera based on the change information between the surgical tools in each image. can

Alternatively, the computer may determine whether a surgical tool is included in at least one image among images acquired from the start of movement of the camera until the end of movement of the camera. For example, the computer may determine whether a surgical tool is included in all or a part of an image frame acquired when the camera starts moving, and determines whether a surgical tool is included in an image frame when the camera moves. Thereafter, by comparing the information of the surgical tool between the image frames including the surgical tool according to the determination result, the position change amount of the camera may be relatively calculated.

Alternatively, the computer may determine whether the surgical tool is included in at least one image frame among images acquired when the camera starts moving or when the camera moves, and calculates a relative change in the position of the camera therefrom. For example, when a surgical tool is included in the first image acquired when the camera starts moving, or the surgical tool is included in the second image acquired when the camera moves when the camera moves, the computer is based on the image including the surgical tool. It is also possible to calculate a relatively moving distance of the camera. That is, since the surgical tool is fixed when the camera is moved, by calculating the distance to the position where the camera is moved (that is, the start position of the camera or the end position of the camera) based on the position of the fixed surgical tool, It is also possible to relatively calculate the amount of change in the position of the camera.

In addition, if the surgical tool is not included in the images (eg, the first image and the second image) acquired from the start of the movement of the camera to the end of the movement of the camera as a result of the determination, the computer repeats the process of FIG. By doing so, it is possible to calculate the amount of change in the position of the camera based on the change between images.

Referring again to FIG. 51 , the computer may calculate the current position of the camera based on the amount of change in the position of the camera with respect to the reference position ( S300 ).

That is, since the amount of change in the position of the camera is calculated according to movement from the reference position, the current position of the camera may be finally calculated by applying the amount of change in the position of the camera in the reference position.

In one embodiment, the computer may calculate the current position of the camera by deriving a change between the first image and the second image through a rotation or translation matrix, and reflecting the derived value to the reference position. have.

According to an embodiment of the present invention, by deriving the current position of the camera, the computer may calculate the position of the surgical site or the position of the surgical tool of the subject currently undergoing surgery based on this.

In addition, according to an embodiment of the present invention, the computer derives the current position of the camera, and based on this, it is possible to match the surgical position in the virtual body model corresponding to the surgical position of the subject to be operated on. That is, by providing the actual surgical location to the virtual body model, it is effective in providing more accurate surgical guide information to medical staff currently undergoing surgery.

In addition, according to an embodiment of the present invention, since the camera may continuously move or move during the operation, the computer may acquire an actual surgical image including a new reference object photographed while the camera is moved or moved. . In this case, the computer may acquire a new reference object from the actual surgical image, and reset the reference position based on the acquired new reference object. Also, the computer may recalculate the current position of the camera based on the reset reference position. This may be applied in the same manner as described in steps S100 to S300 described above.

In one embodiment, when the actual surgical image including the new reference object is obtained, the computer may re-calculate the current position of the camera by resetting the reference position only when the new reference object corresponds to a preset reference object. For example, whenever a new reference object is captured and acquired according to the movement of the camera, the current position of the camera may be updated by performing the above-described steps S100 to S300, and a new object may be acquired based on a preset time period. In this case, the current position of the camera may be updated by performing the above-described steps S100 to S300.

On the other hand, when the reference position of the camera is initially set once and the current position of the camera is continuously calculated based on this, an error may be accumulated if the amount of change in the position of the camera cannot be accurately detected over time. However, in the case of resetting the reference position by acquiring a new reference object as in the above-described embodiment, even if an error occurs in the process of calculating the amount of change in the position of the camera, the current position of the camera is corrected through the reset of the reference position, so that a more accurate Surgical information can be provided.

54 is a diagram illustrating a position of a camera derived according to an embodiment of the present invention on a coordinate plane.

It is important to accurately specify the position of the surgical site and surgical tool during actual surgery. Therefore, in the present invention, in order to accurately provide the position of the surgical site and surgical tool in real time during the actual operation, the position of the camera is first calculated and the position of the surgical site and the surgical tool can be calculated again through this, and the method for this is previously described. has been specifically described.

That is, according to the method of FIGS. 51 to 53, as shown in FIG. 54, by setting a reference position through a specific organ (ie, reference object) of the patient, on the actual coordinate space of the camera (200, absolute coordinate space) A position 220 of can be derived.

In this case, it is possible to know what positional relationship the coordinate space 210 of the camera has on the actual coordinate space 200 . Therefore, it is possible to calculate the actual position of the surgical site and surgical tools based on the position of the camera. First, the relative position of the surgical site and the surgical tool is obtained based on the coordinate space 210 of the camera, and the actual position of the surgical site and the surgical tool can be derived by converting it into a positional relationship in the real coordinate space 200 again. .

55 is a diagram schematically showing the configuration of an apparatus 300 for performing a method of calculating a camera position using an actual surgical image according to an embodiment of the present invention.

Referring to FIG. 55 , the processor 310 includes one or more cores (not shown) and a graphic processing unit (not shown) and/or a connection path (eg, a bus) for transmitting and receiving signals with other components. ) may be included.

The processor 310 according to an embodiment executes one or more instructions stored in the memory 320, thereby performing a camera position calculation method using the actual surgical image described in relation to FIGS. 51 to 53 .

For example, the processor 310 executes one or more instructions stored in the memory 320 to obtain a reference object from an actual surgical image taken by a camera that enters the inside of the patient's body to obtain a reference position for the camera. setting, calculating the amount of change in the position of the camera as the camera moves, and calculating the current position of the camera based on the amount of change in the position of the camera with respect to the reference position.

Meanwhile, the processor 310 temporarily and/or permanently stores a signal (or data) processed inside the processor 310 . , not shown) may be further included. In addition, the processor 310 may be implemented in the form of a system on chip (SoC) including at least one of a graphic processing unit, a RAM, and a ROM.

The memory 320 may store programs (one or more instructions) for processing and controlling the processor 310 . Programs stored in the memory 320 may be divided into a plurality of modules according to functions.

The method for calculating a camera position using an actual surgical image according to an embodiment of the present invention described above may be implemented as a program (or application) to be executed in combination with a computer, which is hardware, and stored in a medium.

With reference to FIGS. 56 to 59, a method for providing position information of a surgical tool according to an embodiment of the present invention will be described in detail.

In this specification, "image" may mean multidimensional data composed of discrete image elements (eg, pixels in a 2D image and voxels in a 3D image). For example, the image may include a medical image of the object obtained by the CT imaging apparatus.

As used herein, an “object” may be a person or an animal, or a part or all of a person or an animal. For example, the object may include at least one of organs such as liver, heart, uterus, brain, breast, abdomen, and blood vessels.

As used herein, a “user” may be a medical professional, such as a doctor, a nurse, a clinical pathologist, or a medical imaging specialist, and may be a technician repairing a medical device, but is not limited thereto.

As used herein, "medical image data" is a medical image captured by a medical imaging device, and includes all medical images that can be implemented as a three-dimensional model of the body of an object. "Medical image data" may include computed tomography (CT) images, magnetic resonance imaging (MRI), positron emission tomography (PET) images, and the like.

As used herein, the term "virtual body model" refers to a model generated to match the actual patient's body based on medical image data. The "virtual body model" may be generated by modeling medical image data in 3D as it is, or may be corrected after modeling to be the same as during actual surgery.

As used herein, “virtual surgical data” refers to data including rehearsal or simulation actions performed on a virtual body model. “Virtual surgical data” may be image data on which rehearsal or simulation is performed on a virtual body model in a virtual space, or data recorded on a surgical operation performed on the virtual body model.

As used herein, “actual surgical data” refers to data obtained by actual medical staff performing surgery. "Actual surgical data" may be image data of a surgical site taken during an actual surgical procedure, or may be data recorded on a surgical operation performed in an actual surgical procedure.

In the present specification, the "coordinate system (coordinate information) of the surgical robot" is coordinate information used by the surgical robot itself, and may be a coordinate system independently set inside the surgical robot.

As used herein, "the coordinate system of the virtual body model (coordinate information)" refers to the coordinate information used on the virtual body model, and the "coordinate system of the virtual body model" refers to the "surgery The position can be adjusted by matching with the "coordinate system of the robot", and in this way, the coordinate system of the virtual body model matched with the coordinate system of the surgical robot is specifically referred to as "the modified coordinate system of the virtual body model". To distinguish this, the coordinate system of the virtual body model before matching with the coordinate system of the surgical robot is referred to as "the initial coordinate system of the virtual body model" or "the existing coordinate system of the virtual body model".

As used herein, the term “computer” includes various devices capable of providing a result to a user by performing arithmetic processing. For example, computers include desktop PCs, notebooks (Note Books) as well as smart phones, tablet PCs, cellular phones, PCS phones (Personal Communication Service phones), synchronous/asynchronous A mobile terminal of International Mobile Telecommunication-2000 (IMT-2000), a Palm Personal Computer (PC), a Personal Digital Assistant (PDA), and the like may also be applicable. Also, when a head mounted display (HMD) device includes a computing function, the HMD device may be a computer. In addition, the computer may correspond to a server that receives a request from a client and performs information processing.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

56 is a schematic diagram of a system capable of performing robotic surgery according to an embodiment of the present invention.

Referring to FIG. 56 , the robotic surgery system includes a medical imaging device 10 , a server 100 , and a controller 30 , a display 32 , and a surgical robot 34 provided in the operating room. According to an embodiment, the medical imaging device 10 may be omitted from the robotic surgery system according to the disclosed embodiment.

In one embodiment, the surgical robot 34 includes a photographing device 36 and a surgical tool 38 .

In one embodiment, the robotic surgery is performed by the user using the control unit 30 to control the surgical robot (34). In one embodiment, the robotic surgery may be performed automatically by the control unit 30 without the user's control.

The server 100 is a computing device including at least one processor and a communication unit.

The control unit 30 includes a computing device including at least one processor and a communication unit. In one embodiment, the controller 30 includes hardware and software interfaces for controlling the surgical robot 34 .

The photographing device 36 includes at least one image sensor. That is, the imaging device 36 includes at least one camera device, and is used to photograph an object, that is, a surgical site. In an embodiment, the imaging device 36 includes at least one camera coupled to a surgical arm of the surgical robot 34 .

In an embodiment, the image captured by the photographing device 36 is displayed on the display 32 .

In one embodiment, the surgical robot 34 includes one or more surgical tools 38 capable of performing cutting, clipping, fixing, grabbing operations of the surgical site, and the like. The surgical tool 38 is used in combination with the surgical arm of the surgical robot 34 .

The controller 30 receives information necessary for surgery from the server 100 or generates information necessary for surgery and provides it to the user. For example, the controller 30 displays generated or received information necessary for surgery on the display 32 .

For example, the user performs robotic surgery by controlling the movement of the surgical robot 34 by manipulating the controller 30 while viewing the display 32 .

The server 100 generates information necessary for robotic surgery by using the medical image data of the object photographed in advance from the medical imaging device 10 , and provides the generated information to the controller 30 .

The controller 30 provides the information received from the server 100 to the user by displaying it on the display 32 , or controls the surgical robot 34 using the information received from the server 100 .

In one embodiment, the means that can be used in the medical imaging equipment 10 is not limited, for example, CT, X-Ray, PET, MRI, etc., various other medical image acquisition means may be used.

In the disclosed embodiment, the surgical image acquired by the imaging device 36 is transmitted to the controller 30 .

Hereinafter, when performing minimally invasive surgery such as laparoscopic surgery and robotic surgery, information on surgical tools and objects (eg, organs, blood vessels, etc.) inside the body of a subject to be operated are derived and matched to a virtual body model. A method for providing surgical information in real time during surgery will be described. First, the virtual body model will be described.

57 is a flowchart schematically illustrating a method for generating a virtual body model according to an embodiment of the present invention.

Each of the steps shown in FIG. 57 is time-series performed by the server 100 or the controller 30 shown in FIG. 56 . Hereinafter, each step is described as being performed by a computer for convenience of explanation, but the subject performing each step is not limited to a specific device, and all or part of it may be performed by the server 100 or the control unit 30 . can

The computer may acquire the medical image data of the surgery subject taken by CT, MRI, PET, etc. (S100).

In one embodiment, it is important that the virtual body model provides the same virtual surgical state as the actual surgical state. Accordingly, the computer may calculate the photographing posture of the subject for surgery based on the surgical posture determined based on the location of the affected area or the type of surgery. Since the surgical posture may vary depending on the patient's location of the affected area (that is, the surgical site), the type of surgery, etc. Medical image data can be obtained.

The computer may perform 3D modeling based on the medical image data of the surgical subject to generate a virtual body model to match the surgical subject's body (S110). That is, the virtual body model may be 3D modeling data generated by performing 3D rendering on medical image data captured by a medical imaging apparatus such as CT, MRI, or PET.

For example, when using CT image data obtained as a specific Hounsfield Unit (HU) value by applying the same imaging posture as the surgical posture to a specific patient, the color (ie, gray scale), the density of each point can be calculated to generate 3D modeling data of the patient's body.

In addition, the computer may generate 3D modeling data by applying a correction algorithm that changes the image data taken under general shooting conditions to a body state at the time of a specific operation. For example, the computer can generate 3D modeling data by applying changes in the body due to gravity in a tilted state to medical image data of a patient lying down in a normal state or changes in the body formed as a relief model according to carbon dioxide injection. . In this case, the correction algorithm may be calculated by learning the learning data formed by matching the medical image data for general imaging conditions and physical conditions with the body state data during surgery for specific physical conditions and surgical conditions. The body state data during surgery may include image data obtained by photographing the shape or arrangement of internal organs in the body or image data obtained by photographing the body surface during actual surgery.

In one embodiment, when forming the relief model during laparoscopic surgery or robotic surgery, the computer may match the patient's medical image data with the body surface data during surgery to build the learning data to be learned. The body surface data may be taken by a medical staff before surgery. For example, the medical staff may acquire body surface data by photographing the abdominal surface deformed by injecting carbon dioxide before performing laparoscopic surgery or robotic surgery. As the computer learns the learning data, it can generate a correction algorithm (eg, a relief forming algorithm) that forms 3D modeling data in a general state into a relief model during laparoscopic surgery or robotic surgery. Thereafter, when the new patient's medical image data is obtained, the computer may generate a relief model by transforming the 3D modeling data generated based on the medical image data through a correction algorithm.

Through this, the computer can implement a 3D modeled virtual body model identical to the physical state during actual surgery for a specific patient. Accordingly, the medical staff can perform a simulation or rehearsal that provides a virtual surgery through the virtual body model. In addition, it is possible to provide a 3D modeling image through a virtual body model together with an actual surgical image during an actual operation. Accordingly, it is possible to provide more accurate surgical information by displaying information on body parts that are not displayed or difficult to identify in the actual surgical image on the virtual body model.

In general, in the case of minimally invasive surgery, such as laparoscopic surgery or robotic surgery, surgery is performed by inserting a surgical tool and a camera into the body of a patient (ie, a surgical target), that is, a surgical site. In this case, the operation is performed by grasping the state of the surgical site or the movement of the surgical tool through the surgical image captured by the camera that enters the patient's body. However, since the field of view of the camera is limited, it is impossible to grasp information about surgical tools or objects inside the body located in an area outside the field of view of the camera. In particular, during surgery such as minimally invasive surgery, there is a restriction that the surgical tool does not move when the camera moves. For example, when the first surgical site is operated by the surgical robot, the camera photographs the first surgical site existing within the field of view of the camera and the surgical tool positioned therein. Afterwards, when the operation needs to be performed by moving from the first surgical site to the second surgical site, the camera moves to the second surgical site, which is the target point, by the surgical robot, and then the surgical tool is moved. At this time, only the second surgical site is photographed in the field of view of the camera moved to the second surgical site, and the surgical tool is not photographed by the camera. Therefore, since location information about the surgical tool or information about the inside of the body according to the movement of the surgical tool cannot be grasped through the surgical image, the medical staff performing the operation cannot be provided with accurate surgical information.

In order to solve this problem, the present invention proposes a method of providing accurate information on the position of a surgical tool and an object inside the body of a subject for surgery by applying a virtual body model during minimally invasive surgery such as laparoscopic surgery or robotic surgery.

Although it will be described in detail below, according to the present invention, after matching the coordinate system of the surgical robot itself and the initial coordinate system of the virtual body model based on a specific reference point, the position changes as the operation proceeds using the matched two coordinate systems. The position of the surgical tool is specified and the relationship with the object inside the body can be grasped based on this.

58 is a flowchart illustrating a method for providing location information of a surgical tool according to an embodiment of the present invention.

Each of the steps shown in FIG. 58 is performed in time series by the server 100 or the controller 30 shown in FIG. 56 . Hereinafter, each step is described as being performed by a computer for convenience of explanation, but the subject performing each step is not limited to a specific device, and all or part of it may be performed by the server 100 or the control unit 30 . can

Referring to FIG. 58, the method for providing position information of a surgical tool according to an embodiment of the present invention includes: a computer acquiring coordinate information of a surgical robot including a surgical tool based on a reference point of a subject for surgery (S200); Step (S210) of matching the coordinate information of the virtual body model generated in accordance with the body state of the surgery target with the coordinate information of the surgical robot, in the virtual body model corresponding to the position of the surgical tool obtained from the coordinate information of the surgical robot. It may include calculating the position of the surgical tool (S220). Hereinafter, detailed description of each step is described.

The computer may acquire coordinate information (coordinate system) of the surgical robot including the surgical tool based on the reference point of the surgical subject (S200).

Specifically, the computer may specify a reference point on the body surface of the subject to be operated on, and obtain coordinate information of the surgical robot set based on the specified reference point. The reference point refers to a specific point on the body surface of a subject to be operated on, for example, a central point on the body or a representative point may be determined as the reference point.

In one embodiment, the computer may specify a reference point from the body surface of the surgical subject using a marker that is responsive to a particular light source. Alternatively, an identification mark (ie, a reference point) may be displayed at a specific point on the body surface by projecting light such as infrared light through a projector installed on the top of the operating table or a surgical robot during surgery. Thereafter, the surgical robot may detect a point corresponding to the marker or identification mark displayed on the body surface, and set the coordinate space (coordinate system) of the surgical robot based on this. The computer may acquire coordinate space information of the surgical robot set based on the reference point from the surgical robot. At this time, the coordinate space (coordinate system) of the surgical robot is a coordinate system in which the initial coordinate space of the surgical robot is set as the center (eg, origin) of the operation target's reference point as the actual operation on the surgical subject (real patient) begins. say That is, it may be a coordinate space in which the reference point of the surgical subject and the initial coordinate space of the surgical robot are matched.

In another embodiment, the computer may display an identification mark (ie, a reference point) on the body surface of the surgical subject using a 3D projection method. During the operation, the computer projects a grid pattern (grid) at multiple intervals through a projector installed on the top of the operating table or on the surgical robot, and displays it on the surface of the patient's body through the camera installed on the top of the operating table or on the surgical robot (i.e., the surgical patient). It is possible to photograph a grid pattern (deformed by the shape of the body surface). Thereafter, the computer may recognize the shape of the body surface of the surgical subject as 3D stereoscopic, determine a specific point as a reference point based on the grid pattern displayed on the body surface, and set the coordinate space (coordinate system) of the surgical robot based on this. The computer may acquire coordinate space information of the surgical robot set based on the reference point of the surgical robot.

Meanwhile, the computer may acquire coordinate information (coordinate system) of the surgical robot including the surgical tool directly from the surgical robot. That is, since the surgical robot has its own coordinate information, the computer may obtain coordinate information of the surgical robot including the surgical tool directly from the surgical robot.

The computer may acquire initial coordinate information (coordinate system) of the virtual body model generated in accordance with the physical condition of the subject to be operated on and match it with coordinate information (coordinate system) of the surgical robot (S210). As described above, the virtual body model is 3D modeling data generated based on the patient's medical image data, and is implemented in the same way as the physical state of the surgery target during the actual operation, so that AR (Augmented Reality) or MR (Mixed Reality) Using technology, it can be provided to medical staff together with actual surgical images during actual surgery.

In one embodiment, first, the computer may acquire a reference point of the surgery subject from the virtual body model. For example, the computer may set the same point as the reference point of the surgery subject used when setting the coordinate information of the surgical robot in step S200 as the reference point of the surgery subject in the virtual body model. Alternatively, the computer may set a specific point on the body surface of the surgery subject as a reference point in the virtual body model.

Thereafter, the computer may match the initial coordinate information of the virtual body model with the coordinate information of the surgical robot based on the reference point in the virtual body model. For example, by matching the reference point in the virtual body model and the reference point in the surgical robot, the coordinate space (coordinate system) of the surgical robot and the coordinate space (coordinate system) in the virtual body model can be mapped to each other. Therefore, the computer can derive the transformed coordinate information of the virtual body model centered on the reference point of the surgery subject by mapping the coordinate information of the surgical robot to the initial coordinate information of the virtual body model.

The computer may calculate the position of the surgical tool in the virtual body model corresponding to the position of the surgical tool obtained from the coordinate information of the surgical robot based on the mutual coordinate information matched in step S210 (S220). Here, the coordinate information of the surgical robot means coordinate information set by matching the initial coordinate space of the surgical robot with the reference point of the operation target as described above.

In one embodiment, the computer may acquire location information of a surgical tool located inside the body of a surgical subject during actual surgery, based on coordinate information of the surgical robot, and a virtual body mapped with coordinate information of the surgical robot. It can be calculated by converting it into coordinate information in the model. Thereafter, the computer can provide a virtual image identical to the actual surgical environment by converting the coordinate information in the virtual body model and reflecting the surgical tool on the virtual body model based on the calculated position of the surgical tool. Therefore, the medical staff performing the operation can grasp the actual position of the surgical tool located inside the body of the surgery target from the virtual body model.

In addition, according to an embodiment of the present invention, the computer can derive a positional relationship between an object inside the body of the subject to be operated and the surgical tool in the virtual body model based on the position of the surgical tool in the virtual body model, It is possible to provide various information during surgery based on the positional relationship.

Here, the object inside the body of the surgery target may include a body part existing inside the body, an object introduced from the outside, an object generated by itself, and the like. The body part may be an object existing inside the body, such as organs, blood vessels, bones, tendons, nervous tissue, and the like. The object introduced from the outside may be consumables necessary for surgery, such as gauze and clips. An object created by itself may include bleeding occurring in a body part.

In one embodiment, the computer may determine whether an object inside the body of the surgery subject collides with the surgical tool in the virtual body model based on the position of the surgical tool in the virtual body model, and may provide a determination result. In other words, since the virtual body model is created including information on the position, size, shape, etc. of each body part for the subject to be operated on, it is possible to know the position information of the surgical tool that does not exist in a fixed position during actual surgery and that moves. If possible, the positional relationship between the object inside the body and the surgical tool can be calculated through the virtual body model. Through this, the computer determines in real time whether the surgical tool is close to or in contact with an object inside the body in real time during surgery, and whether a collision occurs when the surgical tool is moved in a specific direction. It can detect dangerous situations and provide guide information for them.

In another embodiment, when the state of the object inside the body of the subject to be operated is changed during surgery by the surgical robot, the computer reflects the changed state of the object inside the body in the virtual body model, and uses this virtual body model to determine the position of the surgical tool It can also be determined whether a collision with an object inside the body is based on the basis. For example, during surgery by a surgical robot, when an organ is lifted with a surgical tool (e.g., forceps) or a surgical operation such as cutting a blood vessel is performed, the position, size, shape, etc. of the organ may change. do. In this case, since information on surgical tools and internal body objects (eg, organs, etc.) is changed, a dangerous situation such as a collision situation or a sudden situation may occur. Therefore, the computer can reflect the operation process during actual surgery through the following process for the virtual body model or the simulation performance result. That is, the computer can provide accurate information on whether or not a dangerous situation is present by recognizing the position of the surgical tool and the changed position of the internal body object by reflecting the state of the internal body object changed during surgery in the virtual body model.

In another embodiment, when it is necessary to move the surgical tool during surgery by the surgical robot, the computer calculates a position change according to the movement of the surgical tool from the coordinate information of the virtual body model, and according to the calculated position change, the virtual body model can be used to guide the movement of surgical tools. For example, when it is necessary to move from the first surgical site to the second surgical site during surgery by a surgical robot, the camera moves to the second surgical site and then moves to the second surgical site due to the limitations of robotic surgery as described above. The tool moves. At this time, since the surgical tool is out of the camera's field of view, the medical staff performing the operation cannot determine the position of the surgical tool through the actual surgical image. Therefore, when moving the surgical tool to the second surgical site, a dangerous situation such as a collision or a sudden situation may occur. In this case, when the computer moves the surgical tool from the first surgical site to the second surgical site, the computer may acquire information on the internal body object existing on the movement path using the virtual body model. That is, the computer may calculate a change in position according to the movement path of the surgical tool, and calculate a positional relationship with an object inside the body in the virtual body model based on the change in position. By using this positional relationship, it is possible to determine whether a collision occurs due to movement of the surgical tool, and provide a determination result. For example, the result of simulating the movement of the surgical tool through the virtual body model may be provided.

Meanwhile, according to an embodiment, as described above, the computer may acquire coordinate information of the surgical robot itself including the surgical tool directly from the surgical robot. In this case, even if the camera moves (moves from the first surgical site to the second surgical site) and then the surgical tool moves (moves from the first surgical site to the second surgical site), the surgical robot can Because it can be used, it is possible to directly grasp the position of the surgical tool according to the movement of the surgical tool. That is, the computer directly acquires the position of the surgical tool from the surgical robot, and applies it to the virtual body model based on this to determine whether or not the surgical tool collides with an object inside the body according to the movement path of the surgical tool.

According to the method for providing position information of the surgical tool of the present invention described above, the position of the surgical tool invisible through the surgical image (screen) can be accurately identified, and this can be expressed in the virtual body model. Therefore, based on this, the part that is not visible through the surgical image can be reflected and displayed on the actual surgical image by using the virtual body model.

In one embodiment, the computer generates data (eg, virtual image data, etc.) including the same information as the actual surgery by reflecting the surgical tool on the virtual body model based on the location of the surgical tool acquired through the virtual body model. can do. At this time, if there is a surgical tool (eg, a part of the surgical tool) that is not visible through the actual surgical image due to the camera's field of view, the computer adjusts the camera's field of view on the virtual body model to perform the invisible surgery through the actual surgical image. It is possible to acquire virtual image data including a tool, and display it by adding it to the actual surgical image.

In addition, according to the method of providing position information of a surgical tool of the present invention described above, the position of an invisible surgical tool can be accurately calculated through a surgical image. Conflicts between them can be avoided. For example, in the case of replacing or wiping a surgical tool, the surgical tool is removed from the inside of the body and reinserted. At this time, the camera is zoomed in and the surgical tool is moving slowly, so It takes a considerable amount of time for it to enter the field of view of the camera. In this case, the position of the surgical tool that is not in the field of view of the camera can be realized and displayed in the form of a mini map. Therefore, even if the surgical tool is not in the field of view of the camera, the medical staff can determine the location of the surgical tool through the mini-map, and thus, it is possible to prevent a dangerous situation by determining whether there is a collision between the surgical tools or between the surgical tools and the organs.

In addition, according to the embodiments of the present invention described above, based on the location of the surgical tool in the virtual body model, it is determined whether a collision between a surgical tool and an object inside the body, or whether a collision between the surgical tools, etc. is in a dangerous situation. provide information. In providing information on whether such a dangerous situation is present, it may be displayed on the screen of the user (eg, medical staff such as doctors and nurses, or users other than medical staff) as an embodiment. In another embodiment, it may be displayed on the screen used by the assistant personnel. Here, the auxiliary personnel refers to personnel who do not perform actual surgery (eg, perform direct medical actions using a surgical robot) but are in charge of auxiliary operations around the patient. It may be an assistant (eg, a nurse, etc.) who performs an auxiliary operation of the surgical robot, such as taking it out and inserting it to the outside. In another embodiment, when the medical staff operates the surgical robot, a stimulus such as a haptic may be provided to a device such as a controller operated by the medical staff to inform whether the medical staff is in a dangerous situation. Alternatively, medical staff may be provided with information about a dangerous situation during an actual operation by wearing a device with a built-in haptic function provided separately from a manipulation device such as a controller.

59 is a diagram schematically showing the configuration of an apparatus 300 for performing a method for providing position information of a surgical tool according to an embodiment of the present invention.

Referring to FIG. 59 , the processor 310 includes one or more cores (not shown) and a graphic processing unit (not shown) and/or a connection path (eg, a bus) for transmitting and receiving signals with other components. ) may be included.

The processor 310 according to an embodiment executes one or more instructions stored in the memory 320, and a method of generating a virtual body model described in relation to FIGS. 57 to 58 and a method of providing location information of a surgical tool carry out

For example, the processor 310 obtains coordinate information of a surgical robot including a surgical tool based on the reference point of the surgical subject by executing one or more instructions stored in the memory 320, and conforms to the physical condition of the surgical subject. It is possible to match the coordinate information of the generated virtual body model with the coordinate information of the surgical robot, and calculate the position of the surgical tool in the virtual body model corresponding to the position of the surgical tool obtained from the coordinate information of the surgical robot.

Meanwhile, the processor 310 temporarily and/or permanently stores a signal (or data) processed inside the processor 310 . , not shown) may be further included. In addition, the processor 310 may be implemented in the form of a system on chip (SoC) including at least one of a graphic processing unit, a RAM, and a ROM.

The memory 320 may store programs (one or more instructions) for processing and controlling the processor 310 . Programs stored in the memory 320 may be divided into a plurality of modules according to functions.

The method for providing location information of a surgical tool according to an embodiment of the present invention described above may be implemented as a program (or application) and stored in a medium to be executed in combination with a computer, which is hardware.

Hereinafter, various embodiments of a technique for performing learning based on a surgical image and recognizing a surgical operation from a surgical image using the learning result are disclosed.

In one embodiment, a learning-based surgical operation recognition method performed by a computer, comprising: acquiring a surgical image; recognizing a surgical operation by learning an image frame in the surgical image; and extracting an image frame set from among the image frames in the surgical image based on the recognized surgical operation, and deriving the meaning of the surgical operation through learning. A detailed description thereof will be described later with reference to FIGS. 60 to 68 .

In another embodiment, as a learning-based surgical motion recognition method performed by a computer, the method comprising: acquiring a surgical image sequence; performing deep learning-based learning based on loss information for the surgical image sequence; and recognizing a surgical operation for the surgical image sequence based on the learning. A detailed description thereof will be described later with reference to FIGS. 69 to 73 .

With reference to FIGS. 60 to 68, a learning-based surgical motion recognition method according to an embodiment of the present invention will be described in detail.

60 is a schematic diagram of a system capable of performing robotic surgery according to an embodiment of the present invention.

Referring to FIG. 60 , the robotic surgery system includes a medical imaging device 10 , a server 100 , and a controller 30 , a display 32 , and a surgical robot 34 provided in the operating room. According to an embodiment, the medical imaging device 10 may be omitted from the robotic surgery system according to the disclosed embodiment.

In one embodiment, the surgical robot 34 includes a photographing device 36 and a surgical tool 38 .

In one embodiment, the robotic surgery is performed by the user using the control unit 30 to control the surgical robot (34). In one embodiment, the robotic surgery may be performed automatically by the control unit 30 without the user's control.

The server 100 is a computing device including at least one processor and a communication unit.

The control unit 30 includes a computing device including at least one processor and a communication unit. In one embodiment, the controller 30 includes hardware and software interfaces for controlling the surgical robot 34 .

The photographing device 36 includes at least one image sensor. That is, the imaging device 36 includes at least one camera device, and is used to photograph an object, that is, a surgical site. In an embodiment, the imaging device 36 includes at least one camera coupled to a surgical arm of the surgical robot 34 .

In an embodiment, the image captured by the photographing device 36 is displayed on the display 340 .

In one embodiment, the surgical robot 34 includes one or more surgical tools 38 capable of performing cutting, clipping, fixing, grabbing operations of the surgical site, and the like. The surgical tool 38 is used in combination with the surgical arm of the surgical robot 34 .

The controller 30 receives information necessary for surgery from the server 100 or generates information necessary for surgery and provides it to the user. For example, the controller 30 displays generated or received information necessary for surgery on the display 32 .

For example, the user performs robotic surgery by controlling the movement of the surgical robot 34 by manipulating the controller 30 while viewing the display 32 .

The server 100 generates information necessary for robotic surgery by using the medical image data of the object photographed in advance from the medical imaging device 10 , and provides the generated information to the controller 30 .

The controller 30 provides the information received from the server 100 to the user by displaying it on the display 32 , or controls the surgical robot 34 using the information received from the server 100 .

In one embodiment, the means that can be used in the medical imaging equipment 10 is not limited, for example, CT, X-Ray, PET, MRI, etc., various other medical image acquisition means may be used.

In the disclosed embodiment, the surgical image acquired by the imaging device 36 is transmitted to the controller 30 .

In an embodiment, the controller 30 may segment a surgical image acquired during surgery in real time.

In an embodiment, the control unit 30 transmits a surgical image to the server 100 during surgery or after surgery is completed.

The server 100 may receive and analyze the surgical image.

The server 100 learns and stores at least one model for analyzing the surgical image.

The server 100 uses training data to learn at least one model, and the training data includes, but is not limited to, a surgical image and information on a surgical image.

Hereinafter, a method for the server 100 to learn a surgical image and recognize a surgical operation based on the learning will be described. However, the embodiments disclosed below are not applicable only in connection with the robotic surgery system shown in FIG. It can also be applied to embodiments.

In addition, for convenience of description, it will be described below that the "computer" performs the surgical image learning method and the learning-based surgical motion recognition method according to the embodiment disclosed in the present specification. “Computer” may mean the server 100 of FIG. 60 , but is not limited thereto and may be used to encompass a device capable of performing computing processing.

61 is a flowchart illustrating a learning-based surgical motion recognition method according to an embodiment of the present invention.

Referring to FIG. 61 , the learning-based surgical motion recognition method according to an embodiment of the present invention includes a computer acquiring a surgical image (S100), learning a plurality of image frames included in the surgical image to perform a surgical operation. It may include a step of recognizing (S110), extracting a set of consecutive image frames from among the plurality of image frames based on the recognized surgical operation, and deriving the meaning of the surgical operation through learning (S120). . Hereinafter, detailed description of each step is described.

The computer may acquire a surgical image (S100).

Here, the surgical image may be an actual surgical image or a virtual image for simulation. In one embodiment, the actual surgical image refers to data obtained as an actual medical staff performs an operation, and may be, for example, an image obtained by photographing an actual operation scene actually performed by the surgical robot 34 . That is, the actual surgical image is data recorded about the surgical site and surgical operation in the actual surgical process. The virtual image for simulation refers to a simulation image generated based on a medical image captured by the medical imaging device 10 , and may be, for example, a simulation model generated by modeling a medical image of a real patient in three dimensions. In this case, as rehearsal or simulation is performed on the simulation model in a virtual space, a virtual surgical image may be generated. Accordingly, the virtual image may be data recorded on the surgical site and surgical operation in the surgical procedure performed on the simulation model.

In addition, the surgical image may include one or more image frames. Each image frame may include a part of a body part of an object (eg, a patient), that is, a surgical site. In addition, each image frame may include not only the surgical site of the object, but also surgical tools, consumables necessary for surgery, and the like. In other words, the surgical image refers to data composed of image frames in which the surgical operation according to time in the surgical procedure is recorded for each scene (scene).

The computer may recognize the surgical operation by learning the image frame in the acquired surgical image (S110).

In one embodiment, the computer calculates an attribute for each image frame in the surgical image using a pre-generated motion recognition learning model, and a surgical operation for each image frame in the surgical image based on the calculated attribute can be recognized Here, the motion recognition learning model is a model trained for motion recognition using an image frame included in the surgical image as learning data, and may be a model learned using various learning methods. For example, the learning method may use a machine learning method such as supervised learning, unsupervised learning, reinforcement learning, for example, a deep learning-based convolutional neural network (CNN) may be used.

As described above, each image frame is an image in which a surgical operation is recorded, and includes information related to the surgical operation. In other words, if information related to the surgical operation included in the image frame is used, it is possible to determine which surgical operation is recorded in each image frame during the operation.

Accordingly, in order to recognize a surgical operation from each image frame of the surgical image, the computer may first extract information related to the surgical operation, that is, an attribute from each image frame. Attribute is information of an object for identifying a surgical operation, for example, object information such as a surgical site, type of operation, surgical tool, article used during surgery, location information, direction information or motion information of an object, and image Camera information such as a viewpoint, a direction, and a movement of the camera may be included as object information. As an example, attributes can be defined as shown in Table 1 below.

In one embodiment, the computer may store predefined attribute information. Attribute information as shown in Table 1 can be derived based on the result of learning using image frames as training data. Therefore, when the computer uses the motion recognition learning model to input the surgical image, it can calculate the attribute information for each image frame included in the surgical image as output. For example, when the computer inputs the first image frame of the surgical image to the motion recognition learning model, it extracts at least one attribute information (that is, object information), and outputs a binary value defined in response to each extracted attribute. can be calculated. The computer may recognize the surgical operation included in the first image frame based on attribute information (eg, binary value data) calculated for the first image frame.

The computer may extract an image frame set from among the image frames included in the surgical image based on the recognized surgical operation for each image frame, and derive the meaning of the surgical operation through learning (S120).

In one embodiment, the computer extracts an image frame set expressing a series of surgical operations using a pre-generated meaning derivation learning model based on the attributes calculated for each of the image frames in the surgical image, and performs a series of operations The meaning of the surgical operation can be derived based on the detailed surgical operation corresponding to the operation. Here, the meaning derivation learning model is a model learned to derive the meaning of an operation using a set of image frames representing a series of surgical operations as learning data, and may be a model learned using various learning methods. For example, the learning method may use a machine learning method such as supervised learning, unsupervised learning, reinforcement learning, for example, a deep learning-based recurrent neural network (RNN) may be used.

As described above, the surgical image includes continuous image frames in which the surgical process is recorded over time. Therefore, if it is possible to determine what kind of motion each image frame expresses, it is possible to find related image frames expressing one motion, and based on these related image frames, it means what kind of surgical operation is being performed in the entire surgical process. can figure out That is, in the present invention, the related image frames are referred to as an image frame set. For example, since the surgical image is composed of consecutive image frames according to time, the associated image frames (ie, an image frame set) may be composed of consecutive image frames.

In an embodiment, the computer may store information about a predefined detailed surgical operation. This may be derived based on the results of learning using image frame sets representing a series of surgical operations as learning data. As an example, information on detailed surgical operation may be defined as shown in Table 2 below.

The detailed surgical operation represents the minimum operation unit constituting a surgical process determined according to a specific criterion, and the detailed surgical operation may be divided by several criteria. For example, detailed surgical motions include the type of surgery (eg, laparoscopic surgery, robotic surgery, etc.), anatomical body parts where the surgery is performed, the surgical tools used, the number of surgical tools, and the direction in which the surgical tools appear on the screen. Alternatively, it may be divided based on the position and the movement of the surgical tool (eg, forward/reverse). That is, the detailed surgical operation may be determined based on the above-described attribute information.

Therefore, when the computer acquires attribute information from each image frame in the surgical image, and inputs the attribute information of each acquired image frame to the meaning derivation learning model, a series of surgical operations that are related to each other based on the attribute information (i.e., detailed surgery) operation) can be calculated as an output. For example, when the computer inputs n image frames to the motion recognition learning model, it outputs attribute information for each of the n image frames, and when it is input to the meaning derivation learning model, a plurality of (n or less) n image frames It is possible to output information about detailed surgical operation by detecting an image frame of The computer may derive the meaning of the surgical operation from a plurality of image frames based on the detailed surgical operation information.

As described above, in an embodiment of the present invention, a specific surgical operation may be identified through attribute information recognized from a surgical image, and detailed surgical operation may be recognized through the continuity of each of the specific surgical operations. In addition, it is possible to recognize the upper surgical motions that a series of detailed motions mean through learning, and it is possible to provide a learning model capable of recognizing the surgical motions of larger units step by step and even the types of surgery.

62 to 66 are diagrams for explaining a process of recognizing a surgical motion by acquiring a surgical image in the learning-based surgical motion recognition method according to an embodiment of the present invention.

62 and 63 , the computer may acquire the surgical image 200 including n image frames. The n image frames may be continuously acquired according to time.

The computer may calculate the attribute 210 of each of the image frames 200 by using the motion recognition learning model for each of the n image frames 200 . For example, the motion recognition learning model may be a model learned using CNN technology of deep learning. can be generated as an output value.

Next, the computer applies the attribute 210 for each of the n image frames 200 to the meaning derivation learning model to extract at least one image frame set 220 expressing a series of surgical operations, and corresponding Detailed surgical action can be derived. For example, the meaning derivation learning model may be a model learned using RNN technology of deep learning, and when the RNN is applied by inputting the attributes 210 of each image frame 200 as an input, the detailed surgery included in the surgical image You can create an action as an output.

64 to 66 are diagrams for explaining a process of calculating an attribute for a surgical tool from an image frame when a surgical image including a surgical tool is acquired. 64 to 66 , the computer is an image frame including at least one or more surgical tools, for example, a first surgical tool 310 (eg, a Harmonic tool) and a second surgical tool 320 (eg, a bipolar tool). (300) can be obtained. At this time, the computer detects each of the first surgical tool 310 and the second surgical tool 320 from the image frame 300 , and then applies a motion recognition learning model to each to calculate each attribute.

For example, referring to FIG. 64 , the computer first recognizes the first surgical tool 310 from the image frame 300 , and the result of the recognized first surgical tool 310 (eg, the first surgical tool 310 ) ) can be obtained. In addition, the computer may acquire the processed image frame 330 by fusing (eg, padding) the first surgical tool 310 obtained in the image frame. In this case, the processed image frame 330 may be an image including only the first surgical tool 310 by padding. The computer may calculate an attribute for the first surgical tool 310 by using the motion recognition learning model for the processed image frame 330 including only the first surgical tool 310 .

Next, referring to FIG. 65 , the computer recognizes the second surgical tool 320 from the image frame 300 , and the result of the recognized second surgical tool 320 (eg, of the second surgical tool 320 ). ID) can be obtained. In addition, the computer may acquire the processed image frame 340 by fusing (eg, padding) the second surgical tool 320 obtained in the image frame. In this case, the processed image frame 340 may be an image including only the second surgical tool 320 by padding. The computer may calculate an attribute for the second surgical tool 320 by using the motion recognition learning model for the processed image frame 340 including only the second surgical tool 320 .

That is, according to FIG. 66 , in the case of the image frame 300 including the first surgical tool 310 and the second surgical tool 320 as shown in FIGS. 64 and 65 , the first surgical tool 310 . Since the processed image frame 330 containing only the processed image frame 330 and the processed image frame 340 containing only the second surgical tool 320 are respectively acquired, the deep learning CNN for each processed image frame 330 and 340 can be applied to create each attribute as an output value. Accordingly, in the case of the image frame 300 including the first surgical tool 310 and the second surgical tool 320 , a total of two attribute calculation processes are performed. Thereafter, the computer applies each of the attributes calculated from each of the processed image frames 330 and 340 to a meaning-derivation learning model (eg, RNN technology of deep learning) to express a series of surgical motions. At least one image frame set can be extracted and the corresponding detailed surgical operation can be derived.

67 is a flowchart illustrating a surgical image learning method according to an embodiment of the present invention.

Referring to FIG. 67, the surgical image learning method according to an embodiment of the present invention includes the steps of obtaining first learning data for recognizing a surgical operation from a plurality of image frames included in the surgical image (S200), the first 1 Step of learning a motion recognition learning model based on learning data (S210), obtaining a set of continuous image frames for deriving the meaning of the surgical operation among the plurality of image frames as second learning data (S220) , it may include a step (S230) of learning the meaning derivation learning model based on the second learning data. Hereinafter, detailed description of each step is described.

The computer may acquire first learning data for recognizing a surgical operation from a plurality of image frames included in the surgical image (S200).

In an embodiment, the first learning data includes an image frame to which attribute information for motion recognition is assigned. Therefore, the computer can acquire the image frame obtained by recognizing an attribute for each of the plurality of image frames included in the surgical image as the first learning data. For example, the process of recognizing an attribute of an image frame may be performed by a person or may be performed automatically by a computer using an image recognition algorithm.

The computer may train the motion recognition learning model based on the first learning data (S210).

In one embodiment, the motion recognition learning model is learned through a learning process such as supervised learning, unsupervised learning, and reinforcement learning using the first learning data, and as a result of learning, a surgical operation corresponding to an attribute of each image frame can be recognized. can For example, as a result of learning, attribute information (type of object, position, direction, motion information, camera information, etc.) of an object as shown in Table 1 may be acquired.

The computer may acquire a set of continuous image frames for deriving the meaning of a surgical operation from the surgical image as the second learning data (S220).

In an embodiment, the second learning data includes a set of continuous image frames representing a series of surgical operations (ie, detailed surgical operations). Therefore, the computer may acquire a plurality of consecutive image frames expressing a series of surgical operations based on the recognized attributes for each image frame of the surgical image as a dataset, and use them as second learning data. For example, the process of recognizing detailed surgical motions based on the attributes recognized for the image frame may be performed by a person or may be automatically performed by a computer using deep learning learning.

The computer may train the meaning derivation learning model based on the second learning data (S230).

In one embodiment, the meaning derivation learning model is learned through a learning process such as supervised learning, unsupervised learning, and reinforcement learning using the second learning data, and as a result of the learning, the meaning of the surgical operation corresponding to the detailed surgical operation can be derived. can For example, as a result of learning, information on detailed surgical operations (name of detailed surgical operation, code information, detailed description, etc.) as shown in Table 2 can be obtained.

When learning to recognize a surgical operation, conventionally, a group of experts, such as a doctor, sees a surgical image, labels what the operation is, and uses the labeled data to teach the computer. In this case, it was difficult to acquire a lot of labeled surgical image data in terms of cost or time. In addition, the accuracy of the labeled data is not guaranteed because different doctors can derive different results in judging the surgical operation.

However, in the present invention, if each image frame is used as a unit for recognizing one surgical operation, and a technique such as image recognition is applied to each image frame, the data labeled by a computer, that is, even by a non-specialist group such as a doctor can be obtained. Therefore, it is easier to acquire data compared to the prior art in terms of cost and time. In addition, in the present invention, a more effective learning model and learning result can be derived because step-by-step learning, such as motion recognition learning and meaning derivation learning, is performed in recognizing a surgical operation.

68 is a diagram schematically showing the configuration of an apparatus 400 for performing a surgical image learning method and a learning-based surgical motion recognition method according to an embodiment of the present invention.

Referring to FIG. 68 , the processor 410 includes one or more cores (not shown) and a graphic processing unit (not shown) and/or a connection path (eg, a bus) for transmitting and receiving signals with other components. ) may be included.

The processor 410 according to an embodiment executes one or more instructions stored in the memory 320, thereby performing the surgical image learning method and the learning-based surgical operation recognition method described in connection with FIGS. 61 to 67.

For example, the processor 410 obtains first learning data for recognizing a surgical operation from a plurality of image frames included in the surgical image by executing one or more instructions stored in the memory 420, the first learning Learning a motion recognition learning model based on data, acquiring a continuous set of image frames for deriving the meaning of the surgical operation from among the plurality of image frames as second learning data, and using the second learning data It is possible to perform a surgical image learning method including the step of learning a meaning derivation learning model based on the

In addition, the processor 410 acquires a surgical image by executing one or more instructions stored in the memory 420, the step of recognizing a surgical operation by learning a plurality of image frames included in the surgical image, and the recognized It is possible to perform a learning-based surgical motion recognition method comprising the step of extracting a set of consecutive image frames from among the plurality of image frames based on the surgical operation, and deriving the meaning of the surgical operation through learning.

Meanwhile, the processor 410 includes a random access memory (RAM) and a read-only memory (ROM) that temporarily and/or permanently store a signal (or data) processed inside the processor 410 . , not shown) may be further included. In addition, the processor 410 may be implemented in the form of a system on chip (SoC) including at least one of a graphic processing unit, a RAM, and a ROM.

The memory 320 may store programs (one or more instructions) for processing and controlling the processor 410 . Programs stored in the memory 420 may be divided into a plurality of modules according to functions.

The surgical image learning method and the learning-based surgical operation recognition method according to an embodiment of the present invention described above may be implemented as a program (or application) to be executed in combination with a computer, which is hardware, and stored in a medium.

With reference to FIGS. 69 to 73, a learning-based surgical motion recognition method according to an embodiment of the present invention will be described in detail.

69 is a schematic diagram of a system capable of performing robotic surgery according to an embodiment of the present invention.

Referring to FIG. 69 , the robotic surgery system includes a medical imaging device 10 , a server 100 , and a controller 30 , a display 32 , and a surgical robot 34 provided in the operating room. According to an embodiment, the medical imaging device 10 may be omitted from the robotic surgery system according to the disclosed embodiment.

In one embodiment, the surgical robot 34 includes a photographing device 36 and a surgical tool 38 .

In one embodiment, the robotic surgery is performed by the user using the control unit 30 to control the surgical robot (34). In one embodiment, the robotic surgery may be performed automatically by the control unit 30 without the user's control.

The server 100 is a computing device including at least one processor and a communication unit.

The control unit 30 includes a computing device including at least one processor and a communication unit. In one embodiment, the controller 30 includes hardware and software interfaces for controlling the surgical robot 34 .

The photographing device 36 includes at least one image sensor. That is, the imaging device 36 includes at least one camera device, and is used to photograph an object, that is, a surgical site. In an embodiment, the imaging device 36 includes at least one camera coupled to a surgical arm of the surgical robot 34 .

In an embodiment, the image captured by the photographing device 36 is displayed on the display 340 .

In one embodiment, the surgical robot 34 includes one or more surgical tools 38 capable of performing cutting, clipping, fixing, grabbing operations of the surgical site, and the like. The surgical tool 38 is used in combination with the surgical arm of the surgical robot 34 .

The controller 30 receives information necessary for surgery from the server 100 or generates information necessary for surgery and provides it to the user. For example, the controller 30 displays generated or received information necessary for surgery on the display 32 .

For example, the user performs robotic surgery by controlling the movement of the surgical robot 34 by manipulating the controller 30 while viewing the display 32 .

The server 100 generates information necessary for robotic surgery by using the medical image data of the object photographed in advance from the medical imaging device 10 , and provides the generated information to the controller 30 .

The controller 30 provides the information received from the server 100 to the user by displaying it on the display 32 , or controls the surgical robot 34 using the information received from the server 100 .

In one embodiment, the means that can be used in the medical imaging equipment 10 is not limited, for example, CT, X-Ray, PET, MRI, etc., various other medical image acquisition means may be used.

In the disclosed embodiment, the surgical image acquired by the imaging device 36 is transmitted to the controller 30 .

In an embodiment, the controller 30 may segment a surgical image acquired during surgery in real time.

In an embodiment, the control unit 30 transmits a surgical image to the server 100 during surgery or after surgery is completed.

The server 100 may receive and analyze the surgical image.

The server 100 learns and stores at least one model for analyzing the surgical image.

The server 100 uses training data to learn at least one model, and the training data includes, but is not limited to, a surgical image and information on a surgical image.

Hereinafter, a method for the server 100 to learn a surgical image and recognize a surgical operation based on the learning will be described. However, the embodiments disclosed below are not applicable only in connection with the robotic surgery system shown in FIG. It can also be applied to embodiments.

In addition, hereinafter, for convenience of description, it will be described that a "computer" performs a learning-based surgical motion recognition method according to an embodiment disclosed herein. "Computer" may mean the server 100 or the control unit 30 of FIG. 69, but is not limited thereto and may be used to encompass a device capable of performing computing processing.

70 is a flowchart illustrating a learning-based surgical motion recognition method according to an embodiment of the present invention.

Referring to Figure 70, the learning-based surgical motion recognition method performed by a computer according to an embodiment of the present invention, the step of obtaining a surgical image sequence (S100), based on loss information for the surgical image sequence It may include a step of performing deep learning-based learning (S110), and a step of recognizing a surgical operation for a surgical image sequence based on the learning (S120). Hereinafter, detailed description of each step is described.

The computer may acquire a surgical image sequence (S100).

Here, the surgical image may be an actual surgical image or a virtual image for simulation. In one embodiment, the actual surgical image refers to data obtained as an actual medical staff performs an operation, and may be, for example, an image obtained by photographing an actual operation scene actually performed by the surgical robot 34 . That is, the actual surgical image is data recorded about the surgical site and surgical operation in the actual surgical process. The virtual image for simulation refers to a simulation image generated based on a medical image captured by the medical imaging device 10 , and may be, for example, a simulation model generated by modeling a medical image of a real patient in three dimensions. In this case, as rehearsal or simulation is performed on the simulation model in a virtual space, a virtual surgical image may be generated. Accordingly, the virtual image may be data recorded on the surgical site and surgical operation in the surgical procedure performed on the simulation model.

In addition, the surgical image may include one or more image frames. Each image frame may include a part of the patient's body, that is, a surgical site. In addition, each image frame may include not only the surgical site of the patient, but also surgical tools and consumables necessary for surgery. In other words, the surgical image refers to data composed of image frames in which the surgical operation according to time in the surgical procedure is recorded for each scene (scene). Therefore, data composed of these image frames is referred to as a surgical image sequence. 71 is a diagram illustrating a surgical image sequence. 71, the surgical image sequence may include a plurality of image frames.

The computer may perform deep learning-based learning based on loss information for the surgical image sequence (S110).

In an embodiment, the computer extracts motion information of an object based on a difference value between frames in a surgical image sequence, and learns loss information about the extracted motion information to minimize surgical operation loss. In addition, the computer recognizes the surgical motion image corresponding to the surgical motion from the surgical image sequence, and learns loss information on the recognized surgical motion image to minimize the surgical motion loss.

The computer may recognize a surgical operation for the surgical image sequence based on the learning (S120).

In an embodiment, the computer may extract a common feature based on a surgical motion image corresponding to the surgical motion, and recognize the surgical motion based on the extracted common feature.

Meanwhile, in step S110, deep learning-based learning may be performed using a Convolutional Neural Network (CNN). In one embodiment, learning is performed through CNN, including at least one main layer for recognizing a surgical operation, a first sub-layer for extracting motion information of an object, and a second sub-layer for recognizing a surgical motion image, As a result of learning, it is possible to output feature values of the image. In this case, the output layer may be a Spatial Pyramid Pooling (SPP) layer in the main layer, and the SPP layer may be connected to the first sub-layer and the second sub-layer to output a learning result. The surgical motion recognition process using CNN-based learning according to an embodiment of the present invention will be described in more detail with reference to FIG. 72 .

72 is a view for explaining a method for recognizing a surgical operation through CNN-based learning according to an embodiment of the present invention.

Referring to FIG. 72 , the computer may acquire a surgical image sequence 200 as an input value. The surgical image sequence 200 may be composed of a plurality of image frames (eg, frame 1, frame 2, ??, frame N).

The computer may perform CNN-based learning on the surgical image sequence 200 . In one embodiment, CNN-based learning is performed by the main network 210 (eg, a surgical motion recognition network), the first subnetwork 220 (eg, a motion information extraction network), and the second subnetwork 230 (eg, a network for extracting motion information). : Surgical motion image recognition network). Each of the networks 210 , 220 , and 230 may include at least one layer, and learning may be performed through each layer.

The main network 210 may function to recognize a surgical operation through learning from the input surgical image sequence 200 . In one embodiment, the main network 210 may include at least one main layer, and the at least one main layer may include a convolutional layer, a fully connected layer, and an SPP layer. have.

The main network 210 may perform learning based on the loss information on the surgical image sequence 200 in conjunction with the first subnetwork 220 and the second subnetwork 230 .

_{First, the main network 210 may calculate the operation loss information (L a} ) through learning from the input surgical image sequence 200 as in Equation 1.

는 f 프레임들을 포함하는 mXn 컬러 영상 시퀀스이고, t_i 는 i번째 타겟 값이고, C 는 수술동작 인식을 위한 클래스의 개수이고, G_1i(x; θc; θa)는 네트워크 파라미터 θc, θa에 대한 x의 i번째 소프트맥스 출력값을 나타낸다. θc, θa는 컨벌루션 레이어, 완전 연결 레이어에 대한 파라미터이다. 이때, θc는 세개의 네트워크(210, 220, 230)에서 공유되는 파라미터이다. here,

is an mXn color image sequence including f frames, t _i is the i-th target value, C is the number of classes for surgical motion recognition, and G _1i (x; θc; θa) is the network parameter θc, θa for Represents the i-th softmax output value of x. θc and θa are parameters for the convolutional layer and the fully connected layer. In this case, θc is a parameter shared by the three networks 210 , 220 , and 230 .

In general, image data acquired during minimally invasive surgery, such as laparoscopic surgery or robotic surgery, has a problem in that it is difficult to learn because the amount of data is small and the movement of objects in the image is fine. When deep learning-based learning is performed using such image data, it operates effectively only on the learned image data, so it is difficult to generalize. Therefore, in the present invention, in order to solve this problem, by learning the loss information, it is possible to effectively learn even with little image data and fine movements to improve the recognition of the surgical operation.

In one embodiment, the main network 210 _{acquires the loss information (L d} ) on the motion information of the object from the first sub-network 220 together with the _{surgical operation loss information (L a} ) and performs learning, Through this, it is possible to minimize the loss of recognition of the surgical motion. Furthermore, the main network 210 performs learning and obtaining loss information (L _S) for the surgical operation image for the surgical operation from the second sub-network 230, and to minimize the loss of the surgical operation recognition through them can do.

The first subnetwork 220 extracts the motion information of the object from the input surgical image sequence 200, and performs learning based on this to calculate the _{loss information (L d ) for the motion information of the object.} can Also, the first subnetwork 220 may serve as a decoder and may generate motion information of an object. As an embodiment, the first sub-network 220 may include at least one first sub-layer, and the at least one first sub-layer may include a fully connected layer.

Here, the object is included in the image data captured by the camera entering the body of the patient who is the subject of minimally invasive surgery, for example, a surgical tool, a part of the patient's body (eg, organs, blood vessels, etc.), the patient's body It refers to objects (eg, bleeding, etc.) created from

First, the first sub-network 220 may extract motion information of an object based on a difference value between adjacent frames in the input surgical image sequence 200, and may be calculated as in Equation (2).

Here, x(i) is the i-th frame, and ε(x) is the sum of absolute values (SAD) of the difference values between adjacent frames in the surgical image sequence 200. If the movement is large, If it has a large value and the motion is small, it can have a small value. That is, the motion information may be calculated based on the amount of change in pixel values between two frames. Since there is little change in the background image in the surgical image, it is possible to effectively extract minute movements of an object such as a surgical tool or an organ through Equation (2).

Next, the first subnetwork 220 may _{calculate the loss information (L d} ) from the motion information of the object, and may be calculated as in Equation (3). In this case, a feature value (ie, feature map information) extracted from the SPP layer of the main network 210 may be obtained and used as an input value.

Here, G ₂ (x; θc; θd) is a decoder having weight parameters θc and θd that generates an image of the same size from one frame of an input image sequence, and ε is the SAD of the images. For example, when L _d is 0, since the combination of G2(x; θc; θd) must represent SAD, motion information can be extracted. Since θc is a parameter shared with the main network, object motion information can be utilized when recognizing a surgical operation.

A second sub-network 230 is aware of the surgical operation image for the surgical operation from the input operation image sequence 200, and it performs the learning is based on calculating the loss information (L _S) for the surgical operation image function can be As an embodiment, the second sub-network 230 may include at least one second sub-layer, and the at least one second sub-layer may include a fully connected layer.

First, the second subnetwork 230 may recognize a surgical motion image corresponding to a surgical motion from the input surgical image sequence 200 , and may be calculated as in Equations 4 and 5 .

Here, S ∈ {0, 1}-- ^M , M is the number of operations, and S _i is the i-th element of S. For example, if there are three independent surgical images, M = 3 may be. In this case, the second subnetwork 230 may use a gradient reversal layer (GRL), which may change the sign of the gradient calculated during backpropagation. Therefore, θc can be updated using a negative gradient as in Equation 5.

Here, η is the learning rate, and P is a parameter controlling the learning intensity. This parameter can be set to 0.1.

Next, the second sub-network 230 is possible to calculate the loss of information (L _S) from the surgical operation image for the surgical operation, and can be calculated as Equation (6). In this case, a feature value (ie, feature map information) extracted from the SPP layer of the main network 210 may be obtained and used as an input value.

Here, θs is a learnable _{parameter of the second subnetwork 230 G 3} , and G _3i is an i-th softmax value of the network. At this time, S _i is required, which may be the first element automatically generated using the surgical image index number v.

That is, the second sub-network 230 is learning to minimize the loss of information (L _S) for the surgical operation image, the loss information (L _S) is to identify the surgical operation image in the operation image sequence 200.

As described above, in general, when CNN-based learning tends to remember only input data as a method of minimizing loss information, the ratio of objects (eg, surgical tools, organs, consumables, etc.) If the background image excluding the object occupies a large proportion and hardly moves, there is a problem in that it does not operate properly. In this case, since it is important to perform learning by generating a loss by giving a bad influence, in the present invention, unnecessary information can be removed and only useful information can be extracted using the _{loss information (L a} , L _d , L _{s ) described above.} made to be

Finally, the main network 210 may calculate the final loss information (L _{t ) based on the loss information (L a} , L _d , L _s ) derived through Equations 1 to 6, and Equations 7 and can be calculated together.

Here, λ _d and λ _s represent parameters for adjusting the ratio of loss values.

In one embodiment, the main network 210 can extract common feature information from the surgical image sequence 200 by calculating the _{final loss information (L t ) through learning, and based on the extracted common feature information} The corresponding surgical operation can be recognized. The main network 210 may output feature information from the SPP layer, and in this case, it may process the feature information by additionally using camera-related information. For example, scale information related to zoom-in/zoom-out of the camera may be processed, and a case in which the camera zooms in/outs due to movement of a surgical tool during robotic surgery may be included.

Here, the surgical operation may mean a minimum operation unit constituting the surgical process. As described above, the surgical image includes continuous image frames in which the surgical process is recorded over time. Therefore, by performing learning on the surgical image sequence 200 including these continuous image frames, it is possible to recognize what kind of surgical operation is being performed in the entire surgical process as a minimum operation unit, so that the meaning of the corresponding surgical operation can be more effectively understood. be able to comprehend

According to an embodiment, the computer may store information about a predefined surgical operation. This is information representing one surgical operation, and may be determined based on a standardized name. For example, it may be generated as code data of a specific number of digits. Table 3 below shows an example of information expressing the surgical operation.

In one embodiment, a surgical operation represents a minimum operation unit constituting a surgical process determined according to a specific criterion, and the surgical operation may be divided by several criteria. For example, the surgical operation may be determined by the type of operation (eg, laparoscopic surgery, robotic surgery, etc.), the anatomical body part where the surgery is performed, the surgical tools used, the number of surgical tools, the direction in which the surgical tools appear on the screen, or It can be divided based on the position, the movement of the surgical tool (eg, forward/reverse), and the like.

73 is a diagram schematically showing the configuration of an apparatus 300 for performing a learning-based surgical motion recognition method according to an embodiment of the present invention.

Referring to FIG. 73 , the processor 310 includes one or more cores (not shown) and a graphic processing unit (not shown) and/or a connection path (eg, a bus) for transmitting and receiving signals with other components. ) may be included.

The processor 310 according to an embodiment executes one or more instructions stored in the memory 320, and a method of generating a virtual body model described in relation to FIGS. 70 to 72 and a method of providing location information of a surgical tool carry out

For example, the processor 310 obtains a surgical image sequence by executing one or more instructions stored in the memory 320, and performs deep learning-based learning based on loss information for the surgical image sequence, Based on the learning, it is possible to recognize the surgical operation for the surgical image sequence.

Meanwhile, the processor 310 temporarily and/or permanently stores a signal (or data) processed inside the processor 310 . , not shown) may be further included. In addition, the processor 310 may be implemented in the form of a system on chip (SoC) including at least one of a graphic processing unit, a RAM, and a ROM.

The memory 320 may store programs (one or more instructions) for processing and controlling the processor 310 . Programs stored in the memory 320 may be divided into a plurality of modules according to functions.

The learning-based surgical operation recognition method according to an embodiment of the present invention described above may be implemented as a program (or application) and stored in a medium to be executed in combination with a computer, which is hardware.

Hereinafter, embodiments related to bleeding evaluation in which a bleeding area and an amount of bleeding are measured based on whether or not bleeding has occurred during surgery are recognized using a surgical image will be described below.

In one embodiment, there is provided a method for evaluating bleeding using a surgical image performed by a computer, the method comprising: acquiring a surgical image; Recognizing whether a bleeding region exists in the surgical image based on deep learning-based learning; and estimating the location of the bleeding region from the surgical image based on the recognition result. A detailed description thereof will be described later with reference to FIGS. 74 to 77 .

A bleeding evaluation method using a surgical image according to an embodiment of the present invention will be described in detail with reference to FIGS. 74 to 77 .

74 is a flowchart illustrating a bleeding evaluation method using a surgical image according to an embodiment of the present invention.

Although the method of FIG. 74 is described as being performed by a computer for convenience of description, the subject performing each step is not limited to a specific device and may be used to encompass devices capable of performing computing processing. That is, in this embodiment, the computer may mean a device capable of performing the bleeding evaluation method using the surgical image according to the embodiment of the present invention.

Referring to FIG. 74, the bleeding evaluation method using a surgical image according to an embodiment of the present invention includes: acquiring a surgical image (S100), based on deep learning-based learning, whether a bleeding region exists in the surgical image Recognizing (S200), estimating the location of the bleeding area from the surgical image based on the recognition result (S300), and calculating the amount of bleeding in the bleeding area based on the location of the bleeding area (S400) can do. Hereinafter, detailed description of each step is described.

The computer may acquire a surgical image (S100).

Here, the surgical image may be an actual surgical image or a virtual image for simulation.

In one embodiment, the actual surgical image refers to data obtained as a medical staff performs an actual operation, for example, an actual image taken by a camera inserted into a patient's body during minimally invasive surgery such as a surgical robot, a laparoscope, an endoscope, etc. It may be an image including a surgical scene. That is, the actual surgical image is data recorded about the surgical site and surgical operation in the actual surgical process.

In one embodiment, the virtual image for simulation refers to a simulation image generated based on a medical image taken from a medical imaging device such as CT, MRI, PET, etc., for example, generated by modeling a medical image of a real patient in three dimensions. It may be a simulated model. In this case, as rehearsal or simulation is performed on the simulation model in a virtual space, a virtual surgical image may be generated. Accordingly, the virtual image may be data recorded on the surgical site and surgical operation in the surgical procedure performed on the simulation model.

The computer may recognize whether there is a bleeding region in the surgical image acquired in step S100 based on deep learning-based learning (S200).

In one embodiment, the computer uses a pre-trained learning model to recognize whether the surgical image obtained in step S100 is a bleeding image including a bleeding area (that is, to recognize whether a bleeding area exists in the surgical image). Available.

In this case, in using the pre-learned learning model, the computer may first perform learning to recognize the presence or absence of a bleeding region in the surgical image using deep learning based on the surgical image dataset. The surgical image dataset may be a learning dataset in which labeling is performed on surgical images through various learning methods. For example, the surgical image dataset may be data learned using machine learning methods such as supervised learning, unsupervised learning, and reinforcement learning. Therefore, the computer acquires a surgical image dataset as learning data, and uses it to perform learning to recognize the presence or absence of a bleeding area (ie, bleeding region) in a surgical image to learn a learning model (eg, bleeding presence recognition model). can be built in advance. In this case, the computer may build and store a learning model learned based on the surgical image dataset in advance, or use the learning model built in another device.

Then, when the computer acquires a new surgical image (that is, the surgical image in step S100), the bleeding area in the new surgical image is created using a learning model (eg, bleeding recognition model) learned based on the surgical image dataset. It can be recognized whether it exists or not. That is, the computer can quickly and effectively determine whether the newly acquired surgical image is a bleeding image containing a bleeding area or a non-bleeding image through a deep learning-based pre-established learning model (eg, bleeding presence recognition model). have.

The computer may estimate the location of the bleeding area from the surgical image based on the surgical image recognition result of step S200 (that is, the recognition result of whether the surgical image is a bleeding image or a non-bleeding image) (S300).

In one embodiment, when the computer recognizes that the bleeding area in the surgical image is a bleeding image based on deep learning-based learning, it can specify the bleeding area in the surgical image and estimate the location of the specified bleeding area. have. For example, the computer may extract feature information from the surgical image through deep learning-based learning, and recognize and specify the bleeding area in the surgical image based on the extracted feature information. In this case, the characteristic information is information indicating the characteristics of bleeding, and texture information such as color, shape, and texture for specifying bleeding may be used. A detailed process for this will be described later with reference to FIG. 75 .

The computer may calculate the amount of bleeding in the bleeding area based on the location of the bleeding area estimated in step S300 (S400).

In an embodiment, the computer may calculate the amount of bleeding by using pixel information of the bleeding area in the surgical image. Alternatively, when the surgical image is a stereoscopic image, the computer acquires depth information of the bleeding region in the surgical image based on a depth map of the surgical image, and estimates the volume corresponding to the bleeding region based on this to estimate the amount of bleeding. can be calculated. Alternatively, when gauze is included in the surgical image, the computer may calculate the amount of bleeding in the bleeding area using the gauze information.

75 is a view showing an example to which the bleeding evaluation method using a surgical image according to an embodiment of the present invention can be applied. That is, FIG. 75 is a specific embodiment that can be applied to the method of FIG. 74 , and a detailed description of the overlapping process with that of FIG. 74 will be omitted.

Referring to FIG. 75 , the computer may acquire a surgical image (S100).

In an embodiment, the computer may acquire at least one surgical image taken during a specific surgery in order to evaluate the bleeding occurring during a specific surgery (eg, whether or not bleeding has occurred, the degree of bleeding, etc.). The computer may repeatedly perform each step to be described later for each of at least one surgical image taken during a specific surgery. Therefore, since the computer can determine the bleeding area and the amount of bleeding from each surgical image taken during a specific operation, it is possible to finally evaluate the total bleeding level during the specific operation.

Based on deep learning-based learning, the computer may recognize whether there is a bleeding region in the surgical image (S200).

More specifically, the computer can perform deep learning-based learning using a convolutional neural network (CNN) by inputting a surgical image to a pre-trained learning model (eg, a bleeding recognition model) based on a surgical image dataset. (S210). In this case, the CNN may recognize whether bleeding has occurred in the surgical image using at least one layer.

The computer can extract feature information from the surgical image through learning using CNN (S220).

In an embodiment, the computer may obtain a feature map by extracting feature information from the surgical image through at least one layer of the CNN. For example, the computer may generate a feature map by extracting information indicating the characteristics of bleeding while undergoing a learning process through at least one layer of CNN for a surgical image. In this case, the bleeding characteristic may be extracted using texture information such as color, texture, and material in the surgical image.

The computer may recognize whether there is a bleeding region in the surgical image based on the characteristic information (S230).

In an embodiment, the computer recognizes whether a bleeding area exists based on a feature map including feature information through a classifier, and whether a bleeding area exists in a bleeding image or does not exist according to the recognition result. Non-bleeding images can be classified.

The computer may estimate the location of the bleeding area from the surgical image based on the recognition result through learning of the surgical image (S300).

More specifically, the computer may specify the bleeding region in the surgical image based on the feature map including the feature information. That is, since the computer can grasp the part representing the bleeding characteristic from the feature map, it can specify the bleeding area in the surgical image.

In one embodiment, the computer may specify the bleeding area from the surgical image based on the degree of influence on the characteristics of the bleeding area from a result (eg, feature information) derived using at least one layer of the CNN. For example, the computer may convert each pixel in the surgical image to a specific value based on a feature map including feature information, and specify a bleeding region based on the specific value of each pixel in the surgical image. Each pixel in the surgical image may be converted into a specific value based on a predetermined weight depending on whether the region corresponds to the bleeding characteristic based on the feature map (ie, according to the degree of influence on the bleeding characteristic). For example, the computer can convert each pixel of the surgical image into a specific value by applying the Grad CAM (Gradient-weighted Class Activation Mapping) technology that inversely estimates the learning result of recognizing the bleeding area in the surgical image through CNN. can Based on the feature map, the computer assigns a high value (eg, high weight) to a pixel recognized as corresponding to a bleeding region in the surgical image, and a low value (eg, to a pixel recognized not to correspond to a bleeding region). Each pixel value of the surgical image can be transformed by giving a low weight). The computer can emphasize the bleeding area in the surgical image more through the converted pixel value, and through this, the bleeding area can be segmented and the location of the corresponding area can be estimated. In addition, the computer can convert each pixel into a probability value by applying the Grad CAM technology to generate a heat map for each pixel in the surgical image based on the feature map. The computer may specify the bleeding area in the surgical image based on the converted probability value of each pixel. For example, the computer may determine a pixel area having a large probability value as a bleeding area.

The computer may calculate the amount of bleeding based on the bleeding area in the surgical image (S400).

In an embodiment, the computer may calculate the amount of bleeding by using pixel information of the bleeding area in the surgical image. For example, the computer may calculate the amount of bleeding by using the number of pixels corresponding to the bleeding region in the surgical image, color information of the pixels (eg, RGB values), and the like.

In another embodiment, when the surgical image is a stereoscopic image, the computer acquires depth information of the bleeding region in the surgical image based on a depth map of the surgical image, and on the basis of the acquired depth information, The amount of bleeding can be calculated by estimating the corresponding volume. When the surgical image is a stereoscopic image, since it has a three-dimensional effect, that is, depth information, the volume of the bleeding region in the three-dimensional space can be grasped. For example, the computer may obtain pixel information (eg, the number of pixels, positions of pixels, etc.) of the bleeding area in the surgical image, and calculate a depth value of the depth map corresponding to the acquired pixel information of the bleeding area. . The computer may calculate the amount of bleeding by determining the volume of the bleeding area based on the calculated depth value.

In another embodiment, when gauze is included in the surgical image, the computer may calculate the amount of bleeding in the bleeding area using the gauze information. For example, the computer may use the number of gauze in the surgical image, color information (eg, RGB value) of the gauze, etc. to calculate the amount of bleeding occurring in the bleeding area, and may reflect it.

In another embodiment, the computer may calculate the amount of bleeding in the bleeding area based on the 3D surgical simulation model generated to match the patient's body. For example, the computer may obtain an area corresponding to the bleeding area from the 3D surgery simulation model, and calculate the amount of bleeding by estimating pixel information or volume information (depth information) on the area in the obtained 3D surgery simulation model. .

In another embodiment, the computer may segment the bleeding area in the surgical image and calculate the amount of bleeding using pixel information and a depth map of the segmented bleeding area. In this case, since the bleeding amount is calculated preferentially for the portion estimated as the bleeding area (segmented portion), more effective and accurate bleeding amount calculation is possible. Even in the case of having , it is possible to reduce the calculated error by being reflected in the calculation of the bleeding amount. A detailed process for this will be described with reference to FIG. 76 .

76 is a diagram illustrating an example of calculating a bleeding amount by segmenting a bleeding region in a surgical image according to an embodiment of the present invention. Referring to FIG. 76 , the computer acquires the surgical image 10 and may estimate the bleeding region by recognizing whether a bleeding region exists in the surgical image 10 based on deep learning-based learning. In this case, the computer may segment 20 only the portion corresponding to the bleeding area from the surgical image by applying the Grad CAM technology as described above. The computer may obtain a depth map 30 corresponding to the segmented bleeding region 20 . Then, the computer generates pixel information (eg, the number of pixels, RGB color values of pixels, etc.) about the bleeding area 20 based on the surgical image 10, the segmented bleeding area 20, and the depth map 30. The amount of bleeding can be calculated using volume information (depth information). In this case, since the bleeding amount can be calculated by focusing on the segmented bleeding area, a more accurate bleeding amount can be obtained.

In calculating the amount of bleeding in the bleeding area, the computer may also calculate the amount of bleeding by comparing it with the non-bleeding surgical image in which the bleeding area is present but not in the non-bleeding surgery image. In one embodiment, the computer acquires pixel information or volume information (depth information) of a portion corresponding to the bleeding area of the bleeding surgical image in the non-bleeding surgical image, and uses the pixel information or volume information of the bleeding area of the bleeding surgical image. (depth information) can be compared. Accordingly, since the computer can grasp accurate pixel information or volume information (depth information) at the time of non-bleeding through comparison between the two images, it is possible to more accurately calculate the amount of bleeding in the bleeding area.

In addition, according to the type of surgery, the surgical site, etc., the computer calculates the amount of bleeding using pixel information on the bleeding area in the surgical image, whether to calculate the amount of bleeding using volume information (depth information), or using gauze information. It is also possible to determine whether to calculate the amount of bleeding or whether to calculate the amount of bleeding using a three-dimensional surgical simulation model. For example, since a lot of bleeding occurs in gastric cancer surgery, the operation is performed by continuously absorbing the bleeding site using gauze. In this case, since it is difficult to measure the volume in the bleeding region, the computer may calculate the amount of bleeding by reflecting the number of gauze used and the color change as the gauze absorbs blood. In the case of rectal cancer surgery, since a lot of blood vessels are distributed around it, bleeding occurs little by little. In this case, since the medical staff performs irrigation (ie, spraying water) several times to identify the bleeding site, it is impossible to accurately calculate the amount of bleeding by measuring the volume in the bleeding area. Accordingly, the computer may calculate the amount of bleeding by reflecting the color change of pixels in the bleeding area.

According to an embodiment of the present invention, it is possible to provide a learning model specialized for bleeding by estimating the bleeding area in the surgical image and calculating the amount of bleeding.

In addition, according to an embodiment of the present invention, since the bleeding area and the amount of bleeding can be identified from the surgical image, the total degree of bleeding that occurred during a specific operation can be estimated based on this, and furthermore, the criteria for evaluating the operation through the degree of bleeding can be determined. can provide

In addition, according to an embodiment of the present invention, it is possible to automatically derive the presence or absence of bleeding and the degree of bleeding from the surgical image without the intervention of a medical staff.

77 is a diagram schematically showing the configuration of an apparatus 100 for performing a bleeding evaluation method using a surgical image according to an embodiment of the present invention.

Referring to FIG. 77, the processor 110 includes one or more cores (not shown) and a graphic processing unit (not shown) and/or a connection path (eg, a bus) for transmitting and receiving signals with other components. ) may be included.

The processor 110 according to an embodiment executes one or more instructions stored in the memory 120 to perform the bleeding evaluation method using the surgical image described in relation to FIGS. 74 to 76 .

For example, the processor 110 acquires a surgical image by executing one or more instructions stored in the memory 120, recognizes whether a bleeding region exists in the surgical image based on deep learning-based learning, and recognizes the result Based on this, the location of the bleeding area can be estimated from the surgical image.

On the other hand, the processor 110 is a RAM (Random Access Memory, not shown) and ROM (Read-Only Memory: ROM) for temporarily and/or permanently storing signals (or, data) processed inside the processor 110 . , not shown) may be further included. In addition, the processor 110 may be implemented in the form of a system on chip (SoC) including at least one of a graphic processing unit, a RAM, and a ROM.

The memory 120 may store programs (one or more instructions) for processing and controlling the processor 110 . Programs stored in the memory 120 may be divided into a plurality of modules according to functions.

The bleeding evaluation method using a surgical image according to an embodiment of the present invention described above may be implemented as a program (or application) and stored in a medium to be executed in combination with a computer, which is hardware.

Hereinafter, embodiments of a technique for providing information about organs by estimating organs observable in a recognized surgical stage by recognizing a surgical stage in a specific operation based on a surgical image are disclosed.

In one embodiment, there is provided a method for providing surgical information using a surgical image performed by a computer, the method comprising: acquiring a surgical image for a specific surgery; recognizing a surgical step in the specific surgery corresponding to the surgical image; estimating a candidate organ group including at least one organ that can be extracted in the operation step; and specifying an organ region in the surgical image based on a positional relationship between organs in the organ candidate group. A detailed description thereof will be described later with reference to FIGS. 78 to 83 .

A method of providing surgical information using a surgical image according to an embodiment of the present invention will be described in detail with reference to FIGS. 78 to 83 .

78 is a schematic diagram of a system capable of performing robotic surgery according to an embodiment of the present invention.

Referring to FIG. 78 , the robotic surgery system includes a medical imaging device 10 , a server 100 , and a controller 30 , a display 32 , and a surgical robot 34 provided in the operating room. According to an embodiment, the medical imaging device 10 may be omitted from the robotic surgery system according to the disclosed embodiment.

In one embodiment, the surgical robot 34 includes a photographing device 36 and a surgical tool 38 .

In one embodiment, the robotic surgery is performed by the user using the control unit 30 to control the surgical robot (34). In one embodiment, the robotic surgery may be performed automatically by the control unit 30 without the user's control.

The server 100 is a computing device including at least one processor and a communication unit.

The control unit 30 includes a computing device including at least one processor and a communication unit. In one embodiment, the controller 30 includes hardware and software interfaces for controlling the surgical robot 34 .

The photographing device 36 includes at least one image sensor. That is, the imaging device 36 includes at least one camera device, and is used to photograph an object, that is, a surgical site. In an embodiment, the imaging device 36 includes at least one camera coupled to a surgical arm of the surgical robot 34 .

In an embodiment, the image captured by the photographing device 36 is displayed on the display 340 .

In one embodiment, the surgical robot 34 includes one or more surgical tools 38 capable of performing cutting, clipping, fixing, grabbing operations of the surgical site, and the like. The surgical tool 38 is used in combination with the surgical arm of the surgical robot 34 .

The controller 30 receives information necessary for surgery from the server 100 or generates information necessary for surgery and provides it to the user. For example, the controller 30 displays generated or received information necessary for surgery on the display 32 .

For example, the user performs robotic surgery by controlling the movement of the surgical robot 34 by manipulating the controller 30 while viewing the display 32 .

The server 100 generates information necessary for robotic surgery by using the medical image data of the object photographed in advance from the medical imaging device 10 , and provides the generated information to the controller 30 .

The controller 30 provides the information received from the server 100 to the user by displaying it on the display 32 , or controls the surgical robot 34 using the information received from the server 100 .

In one embodiment, the means that can be used in the medical imaging equipment 10 is not limited, for example, CT, X-Ray, PET, MRI, etc., various other medical image acquisition means may be used.

As described above, when performing robotic surgery, it is possible to acquire data including surgical images captured in the surgical process or various surgical information in the control process of the surgical robot. As described above, in the present invention, it is intended to provide more meaningful information to medical staff who are performing surgery by using surgical images or surgical data that can be obtained in the course of surgery.

Hereinafter, for convenience of description, it will be described that a "computer" performs a method of providing surgical information using a surgical image according to an embodiment disclosed herein. “Computer” may mean the server 100 or the controller 30 of FIG. 78 , but is not limited thereto and may be used to encompass a device capable of performing computing processing. For example, the computer may be a computing device provided separately from the device shown in FIG. 78 .

In addition, the embodiments disclosed below may not be applied only in connection with the robotic surgery system shown in FIG. 78, but may be applied to all kinds of embodiments in which a surgical image may be acquired and utilized in the surgical process.

79 and 80 are flowcharts illustrating a method of providing surgical information using a surgical image according to an embodiment of the present invention.

79 and 80 , the method for providing surgical information using a surgical image according to an embodiment of the present invention includes: acquiring a surgical image for a specific surgery (S100), the specific surgery corresponding to the surgical image Recognizing the operation stage in the operation step (S200), estimating a candidate organ group including at least one organ that can be extracted in the operation step (S300), and based on the positional relationship between the organs in the organ candidate group and specifying an organ region in the surgical image (S400). Hereinafter, detailed description of each step is described.

The computer may acquire a surgical image for a specific surgery (S100).

When a specific operation is performed on a subject for surgery, medical staff may directly perform an actual operation on the subject of surgery, and may perform minimally invasive surgery using a laparoscopy or an endoscope as well as a surgical robot as described in FIG. .

In this case, the computer may acquire a surgical image obtained by photographing a scene including a surgical operation performed in the surgical procedure and related surgical tools, surgical sites, and the like. In one embodiment, the computer may acquire a surgical image that captures a scene including a surgical site and a surgical tool currently being operated from a camera that has entered the body of a subject to be operated on.

The surgical image may include one or more image frames. Each image frame may represent one scene in which a surgical operation is performed, including a surgical site of a subject to be operated, a surgical tool, and the like. For example, the surgical image may be composed of image frames in which a surgical operation according to time is recorded for each scene in a surgical procedure. Alternatively, the surgical image may be composed of image frames in which each surgical scene is recorded according to spatial movement, such as a position of a surgical site or a camera, during a surgical procedure.

The computer may recognize a surgical workflow stage in a specific surgical procedure corresponding to the surgical image obtained in step S100 (S200).

In one embodiment, the computer may perform learning using deep learning for a surgical image (eg, at least one image frame) acquired in a specific surgical procedure, and context information about the surgical image through learning can be derived. In addition, the computer may recognize a surgical step corresponding to the corresponding surgical image in a specific surgical procedure consisting of at least one surgical step based on context information on the surgical image. A detailed process for this will be described with reference to FIG. 81 .

81 is an example showing a process of recognizing a surgical stage based on a surgical image according to an embodiment of the present invention.

Referring to FIG. 81 , the computer may learn a surgical image (ie, at least one image frame) acquired during a specific operation.

In one embodiment, first, the computer may perform learning using a Convolutional Neural Network (CNN) for each image frame included in the surgical image (S210). For example, the computer may learn features of the surgical image by inputting each image frame to at least one layer (eg, a convolution layer). As a result of this learning, the computer can infer what or what each image frame represents.

Next, the computer may perform learning using a recurrent neural network (RNN) on the surgical image (ie, at least one image frame) derived as a result of learning using the CNN ( S220 ). In one embodiment, the computer may receive the surgical image in frame units and learn what the surgical image in the corresponding frame means by using RNN (eg, LSTM method). For example, the computer receives a surgical image (eg, a first image frame) of a current time point and at least one surgical image (eg, a second to n-th image frame) of a previous time point using RNN (eg, LSTM). Learning is performed, and as a result of the learning, context information on the surgical image of the current viewpoint (ie, the first image frame of the current viewpoint) may be derived. Here, the context information is information indicating what the surgical image means, and may include information related to a specific surgical operation in a surgical procedure.

The computer can recognize the meaning of the surgical image (ie, context information) based on the learning result of the surgical image, and recognize the surgical step corresponding to the corresponding surgical image. In one embodiment, the computer may recognize the operation step for each image frame based on the context information for each image frame derived as a result of learning of the RNN (S230).

In this case, a specific surgical procedure may consist of at least one surgical step. In one embodiment, at least one surgical step may be predefined for each specific operation. For example, each surgical stage may be classified according to the lapse of time during which a specific operation is performed, or each surgical stage may be classified to correspond to each surgical site based on the surgical site. Alternatively, each surgical step may be classified based on the position of the camera or the movement range of the camera during a specific surgery, and each surgery may be based on a change (eg, replacement, etc.) of a surgical tool during a specific surgery. It may be a classification of steps.

In addition, the surgical steps classified as described above may be configured in a hierarchical structure. In an embodiment, a specific surgical procedure may be classified in stages from the lowest layer (level) to the highest layer (level) to form a hierarchical structure. Here, the lowest layer is composed of the smallest units representing the surgical procedure, and may include a minimum surgical operation having a meaning as one minimum operation. For example, the computer may recognize a surgical operation representing one constant operation pattern, such as cutting, grabbing, moving, etc., as a minimal surgical operation, and may configure it as the lowest level (ie, the lowest surgical stage). In addition, the computer may group at least one minimum surgical operation according to a specific classification criterion (eg, time lapse, surgical site, movement of a camera, change of surgical tool, etc.) and configure it as an upper layer. For example, when each minimal surgical operation representing grabbing, moving, cutting, etc. of a surgical tool is performed in a predetermined order and has a meaning as a specific operation by connecting the predetermined order, it can be grouped into one upper layer. have. As another example, when each minimal surgical operation such as clipping, moving, and cutting is performed continuously, it may be recognized that it is a surgical operation of cutting a blood vessel and grouped into one upper layer. As another example, when each minimal surgical operation such as grabbing, lifting, cutting, and removing is performed continuously, it may be recognized that this is a surgical operation to remove fat and grouped into one upper layer. In this way, when it is recognized as an upper surgical operation (ie, an upper surgical step) having a specific meaning from the lower-level surgical operations (ie, lower-level surgical steps), it can be classified into a higher-level operation and this process is repeated. can be done with Therefore, finally, a hierarchical structure can be formed up to the highest layer (ie, the highest surgical stage) of a specific surgical procedure. For example, a specific surgical procedure may have a hierarchical structure such as a tree shape.

That is, since the computer can estimate the meaning of the surgical image by deriving context information for each image frame, it can recognize the specific surgical stage to which the image frame belongs from among the surgical stages predefined for a specific surgery. . In this case, the recognized specific surgical stage is a specific hierarchical level (eg, second level to (n-1) level) from the lowest layer (eg, first level) to the highest layer (n-th level) hierarchically configured for a specific operation. It may mean any one surgical operation belonging to any one level). That is, the computer can recognize a surgical step belonging to a specific layer on the hierarchical structure of the surgical procedure through context information for each image frame.

Referring back to FIGS. 79 and 80 , the computer may estimate a candidate organ group including at least one organ that can be extracted from the operation step recognized in step S200 ( S300 ).

As described above, since a specific surgical procedure may consist of predetermined surgical steps, various information (eg, specific surgical operation, specific surgical site, specific surgical tool, camera position, etc.) can be obtained from each predetermined surgical step. can be extracted. In one embodiment, the computer may extract information on the body organs observable in each surgical stage to generate an organ candidate group for each surgical stage. Alternatively, the computer may acquire a group of long-term candidates for each surgical stage that has been generated in advance.

For example, as shown in FIG. 3 , the computer may recognize the first to nth surgical steps for each image frame. In this case, the computer may extract organs that can be observed at the time of surgery from each of the first to nth surgical stages, and configure a candidate organ group including the extracted organs for each surgical stage. For example, the computer recognizes the k th surgical stage (eg, left hepatic artery (LHA) incision stage) from a specific surgical image (k th image frame), and converts the stomach, liver, and spleen into observable organs in the recognized k th surgical stage. By estimating it, it can be configured as a long-term candidate group.

The computer may specify an organ region in the surgical image (ie, at least one image frame) based on the information on at least one organ in the organ candidate group estimated in step S300 ( S400 ).

In an embodiment, the computer may specify an organ region in the corresponding surgical image based on at least one of the positional relationship between the organs in the organ candidate group and the texture information of the organs estimated from the surgical stage for the corresponding surgical image. can A detailed process for this will be described with reference to FIG. 82 .

82 is an example illustrating a process of specifying an organ region in a surgical image based on information on an organ in an organ candidate group according to an embodiment of the present invention.

Referring to FIG. 82 , the computer may calculate the positional information of at least one organ included in the surgical image by learning the positional relationship between the organs in the organ candidate group ( S410 ).

As an embodiment, the computer may configure a neural network (eg, a backbone network) including at least one layer (eg, a convolution layer) in order to learn a positional relationship between organs in the organ candidate group. For example, the computer may input a surgical image (ie, a specific image frame) and a group of organ candidates to the backbone network, and may recognize the positional relationship between organs in the organ candidate group through learning in the first convolution layer. The computer may calculate the position information of at least one organ included in the surgical image based on the learning result on the positional relationship between the organs in the organ candidate group. Here, as the organ location information, coordinate values of a specific point (eg, a central point) among regions in which organs are distributed in the surgical image may be used.

Each organ has a fixed position within the body. In addition, since the types of organs included in the long-term candidate group can be known, location information of the organs in the long-term candidate group can also be acquired. Accordingly, the computer may perform learning based on the fixed positions of organs in the organ candidate population. That is, organs having a fixed position may be photographed as if they exist in different positions depending on the field of view (FOV) of the camera, the angle of the camera, the position of the camera, etc., but the positional relationship between the organs is maintained do. In other words, since the spatial topological relationship between organs is maintained, the computer can learn the spatial topological relationship between organs in the organ candidate group. The computer can recognize the arrangement of organs in the surgical image by learning the spatial phase relationship between these organs. In addition, the computer may calculate the position information of the organs in the surgical image based on the arrangement of the organs.

The computer may learn texture information on organs in the organ candidate group to calculate texture information of at least one organ included in the surgical image (S420).

As an embodiment, the computer may include at least one layer (eg, a convolution layer) in a neural network (eg, a backbone network) to learn texture information on organs in the organ candidate group. For example, the computer may input a surgical image (ie, a specific image frame) and a candidate organ group to the backbone network, and derive texture information of organs in the organ candidate group through learning in the second convolution layer. The computer may calculate texture information on at least one organ included in the surgical image based on the learning result of the texture information of the organs in the organ candidate group.

The computer may specify an organ region in which an organ exists in the surgical image based on the position information of the organs in the surgical image calculated in step S410, but may detect a region corresponding to the texture information of the organ calculated in step S420 ( S430).

In one embodiment, the computer recognizes an organ at the corresponding position based on the position information of the organs present in the surgical image, and uses the texture information calculated for the recognized organ within the surgical image having the same or similar texture information. The distribution area can be detected. For example, if the computer calculates the position information for three organs (eg, A, B, C organs) in the surgical image through learning, the computer recognizes the A organ in the position for the A organ in the surgical image and , it is possible to detect an organ region A matching the texture information of organ A in the surgical image based on the texture information of organ A calculated through learning. In addition, the computer can detect the organ B and organ C region by repeating the same process for the organ B and organ C. Therefore, the computer can finally specify each organ region in which each organ included in the surgical image exists.

According to the method for providing surgical information using a surgical image according to an embodiment of the present invention described above, the computer may display a specified organ region in the surgical image and provide it to medical staff performing an actual operation in real time. For example, when a surgical image is output on a screen provided to medical staff during an actual surgery, the computer can specify an organ region existing in the surgical image through the steps S100 to S400, so that the specified organ region is selected at this time. It can be displayed on the corresponding surgical image output on the screen. Accordingly, it is possible to more effectively provide meaningful information (important organ information) through the surgical image during the operation.

In addition, the computer may match the organ region specified in the surgical image on the virtual body model. Then, the computer may perform a simulation based on the organ region matched on the virtual body model.

Here, the virtual body model may be 3D modeling data generated based on medical image data (eg, medical images photographed through CT, PET, MRI, etc.) photographed in advance of the inside of the patient's body. . For example, it is modeled in conformity with the body of the subject to be operated on, and may be calibrated to the same state as the actual surgical state. Medical staff can perform rehearsals or simulations using a virtual body model that is implemented in the same way as the physical state of the subject to be operated on, and through this, they can experience the same state as during the actual operation. In addition, when performing virtual surgery using a virtual body model, virtual surgery data including rehearsal or simulation actions for the virtual body model can be obtained. For example, the virtual surgery data may be a virtual surgery image including a surgical site on which a virtual surgery has been performed on the virtual body model, or may be data recorded for a surgical operation performed on the virtual body model.

In one embodiment, since the computer can acquire the type of organ present in the surgical image, the position information of the organ, the positional relationship between the organs, the texture information of the organ, etc. It can be precisely matched on the model. Through this matching, the same situation that is currently being performed in real surgery can be implemented in the virtual body model. Therefore, the computer can perform simulation by applying surgical tools and surgical motions used in the surgical procedure actually performed by medical staff based on the organ region matched on the virtual body model. In other words, since the virtual body model can perform the same simulation as the current actual surgery, it can serve as an assistant to the medical staff to perform the surgery more accurately and effectively. In addition, it is possible to reproduce the same surgical procedure as the actual surgical situation during reoperation or learning, thereby improving the learning effect of medical staff.

In addition, the computer may obtain simulation data through simulation on the virtual body model, and generate cue sheet data for a corresponding surgical procedure based on the obtained simulation data.

In one embodiment, the computer may acquire simulation data recording a surgical site to be simulated, a surgical tool, a surgical operation, and the like by performing a simulation using the virtual body model. The computer may generate cue sheet data for the actual surgical procedure being performed by the medical staff based on the simulation data including these records.

Here, the cue sheet data may be data composed of information in which a surgical procedure is sequentially arranged according to time based on a surgical operation performed during surgery. In one embodiment, the cue sheet data may include surgical information about a meaningful surgical operation or a surgical operation that can be performed as a minimum unit. In this case, the surgical information may include surgical tool information, surgical site information, surgical operation information, and the like.

Through such cue sheet data, the same surgical operations as the actual surgical process can be implemented, and through this, the operation can be evaluated or used as a learning model in the future. In addition, the surgical process can be repeatedly reproduced using cue sheet data, and it can be effectively utilized in the same or similar actual surgery.

83 is a diagram schematically showing the configuration of an apparatus 200 for performing a method for providing surgical information using a surgical image according to an embodiment of the present invention.

Referring to FIG. 83 , the processor 210 includes one or more cores (not shown) and a graphic processing unit (not shown) and/or a connection path (eg, a bus) for transmitting and receiving signals with other components. ) may be included.

The processor 210 according to an embodiment executes one or more instructions stored in the memory 220, thereby performing the method of providing surgical information using the surgical image described in relation to FIGS. 79 to 82 .

In one example, the processor 210 acquires a surgical image for a specific surgery by executing one or more instructions stored in the memory 220, the step of recognizing a surgical step in the specific surgery corresponding to the surgical image, the Estimating an organ candidate group including at least one organ that can be extracted in the surgical step, and specifying an organ region in the surgical image based on the positional relationship between the organs in the organ candidate group have.

On the other hand, the processor 210 is a RAM (Random Access Memory, not shown) and ROM (Read-Only Memory: ROM) for temporarily and / or permanently storing signals (or data) processed inside the processor 210 . , not shown) may be further included. Also, the processor 210 may be implemented in the form of a system on chip (SoC) including at least one of a graphic processing unit, a RAM, and a ROM.

The memory 220 may store programs (one or more instructions) for processing and controlling the processor 210 . Programs stored in the memory 220 may be divided into a plurality of modules according to functions.

The method for providing surgical information using a surgical image according to an embodiment of the present invention described above may be implemented as a program (or application) and stored in a medium to be executed in combination with a computer, which is hardware.

Hereinafter, the operating time is predicted based on the surgical image, but for this purpose, embodiments of the technique for predicting the surgical time required in a specific surgical stage through learning of the surgical image are disclosed.

In one embodiment, there is provided a method of predicting an operation time based on a surgical image performed by a computer, the method comprising: obtaining a preset surgical image including a surgical operation for a specific surgical step; generating learning data using the preset surgical image and the operation time obtained based on the preset surgical image; and predicting an operation time in the specific operation stage by performing learning based on the learning data. A detailed description thereof will be described later with reference to FIGS. 84 to 88 .

With reference to FIGS. 84 to 88, a method of predicting an operation time based on a surgical image according to an embodiment of the present invention will be described in detail.

84 is a schematic diagram of a system capable of performing robotic surgery according to an embodiment of the present invention.

Referring to FIG. 84 , the robotic surgery system includes a medical imaging device 10 , a server 100 , and a controller 30 , a display 32 , and a surgical robot 34 provided in the operating room. According to an embodiment, the medical imaging device 10 may be omitted from the robotic surgery system according to the disclosed embodiment.

In one embodiment, the surgical robot 34 includes a photographing device 36 and a surgical tool 38 .

In one embodiment, the robotic surgery is performed by the user using the control unit 30 to control the surgical robot (34). In one embodiment, the robotic surgery may be performed automatically by the control unit 30 without the user's control.

The server 100 is a computing device including at least one processor and a communication unit.

The control unit 30 includes a computing device including at least one processor and a communication unit. In one embodiment, the controller 30 includes hardware and software interfaces for controlling the surgical robot 34 .

The photographing device 36 includes at least one image sensor. That is, the imaging device 36 includes at least one camera device, and is used to photograph an object, that is, a surgical site. In an embodiment, the imaging device 36 includes at least one camera coupled to a surgical arm of the surgical robot 34 .

In an embodiment, the image captured by the photographing device 36 is displayed on the display 340 .

In one embodiment, the surgical robot 34 includes one or more surgical tools 38 capable of performing cutting, clipping, fixing, grabbing operations of the surgical site, and the like. The surgical tool 38 is used in combination with the surgical arm of the surgical robot 34 .

The controller 30 receives information necessary for surgery from the server 100 or generates information necessary for surgery and provides it to the user. For example, the controller 30 displays generated or received information necessary for surgery on the display 32 .

For example, the user performs robotic surgery by controlling the movement of the surgical robot 34 by manipulating the controller 30 while viewing the display 32 .

The server 100 generates information necessary for robotic surgery by using the medical image data of the object photographed in advance from the medical imaging device 10 , and provides the generated information to the controller 30 .

The controller 30 provides the information received from the server 100 to the user by displaying it on the display 32 , or controls the surgical robot 34 using the information received from the server 100 .

In one embodiment, the means that can be used in the medical imaging equipment 10 is not limited, for example, CT, X-Ray, PET, MRI, etc., various other medical image acquisition means may be used.

As described above, when performing robotic surgery, it is possible to acquire data including surgical images captured in the surgical process or various surgical information in the control process of the surgical robot. As described above, in the present invention, it is an object of the present invention to provide a method that can be used in the surgical procedure of another patient through learning by using the surgical image or surgical data that can be obtained in the surgical procedure as learning data, and through this learning.

Hereinafter, for convenience of description, it will be described that a "computer" performs a method of predicting an operation time based on a surgical image according to an embodiment disclosed herein. “Computer” may mean the server 100 or the controller 30 of FIG. 84 , but is not limited thereto and may be used to encompass a device capable of performing computing processing. For example, the computer may be a computing device provided separately from the device shown in FIG. 84 .

In addition, the embodiments disclosed below may not be applied only in connection with the robotic surgery system shown in FIG. 84, but may be applied to all kinds of embodiments in which a surgical image may be acquired and utilized in the surgical process. For example, in addition to robotic surgery, it may be applied in connection with minimally invasive surgery such as laparoscopic surgery or surgery using an endoscope.

85 is a flowchart schematically illustrating a method of predicting an operation time based on an operation image according to an embodiment of the present invention.

Referring to FIG. 85 , the method of predicting an operation time based on a surgical image according to an embodiment of the present invention includes: acquiring a preset surgical image including a surgical operation for a specific surgical step (S100), the Step (S200) of generating learning data by using a preset surgical image and an operation time obtained based on the preset surgical image, and performing learning based on the learning data, so that the operation time in the specific surgical stage It may include a step of predicting (S300). Hereinafter, detailed description of each step is described.

The computer may acquire a preset surgical image including a surgical operation for a specific surgical step (S100).

When a specific surgery (eg, gastric cancer surgery, colon cancer surgery, etc.) is performed on a patient, medical staff may directly perform an actual operation on the patient, and use a laparoscopy or an endoscope as well as a surgical robot as described in FIG. 1 . Minimally invasive surgery can also be performed. In this case, the computer may acquire a surgical image obtained by photographing a scene including a surgical operation performed in the surgical procedure and related surgical tools, surgical sites, and the like. In an embodiment, the computer may acquire a surgical image that captures a scene including a surgical site and a surgical tool currently being operated from a camera that has entered the patient's body.

Meanwhile, each surgery (eg, gastric cancer surgery, colorectal cancer surgery, etc.) proceeds with different surgical procedures, but the same type of surgery may be performed with the same or similar series of surgical procedures. That is, when a specific surgery (eg, the same type of surgery, such as gastric cancer surgery) is performed on a plurality of patients, a specific surgical procedure may be performed by performing the same or similar surgical operations for each patient.

In one embodiment, a specific operation may be composed of predefined surgical procedures (ie, surgical steps) according to classification criteria. For example, the computer may classify the surgical steps according to the lapse of time during which a specific surgery is performed, or may classify the surgical steps corresponding to each surgical site based on the surgical site during a specific surgery. Alternatively, the computer may classify the surgical steps based on the position of the camera or the movement range of the camera during the specific surgery, and the surgical steps based on the change (eg, replacement, etc.) of the surgical tool during the specific surgery. can also be classified. The computer may classify each surgical step according to a specific classification standard as described above, and then pre-define the surgical operations performed in each classified surgical step.

In addition, the surgical steps constituting a specific surgery may be classified into a hierarchical structure. In one embodiment, a specific surgical procedure may be classified in stages from the lowest layer (level) to the highest layer (level) to form a hierarchical structure. Here, the lowest layer is composed of the smallest units representing the surgical procedure, and may include a minimum surgical operation having a meaning as one minimum operation. For example, the computer may recognize a surgical operation representing one constant operation pattern, such as cutting, grabbing, moving, etc., as a minimal surgical operation, and may configure it as the lowest level (ie, the lowest surgical stage). In addition, the computer may group at least one minimum surgical operation according to a specific classification criterion (eg, time lapse, surgical site, movement of a camera, change of surgical tool, etc.) and configure it as an upper layer. For example, when each minimal surgical operation representing grabbing, moving, cutting, etc. of a surgical tool is performed in a predetermined order and has a meaning as a specific operation by connecting the predetermined order, it can be grouped into one upper layer. have. As another example, when each minimal surgical operation such as clipping, moving, and cutting is performed continuously, it may be recognized that it is a surgical operation of cutting a blood vessel and grouped into one upper layer. As another example, when each minimal surgical operation such as grabbing, lifting, cutting, and removing is performed continuously, it may be recognized that this is a surgical operation to remove fat and grouped into one upper layer. In this way, when it is recognized as an upper surgical operation (ie, an upper surgical step) having a specific meaning from the lower-level surgical operations (ie, lower-level surgical steps), it can be classified into a higher-level operation and this process is repeated. can be done with Therefore, finally, a hierarchical structure can be formed up to the highest layer (ie, the highest surgical stage) of a specific surgical procedure. For example, a specific surgery may be expressed in a hierarchical structure such as a tree shape with surgical steps constituting the specific surgery.

Therefore, the surgical image obtained in step S100 is obtained by performing a specific surgery on the patient, and may include image frames obtained by photographing any one surgical step belonging to a specific layer in a specific surgical procedure consisting of a hierarchical structure. have. Here, the specific layer may mean the remaining middle layer except for the lowest layer and the highest layer in the hierarchical structure.

In addition, the surgical image acquired in step S100 may be composed of image frames including a series of pre-defined surgical operations for a specific surgical step of a specific surgery. In one embodiment, when a specific surgery of the same type is performed on a plurality of patients, the computer may acquire each surgical image corresponding to a specific surgical stage predefined in a specific surgical procedure from each of the plurality of patients. . In other words, each surgical image obtained from a plurality of patients may be set as image frames including predetermined surgical operations performed in a specific surgical stage. In this way, in setting image frames including predetermined surgical operations performed in a specific surgical stage with respect to the surgical image obtained from the patient, the medical staff or computer extracts only image frames corresponding to the specific surgical stage from the surgical image. It may have been created in advance.

Hereinafter, for convenience of explanation, a surgical image set with image frames including predetermined surgical operations performed in a specific surgical stage is referred to as a preset surgical image.

The computer may generate the learning data by using the preset surgical image acquired in step S100 and the operation required time acquired based on the preset surgical image (S200).

In one embodiment, the computer may acquire the first operation time required based on a preset operation image. Then, the computer may generate a censored surgical image in which an arbitrary section is removed from a preset surgical image, and acquire a second operation time required based on the censored surgical image. The computer may generate at least one of a preset surgical image and a censored surgical image as learning data based on the first operation time required and the second operation time required. A detailed process for this will be described with reference to FIG. 3 .

On the other hand, even if a specific surgical step is equally applied to a plurality of patients, variability may occur depending on the condition of each patient or the surgical method of each medical staff performing the operation. Therefore, when a specific surgical step is applied to each patient, the actual operation time required to perform the operation may be different for each patient. Therefore, it is difficult to predict the exact operation time or remaining operation time even if the same operation is performed for each patient. In order to solve this problem, in the present invention, it is possible to more accurately predict the surgical time required for each surgical stage by acquiring a surgical image including surgical operations performed in a specific surgical stage, and configuring it as learning data to perform learning. provide a way In addition, it provides a method for estimating the remaining surgical time based on the current surgical image obtained during the actual surgical procedure.

86 is a view for explaining a process of generating learning data based on a surgical image according to an embodiment of the present invention.

Referring to FIG. 86 , the computer may acquire preset surgical images from a plurality of patients (a first patient to an n-th patient).

At this time, each preset surgical image obtained from a plurality of patients is different from the image frames included in each surgical image even if the same surgical step is performed according to the patient's condition or the surgical method of the medical staff performing the operation, as described above. there may be Also, it is not easy for a medical staff or a computer to accurately determine the surgical operations defined in a specific surgical stage from the surgical image and to extract only the corresponding image frames. That is, each preset surgical image obtained from a plurality of patients is composed of image frames for the same surgical stage, but these image frames include not only the surgical operations defined in the corresponding surgical stage but also other surgical stages (eg, It may include surgical operations in the previous or subsequent surgical stage). In addition, even if it consists of the surgical operations defined in the corresponding surgical stage, it may include additional surgical operations other than the essential surgical operations due to various variability during surgery. It is difficult to determine the operation time required for a specific surgical stage only with these surgical images, and it is also not suitable for collectively applying to all patients.

Accordingly, in the present invention, learning data for learning is configured based on a preset surgical image obtained from a patient, and the operation time in the corresponding surgical image (ie, the corresponding surgical stage) is accurately predicted by learning it.

In an embodiment, the computer may generate learning data by applying a survival analysis technique. That is, the computer can generate learning data based on the analysis of the time required for surgery based on the surgical image by applying the survival analysis technique.

For example, first, the computer may generate a preset surgical image and a censored surgical image from which an arbitrary section is removed (S210). Here, the censored surgical image may be an image obtained by removing at least one image frame corresponding to an arbitrary section from a preset surgical image. As described above, the preset surgical image may include image frames of other surgical steps (eg, before or after surgery), and may include additional surgical motions that are not essential surgical motions in the corresponding surgical step. . As such, since the preset surgical image may not include only essential surgical operations of the corresponding surgical stage, the computer generates a censored surgical image in which an arbitrary section is removed from the preset surgical image. For example, an arbitrary section may be determined through a random function, or a section including the front or rear image frame of the surgical image may be determined as an arbitrary section. Alternatively, if there is an image frame determined to be a surgical operation different from a series of operations by analyzing the image frames in the surgical image, or there is an image frame determined to be different from a predetermined surgical operation to be performed in a specific surgical stage, the corresponding section may be determined as an arbitrary interval.

For example, as shown in FIG. 86 , the computer may generate first to nth preset surgical images and first to nth censored surgical images respectively obtained from the first to nth patients. In FIG. 86, the censored surgical image was indicated by image frames included in the box indicated by the dotted line.

Next, the computer acquires the surgical time required to perform a specific surgical step based on a preset surgical image (hereinafter, the first surgical time required), and performs a specific surgical step based on the censored surgical image. The required operation time (hereinafter, referred to as the second operation time required) may be obtained (S220).

For example, as shown in FIG. 86 , the computer uses the first to n-th preset surgical images for the first surgery duration (eg, Actual Duration in FIG. 3 ) and the first to n-th censored surgical images for the second surgery. The required time (eg, Censored Duration of FIG. 86) may be obtained and compared. In this case, the time required for each operation may be a time required when all the image frames in each surgical image are performed. For example, it may be the playback time of each surgical image.

Next, the computer may generate learning data using at least one of a pre-set surgical image and/or a censored surgical image based on the first operation time required and the second operation time required (S230).

For example, as shown in FIG. 86, the computer configures the first operation time required for the first preset surgical image (eg, Actual Duration of FIG. 86) and the second operation time required for the first censored surgical image (eg, FIG. 86), it is possible to determine the surgical image to be used as learning data among the two surgical images for the first patient. The computer may map a surgical image (eg, a first preset surgical image) determined as the learning data from the first patient and the operation time required therefor to generate the learning data.

According to an embodiment, in determining the surgical image to be used as the learning data by comparing the first operation time required with the second operation time required, the computer may use the length of the operation time required or a preset standard. Or it may be arbitrarily selected. For example, the computer may select a surgical image having a shorter required time from among the first required time for surgery and the second required time for surgery. Alternatively, the difference between the first operation time required and the second operation time required may be used. For example, when the difference between the first surgery time required and the second surgery time required is equal to or greater than a predetermined reference value, the computer may select a surgical image having the first required surgery time as the learning data.

As shown in Figure 86, the computer is based on the results of comparing the first to n-th preset surgical image first surgery time and the first to n-th censored surgical image, respectively, the second surgery time required, By selecting one of a preset surgical image or a censored surgical image, and mapping the corresponding surgical time, it can be configured as learning data for a specific surgical stage. For example, when a preset surgical image is selected, the computer may set the surgical status to "event" and record the surgical duration of the selected surgical image to define it as learning data. Alternatively, when a censored surgical image is selected, the computer may set the surgical status to "censored" and record the surgical duration of the selected surgical image to define it as learning data. That is, the computer may generate a learning data set using preset surgical images or censored surgical images obtained from a plurality of patients undergoing surgery for a specific surgical stage.

Referring back to FIG. 85 , the computer may predict the operation time in a specific operation step by performing learning based on the learning data generated in step S200 ( S300 ).

In one embodiment, the computer may perform learning by using each image frame in the learning data and the time required for surgery to perform a specific surgical step based on each image frame. The computer can predict the average surgical time required for a specific surgical stage through learning. For example, when a preset surgical image is generated as learning data, the computer may perform learning based on each image frame in the preset surgical image and the first required time for surgery. Alternatively, when a censored surgical image is generated as learning data, the computer may perform learning based on each image frame in the censored surgical image and the second required time for surgery. A detailed process for this will be described with reference to FIG. 87 .

87 is a diagram for explaining a process of performing learning based on learning data according to an embodiment of the present invention.

Referring to FIG. 87 , the computer may acquire learning data and perform learning using this deep learning. In this case, the learning data may be generated through the steps S210 to S230, and may be generated based on a preset surgical image or a censored surgical image.

In one embodiment, the computer may receive the learning data _{, and extract each image frame (L 1} to L _N ) from the learning data (S310).

For example, when the received learning data is a preset surgical image, the computer may extract each image frame included in the preset surgical image from the received learning data. Alternatively, when the received learning data is a censored surgical image, the computer may extract each image frame included in the censored surgical image from the received learning data.

The computer may perform learning using a convolutional neural network (CNN) from each extracted image frame, and as a result, extract feature information for each image frame (S320).

For example, the computer may learn features of the surgical image by inputting each image frame to at least one layer (eg, a convolution layer). As a result of this learning, the computer can infer what or what each image frame represents.

The computer may perform learning using a recurrent neural network (RNN) for each image frame derived as a result of learning using the CNN (S330).

For example, the computer may receive learning data in units of frames and perform learning using an RNN (eg, LSTM method). At this time, by inputting the characteristic information of each image frame, along with the operation state information of the corresponding learning data (eg, event/censored or not) and the operation time for the corresponding learning data, learning can be performed. In addition, the computer may perform learning while linking at least one image frame of the previous view with the image frame of the current view. Therefore, the computer learns the relationship between each image frame based on the characteristic information of each image frame, so that it is possible to provide information about specific surgical steps, such as whether it is pre-defined essential surgical operations in a specific surgical stage or includes surgical operations in other surgical stages. information can be grasped.

The computer can predict the median surgical duration in a specific surgical stage through the learning process using CNN and RNN as described above (S340).

For example, as described in FIG. 3 , the computer acquires each surgical image from a plurality of patients who have performed a specific surgical step and generates learning data based on this, so it The same learning as described in 87 can be applied repeatedly. Therefore, by repeatedly learning a large amount of learning data, the computer can predict the average operation time required for a specific surgical step from this.

According to an embodiment of the present invention, by performing the above-described steps S100 to S300, it is possible to predict the surgical time required to perform each surgical step in the final surgical procedure. Accordingly, the computer can build a learning model that predicts the operation time required for each operation stage.

In addition, in an embodiment of the present invention, it is possible to apply the surgery process of another patient by using the time required for surgery predicted in each surgical step as described above.

In one embodiment, when there is a patient currently undergoing actual surgery, the computer may acquire the actual surgical image of the patient to be operated on in real time. At this time, the computer may extract the image frame of the current time from the acquired actual surgical image, and recognize the operation stage currently being performed based on the extracted image frame of the current time. Therefore, the computer can calculate the required time for surgery from the current stage of surgery to the present time based on the image frame of the current time. In addition, as described above, the computer can determine the average operation time required for the current surgical stage through learning to predict the operation time required for each surgical stage. Therefore, the computer uses the average operation time of the current surgical stage predicted through learning and the operation time until the current time calculated through the image frame of the current time in which the actual surgery is currently in progress. It is possible to estimate the remaining surgical time required to perform surgical operations in the current surgical stage.

Therefore, in the present invention, it is possible to predict not only the required operation time for each operation stage, but also accurately predict the remaining operation time after the current time point. In this way, by providing the medical staff with the exact time required for surgery and the remaining time for surgery, it is possible to more accurately understand the status of the current surgery, and through this, the remaining surgical procedure can be performed efficiently.

In addition, it is necessary to know the exact time required for surgery and the remaining time for surgery so that the operation can be performed according to the patient's condition. In particular, anesthesia time is important during surgery, and since the present invention can provide the required time for surgery and the remaining time for surgery in real time, it is possible to calculate an accurate anesthetic dose. Furthermore, it is effective to determine an appropriate additional anesthetic dose according to the remaining surgery time.

88 is a diagram schematically illustrating the configuration of an apparatus 200 for performing a method of predicting an operation time based on an operation image according to an embodiment of the present invention.

Referring to FIG. 88, the processor 210 includes one or more cores (not shown) and a graphic processing unit (not shown) and/or a connection path (eg, a bus) for transmitting and receiving signals with other components. ) may be included.

The processor 210 according to an embodiment executes one or more instructions stored in the memory 220, thereby performing a method of predicting an operation time based on the operation image described in relation to FIGS. 85 to 87 .

In one example, the processor 210 executes one or more instructions stored in the memory 220 to obtain a preset surgical image including a surgical operation for a specific surgical step, the preset surgical image and the preset surgical image It is possible to perform the steps of generating learning data using the operation required time obtained based on, and performing learning based on the learning data to predict the operation time in the specific operation stage.

On the other hand, the processor 210 is a RAM (Random Access Memory, not shown) and ROM (Read-Only Memory: ROM) for temporarily and / or permanently storing signals (or data) processed inside the processor 210 . , not shown) may be further included. Also, the processor 210 may be implemented in the form of a system on chip (SoC) including at least one of a graphic processing unit, a RAM, and a ROM.

The memory 220 may store programs (one or more instructions) for processing and controlling the processor 210 . Programs stored in the memory 220 may be divided into a plurality of modules according to functions.

The method for estimating the operation time based on the operation image according to the embodiment of the present invention described above may be implemented as a program (or application) to be executed in combination with a computer, which is hardware, and stored in a medium.

Hereinafter, embodiments relating to a technique for evaluating different degrees of recognition of a surgical image according to medical importance or a medical evaluation standard based on various surgical-related information recognized from the surgical image are disclosed.

In one embodiment, there is provided a method for evaluating the degree of recognition of a surgical image performed by a computer, the method comprising: acquiring a surgical image including a plurality of image frames; Recognizing at least one surgical recognition element that can be recognized as a component of surgery for each of the plurality of image frames, calculating a predicted recognition value for each of the at least one surgical recognition element; and calculating a representative recognition value for evaluating the degree of recognition for the surgical image based on the predicted recognition value for each of the at least one surgical recognition element. A detailed description thereof will be described later with reference to FIGS. 89 to 93 .

With reference to FIGS. 89 to 93, a method for evaluating recognition of a surgical image according to an embodiment of the present invention will be described in detail.

89 is a schematic diagram of a system capable of performing robotic surgery according to an embodiment of the present invention.

Referring to FIG. 89 , the robotic surgery system includes a medical imaging device 10 , a server 100 , and a controller 30 , a display 32 , and a surgical robot 34 provided in the operating room. According to an embodiment, the medical imaging device 10 may be omitted from the robotic surgery system according to the disclosed embodiment.

In one embodiment, the surgical robot 34 includes a photographing device 36 and a surgical tool 38 .

In one embodiment, the robotic surgery is performed by the user using the control unit 30 to control the surgical robot (34). In one embodiment, the robotic surgery may be performed automatically by the control unit 30 without the user's control.

The server 100 is a computing device including at least one processor and a communication unit.

The control unit 30 includes a computing device including at least one processor and a communication unit. In one embodiment, the controller 30 includes hardware and software interfaces for controlling the surgical robot 34 .

The photographing device 36 includes at least one image sensor. That is, the imaging device 36 includes at least one camera device, and is used to photograph an object, that is, a surgical site. In an embodiment, the imaging device 36 includes at least one camera coupled to a surgical arm of the surgical robot 34 .

In an embodiment, the image captured by the photographing device 36 is displayed on the display 340 .

In one embodiment, the surgical robot 34 includes one or more surgical tools 38 capable of performing cutting, clipping, fixing, grabbing operations of the surgical site, and the like. The surgical tool 38 is used in combination with the surgical arm of the surgical robot 34 .

The controller 30 receives information necessary for surgery from the server 100 or generates information necessary for surgery and provides it to the user. For example, the controller 30 displays generated or received information necessary for surgery on the display 32 .

For example, the user performs robotic surgery by controlling the movement of the surgical robot 34 by manipulating the controller 30 while viewing the display 32 .

The server 100 generates information necessary for robotic surgery by using the medical image data of the object photographed in advance from the medical imaging device 10 , and provides the generated information to the controller 30 .

The controller 30 provides the information received from the server 100 to the user by displaying it on the display 32 , or controls the surgical robot 34 using the information received from the server 100 .

In one embodiment, the means that can be used in the medical imaging equipment 10 is not limited, for example, CT, X-Ray, PET, MRI, etc., various other medical image acquisition means may be used.

As described above, when performing robotic surgery, it is possible to acquire data including surgical images captured in the surgical process or various surgical information in the control process of the surgical robot. In this way, the surgical images or surgical data acquired in the surgical process can be used as learning materials, and can also be used in the surgical process of other patients. In addition, since the medical importance or evaluation criteria may be different depending on the type of surgery or the medical staff, different recognition levels are required from the surgical image or surgical data. Accordingly, the present invention provides a method for recognizing and evaluating surgical images with different recognition levels according to medical importance or evaluation criteria.

Hereinafter, for convenience of description, it will be described that a "computer" performs a method of generating learning data based on a surgical image according to an embodiment disclosed herein. “Computer” may mean the server 100 or the controller 30 of FIG. 89, but is not limited thereto and may be used to encompass a device capable of performing computing processing. For example, the computer may be a computing device provided separately from the device shown in FIG. 89 .

In addition, the embodiments disclosed below may not be applied only in connection with the robotic surgery system shown in FIG. For example, in addition to robotic surgery, it may be applied in connection with minimally invasive surgery such as laparoscopic surgery or surgery using an endoscope.

90 is a flowchart schematically illustrating a method for evaluating a degree of recognition of a surgical image according to an embodiment of the present invention.

Referring to FIG. 90 , in the method for evaluating the recognition of a surgical image according to an embodiment of the present invention, the step of acquiring a surgical image including a plurality of image frames (S100), the configuration of surgery for each of the plurality of image frames Recognizing at least one surgical recognition element that can be recognized as an element, calculating a predicted recognition value for each of the at least one surgical recognition element (S200), and predictive recognition for each of the at least one surgical recognition element It may include calculating a representative recognition value for evaluating the degree of recognition for the surgical image based on the value (S300). Hereinafter, detailed description of each step is described.

The computer may acquire a surgical image including a plurality of image frames (S100).

When a specific operation (eg, gastric cancer surgery, colon cancer surgery, etc.) is performed on a patient, medical staff may directly perform an actual operation on the patient, and use a laparoscopy or an endoscope as well as a surgical robot as described in FIG. 1 . Minimally invasive surgery can also be performed.

In this case, the computer may acquire a surgical image obtained by photographing a scene including a surgical operation performed in the surgical procedure and related surgical tools, surgical sites, and the like. In an embodiment, the computer may acquire a surgical image that captures a scene including a surgical site and a surgical tool currently being operated from a camera that has entered the patient's body.

The surgical image may include one or more image frames. Each image frame may represent one scene in which a surgical operation is performed, including a surgical site of a subject to be operated, a surgical tool, and the like. For example, the surgical image may be composed of image frames in which a surgical operation according to time is recorded for each scene in a surgical procedure. Alternatively, the surgical image may be composed of image frames in which each surgical scene is recorded according to spatial movement, such as a position of a surgical site or a camera, during a surgical procedure.

In addition, the surgical image may be composed of all image frames including the entire surgical procedure, but may also be formed in the form of at least one video clip divided according to a specific classification criterion.

According to an embodiment, the computer may acquire a surgical image in the form of a video clip divided into predefined sections for a specific surgery, or acquire a surgical image including all or a part of the surgical procedure, and use it as at least one video clip It can also be divided into surgical images of the form.

In an embodiment, the computer may acquire a surgical image including all or part of a surgical procedure for a specific surgery, and divide it into a surgical image in the form of at least one video clip according to a specific classification criterion. For example, the computer may divide the surgical image according to the lapse of time during which the operation is performed, or the surgical image may be divided according to the position or state change of the surgical site based on the surgical site during surgery. Alternatively, the computer may divide the surgical image based on the position of the camera or the movement range of the camera during the operation, and divide the surgical image based on the change (eg, replacement, etc.) of the surgical tool during the operation. You may. Alternatively, the computer may divide the surgical image based on image frames made up of a series of surgical operations in relation to the surgical operation. In addition, for each operation, a surgical stage classified according to a specific classification criterion may be predetermined. In this case, the computer may divide the surgical image based on the surgical stage.

Here, the surgical image divided in the form of a video clip may be independently configured without overlapping image frames between the divided video clip images, or there may be image frames overlapping each other between the divided video clip images.

That is, the computer may acquire at least one surgical image divided into predetermined sections according to a specific classification criterion, and perform steps S200 to S300 described later.

The computer recognizes at least one surgical recognition element that can be recognized as a component of surgery for each of a plurality of image frames in the surgical image obtained in step S100, and predictive recognition for each recognized at least one surgical recognition element A value can be calculated (S200).

According to an embodiment, the computer recognizes and predicts a surgical recognition element not only for all of the plurality of image frames in the surgical image (ie, a specific video clip), but also for any image frame in the surgical image (ie, a specific video clip). A recognition value can also be calculated. That is, the arbitrary image frame refers to an image frame randomly (randomly) selected from all image frames in the surgical image (ie, a specific video clip).

As described above, since each image frame in the surgical image records a surgical scene, each image frame includes various surgical information related to the surgical scene. For example, it includes various surgical information such as surgical tools, surgical operations, surgical sites (ie, body parts such as organs, blood vessels, tissues, etc.), and whether or not bleeding occurs. Accordingly, in the present invention, each component that can be recognized as surgical information from an image frame is defined as a surgical recognition element. The surgical recognition element may be a concept corresponding to an object, but in the present invention, it refers to a broad concept encompassing not only the object but also information such as motion, function, and state related to the object.

In one embodiment, the computer may determine whether a surgical recognition element is recognized for each of a plurality of image frames in the surgical image. Here, the surgical recognition element may include at least one of a surgical stage, a body part, whether an event has occurred, an operation time, a surgical tool, camera information, and a surgical operation. In addition, the operation recognition element may include a lower level operation recognition element for each element. According to an embodiment, a surgical recognition element to be recognized from the image frame may be predefined.

And, based on the recognition result of the surgical element, the computer may calculate a predicted recognition value for each surgical recognition element based on the recognition ratio of the image frame recognized the surgical recognition element for the entire plurality of image frames.

Table 4 below shows an example of calculating the predicted recognition value (SOM) by recognizing the surgical recognition elements (A to G) for a plurality of image frames (first to Kth image frames) in the surgical image.

Surgical Recognition Elements Sub-Surgery Recognition Elements 1st video frame 2nd video frame ... Kth video frame SOM_Sub SOM_val A. Recognition of the surgical stage
(Phase Recognition) Workflow Analysis T T ... T 0.91 0.89 ... B. Body part recognition
(Organ Recognition) Liver Segmentation F F ... F 0.45 0.37 Spleen Segmentation F F ... T 0.32 Kidney Segmentation F T ... F 0.35 ... ... C. Event Recognition
(Event Recognition) Bleeding Detection T F ... F 0.54 0.62 Vessel Dissection F T T 0.68 ... ... D. Recognition of operating time
(Time Recognition) Remain Surgical Duration for overall F T ?? T 0.87 0.85 ... ... E. Surgical Instrument Recognition
(Instrument
Recognition) Harmony Detection F T ... T 0.94 0.94 Bipolar Detection T T ... T 0.97 Stapler Detection T F ... T 0.93 ... ... F. Camera Recognition
(Camera Recognition) Camera moving detection T T ... T 0.99 0.95 Camera coordinate prediction T T ... T 0.91 ... ... G. Recognition of surgical movements
(Action Recognition) surgical gesture classification T F ... T 0.64 0.64 surgical action representation T F ... T 0.62 ... ...

In calculating the predicted recognition values for the surgical recognition elements, as described with reference to Table 4, the computer can determine whether the surgical recognition elements (A to G) are recognized for each of the first to Kth image frames. . For example, if the computer recognizes the surgical stage from the first image frame, it may determine it as 'T (True)', and if it does not recognize the body part from the first image frame, it may determine it as 'F (False)'. At this time, when a sub-element is included for each surgical recognition element, the computer may determine whether to recognize each sub-element, and calculate a determination result. In this way, the computer may recognize the surgical recognition elements (A to G) for each of the first to Kth image frames, and calculate a recognition result value (eg, T/F) as shown in Table 4.

According to an embodiment, in the case of a surgical recognition element that is not expressed as a binary value such as T/F, such as an operation stage or operation time, the computer may perform a pre-processing process so that it can be expressed in a discrete form. For example, in the case of a surgical stage, the computer may classify the surgical stage into a specific number of classes, and then determine whether a specific scene belongs to each classified class as T/F. In the case of surgery time, the computer can be expressed in discrete form through probability distribution, for example, can be determined as T/F by recognizing whether the recognized surgery time is included in a specific range.

In addition, although the recognition result of the surgical recognition element is expressed as a binary value such as T/F as an example, this is only an example and may be expressed in various ways.

The computer may calculate a predicted recognition value for each surgical recognition element based on the recognized recognition result as shown in Table 1. For example, when calculating the predicted recognition value for the A surgical recognition element (eg, the surgical stage), the computer recognizes that the A surgical recognition element (eg, the surgical stage) is recognized for the entire first to Kth image frames. Based on the ratio of the image frame, it is possible to calculate a predicted recognition value for the A surgical recognition element (eg, surgery stage). In this way, the computer may calculate a predicted recognition value for each of the B to G surgical recognition elements. In addition, when a lower element is included for each surgical recognition element, the computer calculates a predicted recognition value (SOM_sub) for each lower element, and predicts the upper element based on the predicted recognition value (SOM_sub) for the lower elements A recognition value (SOM_val) may be calculated. In this case, a summary of the predicted recognition value (SOM_sub) for each lower element as a representative value of the corresponding upper element may be used as the predicted recognition value (SOM_val) of the corresponding upper element. According to an embodiment, the computer may calculate a predicted recognition value after applying a statistical model for removing outliers to the recognition result value for each surgical recognition element.

Here, the predicted recognition value refers to the degree of recognition of each surgical recognition element recognized in the surgical image (the entire plurality of image frames). In this specification, the prediction recognition value is referred to as a Surgical Occurrence Map (SOM).

The computer may calculate a representative recognition value for evaluating the degree of recognition for the surgical image based on the predicted recognition value for each of the at least one surgical recognition element calculated in step S200 (S300).

Here, the representative recognition value may be used as an index for determining the overall recognition rate for a specific surgical image based on the recognition degree of each surgical recognition element recognized in the surgical image (the entire plurality of image frames). This representative recognition value can be used to understand the degree of abstraction of the surgical image, and the degree of abstraction refers to the degree to which the medical staff can grasp a specific meaning through the surgical image. In this specification, the representative recognition value is referred to as a Surgical Abstract Metric (SAM).

The process of calculating the representative recognition value for the surgical image will be described in detail with reference to FIGS. 91 and 92 .

91 is a view for explaining a process of calculating a representative recognition value for a surgical image according to an embodiment of the present invention.

Referring to FIG. 91 , the computer may acquire a surgical image including a plurality of image frames (S100). As described above, the surgical image may be composed of at least one video clip. In this case, the computer may perform a process of calculating a predicted recognition value (SOM) and a representative recognition value (SAM) for each video clip.

The computer may recognize a surgical recognition element for each of the plurality of image frames, and calculate a predicted recognition value (SOM) for each surgical recognition element (S200). For example, the computer may calculate a predicted recognition value (SOM) for each surgical recognition element as shown in Table 1 above. At this time, the predicted recognition value (SOM) for each surgical recognition element may be expressed in the form of a map as shown in FIG. In addition, the predicted recognition value (SOM) expressed in the form of a map may be mathematically expressed as a matrix.

The computer may acquire an actual recognition value for each of the surgical recognition elements calculated by recognizing the surgical recognition elements in advance for each of the plurality of image frames (S210).

In an embodiment, the ground truth (GT) recognition value (SOM) for the surgical recognition element may be a value obtained based on the medical staff directly recognizing and labeling each surgical recognition element for a plurality of image frames. Alternatively, it may be a value obtained by pre-calculating each surgical recognition element through learning such as supervised learning or unsupervised learning for a plurality of image frames.

The actual recognition value (GT SOM) for each surgical recognition element may be expressed in the form of a map as shown in FIG. 3 , and through this, the recognition level of each surgical recognition element can be grasped. In addition, the actual recognition value (GT SOM) expressed in the form of a map may be mathematically expressed as a matrix.

The computer may calculate a Surgical Abstract Difference (SAD) between the predicted recognition value (SOM) and the actual recognition value (GT SOM) for each surgical recognition element (S220).

In one embodiment, since the predicted recognition value (SOM) and the actual recognition value (GT SOM) for each surgical recognition element can be expressed in a matrix, the computer calculates the difference between the two matrices, thereby the difference value for each surgical recognition element (SAD) can be calculated. At this time, the difference value (SAD) for each surgical recognition element may also be calculated as a matrix.

The computer may calculate a representative recognition value (SAM) for the surgical image based on the difference value (SAD) for each surgical recognition element (S300).

In an embodiment, the computer may summarize the difference value (SAD) matrix for each surgical recognition element into one representative value and derive it as a representative recognition value (SAM). For example, the computer may calculate an average of a difference value (SAD) matrix and use it as a representative recognition value (SAM).

According to an embodiment, in calculating the representative recognition value for the surgical image, the importance of each surgical recognition element may be reflected. A detailed process for this will be described with reference to FIG. 92 .

92 is a view for explaining a process of calculating a representative recognition value for a surgical image according to the importance of the surgical recognition element according to an embodiment of the present invention.

In an embodiment, the computer may obtain the importance for each surgical recognition element, and calculate the representative recognition value (SAM) by reflecting the importance in the difference value (SAD) for each surgical recognition element.

Here, the importance of each surgical recognition element refers to the medical importance or medical evaluation criteria for the surgical recognition element, which may be evaluated differently depending on the type of surgery or the medical staff. Therefore, it is possible to improve the qualitative judgment through the surgical image by providing different degrees of recognition according to the type of surgery or the medical staff.

In one embodiment, the computer may acquire weight information that evaluates the importance of each surgical recognition element from a plurality of medical staff. The computer may calculate an average value of each weight (importance) evaluated by a plurality of medical staff for each surgical recognition element, and reflect the average value of the calculated weight in the difference value (SAD) for each surgical recognition element.

For example, referring to FIG. 92 , a weight may be defined by three axes, and may be expressed as _{W i, j, and k.} Here, i may be a surgical recognition element, j may be a medical staff, and k may be a surgical image (video clip). For example, when the weights for each surgical recognition element obtained from each of a plurality of medical staff for a specific surgical image are configured in a matrix, the computer arranges each of the plurality of medical personnel in a row of a matrix and arranges each of the surgical recognition elements in a column of the matrix can do. In this case, by calculating the average for each column, the computer can obtain an average weight obtained by averaging the importance evaluated by a plurality of medical staff for each surgical recognition element, and the average weight can be used. Alternatively, the median value, the mode value, the percentile value, etc. may be obtained for each column in addition to the average weight according to the characteristics of the medical staff or importance.

In addition, when a plurality of medical staff evaluates the weight (importance) for each surgical recognition factor, each medical team weights the weight to satisfy the condition that the sum of weights that can be given to all surgical recognition factors becomes a specific value (eg, 1). can be set.

The computer can finally calculate the representative recognition value (SAM) by reflecting the weight (importance) of each surgical recognition element in the difference value (SAD).

In an embodiment, the representative recognition value (SAM) for the surgical image may be calculated as in Equation 8 below.

Here, SAM _V is a representative recognition value (SAM) for a specific surgical image v. S is a set of surgical recognition elements, for example, each of the surgical recognition elements shown in Table 1 may be S = {P, O, E, T, I, C, A}. |S| is the total number of elements in S. W _i is a weight vector for each surgical recognition element. SAD _i is the difference value for each surgical recognition element, and is the difference value between the predicted recognition value of each surgical recognition element and the actual recognition value of each surgical recognition element.

According to Equation 8, the representative recognition value (SAM) can be viewed as a linear combination of the i-th row (surgery recognition element) vector and the weight vector of the SAD matrix. In this case, since the representative recognition value (SAM) is the overall average of the difference between the predicted recognition value and the actual recognition value, the closer to 0, the better the recognition rate can be evaluated.

According to the present invention, rather than simply determining whether specific surgical information (surgery recognition element) is included in the surgical image, it is possible to check whether the medical staff recognizes each surgical recognition element at a desired level.

In addition, according to the present invention, it is possible to provide different recognition levels according to the purpose for which the medical staff intends to utilize the surgical image, and accordingly, the surgical image can be efficiently used.

In addition, according to the present invention, since medical importance or medical evaluation criteria can be reflected in calculating the degree of recognition for a surgical image, it is possible to increase the recognition rate for specific factors (eg, surgery time, surgical operation, surgical tools, etc.). In addition, by adjusting the analysis section (eg, unit of the image frame) of the surgical image, the recognition rate for a specific element (eg, operation time, operation operation, surgical tool, etc.) can be increased or the recognition level can be increased.

93 is a diagram schematically showing the configuration of an apparatus 200 for performing a method for evaluating a recognition degree of a surgical image according to an embodiment of the present invention.

Referring to FIG. 93 , the processor 210 includes one or more cores (not shown) and a graphic processing unit (not shown) and/or a connection path (eg, a bus) for transmitting and receiving signals with other components. ) may be included.

The processor 210 according to an embodiment executes one or more instructions stored in the memory 220, thereby performing the method for evaluating the recognition of the surgical image described in relation to FIGS. 90 to 92 .

In one example, the processor 210 may be recognized as a component of surgery for each of the plurality of image frames, the process of acquiring a surgical image including a plurality of image frames by executing one or more instructions stored in the memory 220 Recognizing at least one surgical recognition element in A step of calculating a representative recognition value for evaluating the degree of recognition of the user may be performed.

On the other hand, the processor 210 is a RAM (Random Access Memory, not shown) and ROM (Read-Only Memory: ROM) for temporarily and / or permanently storing signals (or data) processed inside the processor 210 . , not shown) may be further included. Also, the processor 210 may be implemented in the form of a system on chip (SoC) including at least one of a graphic processing unit, a RAM, and a ROM.

The memory 220 may store programs (one or more instructions) for processing and controlling the processor 210 . Programs stored in the memory 220 may be divided into a plurality of modules according to functions.

The method for evaluating the degree of recognition of a surgical image according to an embodiment of the present invention described above may be implemented as a program (or application) to be executed in combination with a computer, which is hardware, and stored in a medium.

Hereinafter, as a technique for optimizing surgery, embodiments of a technique for providing an optimal surgical procedure using a genetic algorithm or providing an optimized position or operation of a surgical tool using a virtual body model are disclosed.

In one embodiment, there is provided a method for optimizing surgery performed by a computer, the method comprising: generating a plurality of genes corresponding to the surgical procedure based on a surgical procedure comprising at least one detailed surgical operation; performing virtual surgery on each of the plurality of genes to evaluate whether the surgery is optimized; selecting at least one gene from among the plurality of genes based on the evaluation result and applying a genetic algorithm; and generating a new gene by applying the genetic algorithm, and deriving an optimal surgical procedure based on the new gene. A detailed description thereof will be described later with reference to FIGS. 94 to 99 .

In another embodiment, there is provided a method for providing an optimized surgical tool performed by a computer, the method comprising: acquiring a virtual body model generated in accordance with a physical condition of a subject to be operated; simulating a surgical operation using a surgical tool in the virtual body model; and deriving a configuration of the surgical tool suitable for applying the surgical operation performed by the surgical tool in the internal space of the body of the surgical subject based on the simulation result. A detailed description thereof will be described later with reference to FIGS. 100 to 104 .

A surgical optimization method using a genetic algorithm according to an embodiment of the present invention will be described in detail with reference to FIGS. 94 to 99 .

In the present specification, "gene" may be a concept corresponding to a solution of a problem used in a genetic algorithm. That is, "gene" may be a concept corresponding to a single surgical procedure in a genetic algorithm process used to derive an optimal surgical procedure. For example, "gene" may be expressed using surgical cue sheet data composed of at least one detailed surgical operation. In addition, "genes" can be reconstructed into surgical cue sheet data consisting of detailed surgical operations optimized according to the evolution process by a genetic algorithm.

In the present specification, "detailed surgical operation" may mean a minimum operation unit constituting a surgical process.

In the present specification, "surgical cue sheet data" is data that records a specific surgical procedure, and may be data composed of at least one detailed surgical operation.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

94 is a flowchart illustrating a surgical optimization method according to an embodiment of the present invention.

Although the method of FIG. 94 is described as being performed by a computer for convenience of description, the subject performing each step is not limited to a specific device and may be used to encompass devices capable of performing computing processing. That is, in the present embodiment, the computer may refer to a device capable of performing the surgery optimization method according to the embodiment of the present invention.

Referring to FIG. 94 , the surgical optimization method according to an embodiment of the present invention includes generating a plurality of genes corresponding to the surgical process based on a surgical process comprising at least one detailed surgical operation (S100), a plurality of performing virtual surgery on each of the genes to evaluate whether it is an optimized surgery (S110), selecting at least one gene among a plurality of genes based on the evaluation result and applying a genetic algorithm (S120), and It may include the step of generating a new gene by applying a genetic algorithm, and deriving an optimal surgical procedure based on the new gene (S130). Hereinafter, detailed description of each step is described.

The computer may generate a plurality of genes corresponding to the surgical procedure based on a surgical procedure comprising at least one detailed surgical operation (S100).

In one embodiment, the computer may acquire surgical cue sheet data comprising at least one detailed surgical operation, and generate a first gene based on the acquired surgical cue sheet data. In addition, the computer may generate a second gene by acquiring other surgical cue sheet data, and may generate a plurality of genes by repeatedly performing this process.

The detailed surgical operation refers to the minimum operation unit constituting the surgical process, and may be, for example, the minimum unit of the surgical operation divided according to a specific criterion in the surgical process. For example, the detailed surgical operation includes the type of operation (eg, laparoscopic surgery, robotic surgery, surgery using an endoscope, etc.), anatomical body parts where the surgery is performed, the surgical tools used during surgery, the number of surgical tools, and the operation on the screen. It may be generated by dividing the surgical operation into the smallest unit based on the direction or position of the tool, the movement of the surgical tool (eg, forward/retreat, etc.). Accordingly, the detailed surgical operation may include at least one of operation type information, operation operation type information, operation site information, and surgical tool information.

The surgical cue sheet data is data that records a specific surgical procedure, and may consist of at least one detailed surgical operation. For example, the surgical cue sheet data may be one in which a specific completed surgical procedure is sequentially arranged in a detailed surgical operation.

Surgical cue sheet data composed of detailed surgical motions and detailed surgical motions may be generated based on actual surgical data obtained in an actual surgical procedure. For example, medical staff may perform surgery directly on a patient, or may perform minimally invasive surgery using a surgical robot, a laparoscope, an endoscope, or the like. Various information (ie, actual surgical data) regarding a surgical operation performed in this surgical procedure, a surgical tool related to a surgical operation, a surgical site, and the like may be acquired. For example, surgical image data obtained by photographing a surgical procedure centered on a surgical site may be acquired as actual surgical data, or data recorded on a surgical operation performed in the surgical process may be acquired as actual surgical data. In addition, the surgical cue sheet data composed of the detailed surgical operation and the detailed surgical operation may be generated based on virtual surgery data obtained by performing a virtual surgery (eg, surgical simulation). For example, the virtual surgery data may be image data obtained by performing a virtual surgery on a patient using a simulator, or may be data recorded on a surgical operation performed in a virtual surgery process.

The computer may acquire the surgical cue sheet data generated based on the actual surgical data or the virtual surgery data as described above, and may generate a gene corresponding to the acquired surgical cue sheet data.

As described above, a gene is a concept corresponding to a solution of a problem used in a genetic algorithm, and in an embodiment of the present invention, one surgical procedure can be expressed as one gene. That is, the gene may be data representing a surgical procedure composed of at least one detailed surgical operation.

95 is a diagram illustrating a process of generating a gene according to an embodiment of the present invention.

Referring to FIG. 95 , the computer may generate one gene corresponding to one specific surgical procedure (ie, surgical cue sheet data). That is, the computer may generate a first gene corresponding to the first surgical procedure (ie, first surgical cue sheet data), and generate a second gene corresponding to the second surgical procedure (ie, second surgical cue sheet data) can do. Therefore, the computer can generate n genes by repeating this process n times. In this case, n genes may be generated in a random manner. In addition, each of the n genes may consist of one completed surgical procedure.

For example, as shown in FIG. 95 , when the first surgical procedure consists of m detailed surgical operations such as cutting, hemostasis, and grabbing, the computer arranges the m detailed surgical operations in order to perform the first operation The first gene may be generated by acquiring the cue sheet data and configuring it in a specific data form (eg, bit string) for representing the gene.

Referring back to FIG. 94 , the computer may evaluate whether the surgery is optimized by performing virtual surgery on each of the plurality of genes ( S110 ).

In one embodiment, the computer may perform virtual surgery on each of a plurality of genes based on detailed surgical operations in the surgical cue sheet data. The computer may calculate a fitness value for each gene through virtual surgery for each of a plurality of genes, and may evaluate whether the operation corresponding to each gene is an optimized operation based on the fitness.

In performing the virtual surgery on each gene, since each gene is generated based on the surgical cue sheet data composed of detailed surgical motions, the computer can perform the virtual surgery according to the detailed surgical motions within each gene. For example, detailed surgical operation includes information on the type of operation (eg, information on laparoscopic surgery, robotic surgery, surgery using an endoscope, etc.), information on the type of operation (eg, information on cutting, hemostasis, grabbing, etc.), information on the surgical site (inside the body) information such as the location and type of a specific surgical site where surgery is being performed, such as organs, blood vessels, and tissues of the Since it is data described including

In calculating the fitness for each gene, the computer may calculate the fitness by using at least one of information about success or failure of surgery, information related to surgery time, and information related to a surgical tool. The computer can use the optimization objective function to derive the optimal solution as a result of the virtual surgery for each gene, and at this time, information on the success of the operation, information on the operation time, information on the surgical tool, and information on the optimization objective function can be used to calculate the fitness for the gene.

In an embodiment, the computer may calculate the fitness by using an optimization objective function as shown in Equation 9 below.

Here, x is a gene, and may be information composed of a bit string including surgical cue sheet data.

w ₁ to w ₄ are weights, and for example, may be determined based on the importance of each variable through learning.

s is information indicating whether the operation was successful, for example, 0 may indicate failure and 1 may indicate success.

b may be (total bleeding time) / (total surgery time).

L(A) may be 1 / (sum of the total distance traveled by the surgical instruments in set A).

c may be 1 / (number of cuts).

d may be 1/ (total operative time).

A may be a set having all surgical tools used for surgery as elements, for example, A = {tool 1, tool 2, tool 3, ... } may be a set.

a may be 1 / n(A), or 1 / (total number of surgical instruments used in surgery).

According to Equation 9, the computer optimizes the operation in which the operation must be successful, the overall operation time is short, the bleeding time is short, the movement distance of the surgical tool is short, the number of cuts is small, and the number of surgical tools used is small. It can be evaluated surgically. That is, the computer can calculate the variables of the optimization objective function as in Equation 9 by performing virtual surgery on each gene, and can finally calculate the fitness for the gene by substituting each variable into Equation 9.

The fitness may be determined according to a desired characteristic to be evolved. Equation 9 above is only an example for calculating the degree of fit, and depending on what the targeted and optimized surgery is, whether the operation was successful or not, information about the operation time (eg total operation time, bleeding time), information about the surgical tool Factors such as (eg, number of surgical tools, movement distance) and surgical operation information (eg, number of cuttings, etc.) may be changed.

The computer may apply the genetic algorithm by selecting at least one gene evaluated as an optimized operation among a plurality of genes based on the evaluation result according to step S110 (S120).

In an embodiment, the computer may select at least one gene having a fitness that meets a preset condition from among a plurality of genes based on the fitness of each gene calculated to evaluate whether the operation is optimized. For example, the computer may set the condition to select a gene having a fitness level greater than or equal to a specific reference value, or may set the condition to select a gene having the greatest fitness level. Thereafter, the computer may apply a genetic algorithm to the selected gene according to a preset condition. For example, genetic algorithms such as selection, crossover, mutation, and replacement may be applied.

96 is a diagram illustrating a process of crossing genes as an example of applying a genetic algorithm according to an embodiment of the present invention.

In one embodiment, the computer generates n genes as shown in FIG. 95 and performs virtual surgery on each of the n genes to calculate the fitness of each gene (eg, fitness using Equation 1). have. The computer may select a gene having the greatest fitness from among the n genes or select a gene having a fitness greater than or equal to a specific reference value based on the fitness calculated for each of the n genes. In this case, a method of selecting a gene may vary depending on which conditions are set.

For example, if two genes (eg, a second gene and a third gene) having a fitness value (eg, the largest fitness value) matching the condition are selected from among n genes, the computer generates two genes (eg, the second gene). gene and a third gene) can be crossed.

Referring to FIG. 96 (a), the computer sets a specific point 200 for crossing the second gene and the third gene, and crosses the second gene and the third gene based on the set specific point 200. can do it

Referring to (b) of FIG. 96 , as a result of crossbreeding with the third gene, the second gene may be exchanged with detailed surgical operations of the third gene based on a specific point 200 . In addition, as a result of crossbreeding with the second gene, the third gene may be exchanged with detailed surgical operations of the second gene based on a specific point 200 .

A specific point 200 for intergenic crossing may be arbitrarily selected. In this case, as the specific point 200, one point may be used, or a plurality of points may be used. In addition, the specific point 200 may be equally selected for each gene or may be selected differently.

A total of 2n new genes can be generated by repeating the above gene hybridization process n times. In one embodiment, the computer may determine the number of gene crossover processes according to the degree to which genes are to be evolved (changed). For example, the computer may repeat the gene crossing process in consideration of the number of initially generated genes, and if the number of initially generated genes (parent genes) is n, the above gene crossing process is repeated n/2 times. can be performed to generate a total of n new genes (child genes).

97 is a diagram illustrating a process of mutating a gene as an example of applying a genetic algorithm according to an embodiment of the present invention.

In one embodiment, the computer generates n genes as shown in FIG. 95 and performs virtual surgery on each of the n genes to calculate the fitness of each gene (eg, fitness using Equation 1). have. The computer may select a gene having the greatest fitness from among the n genes or select a gene having a fitness greater than or equal to a specific reference value based on the fitness calculated for each of the n genes. In this case, a method of selecting a gene may vary depending on which conditions are set.

For example, if one gene (eg, second gene) having a fitness value (eg, the largest fitness value) matching the condition is selected from among n genes, the computer assigns the selected gene (eg, the second gene) to the selected gene (eg, the second gene). A mutation genetic algorithm can be applied to

Referring to FIG. 97 , the computer selects a specific point 300 for mutating the second gene, and converts the gene (eg, cutting detail operation) corresponding to the selected specific point 300 to another gene (eg, hemostasis detail) surgical operation).

A specific point 300 for mutating a gene may be arbitrarily selected. In one embodiment, the specific point 300 may be selected using a probability. For example, a probability value P (eg, P is a small value such as 0.001) is set, a random value is generated using a random function at the location of a gene (ie, each detailed surgical operation within a gene), and then compared with the probability value P, the condition ( Example: A gene (ie, detailed surgical operation) that fits the probability value less than P) can be selected as a specific point 300 . Also, as the specific point 300 , one point may be used, or a plurality of points may be used.

Referring back to FIG. 94, the computer may generate a new gene (child gene) by applying a genetic algorithm from the initial genes (parent gene) generated in step S100, and derive an optimal surgical procedure based on the new gene You can (S130).

In one embodiment, the computer may generate at least one new gene (child gene) by applying a genetic algorithm such as crossbreeding or mutation to the initially generated genes (parent gene) as described above. In this case, the computer may repeatedly perform steps S110 to S120 for a new gene (child gene). In this case, the number of repetitions may be predetermined.

For example, a computer can calculate a fitness by performing virtual surgery on a new gene (child gene). In addition, the computer may determine whether the fitness of a new gene (child gene) meets a preset condition, select a new gene (child gene) that meets the condition, and apply a genetic algorithm such as crossbreeding or mutation. By applying this genetic algorithm to a new gene (child gene), it is possible to generate a new child gene again. That is, the computer repeatedly generates offspring genes from parent genes based on the fitness result to evaluate whether it is the optimal surgery, and finally obtains a gene including the optimal surgical procedure from among the generated offspring genes. have. For example, it is possible to select a gene having the highest fitness among the offspring genes and derive it through an optimized surgical procedure.

In this case, whenever the genetic process such as steps S110 to S120 is repeatedly performed, only genes with high fitness are selected and continuously evolved, so that the child genes have higher fitness values than the parent genes. That is, the child genes may be composed of detailed surgical operations made up of an improved surgical procedure than that of the parent genes. Accordingly, the computer may select the gene having the highest fitness from the finally generated offspring genes, and obtain optimized surgical cue sheet data from the selected gene. In this case, the optimized surgical cue sheet data includes detailed surgical operation information composed of an optimized surgical procedure in terms of operation time, operation of surgical tools, number of surgical tools used, and surgical prognosis.

According to the embodiment of the present invention as described above, by applying the genetic algorithm, it is possible to derive a more improved surgical procedure compared to the surgical process performed in the actual surgical procedure.

In addition, according to an embodiment of the present invention, since surgery cue sheet data including an optimized surgical procedure is derived through a genetic algorithm, it can be provided to actual medical staff. In addition, actual medical staff can utilize the optimized surgical cue sheet data during actual surgery, and through this, more accurate surgical motions and surgical procedures can be performed.

In addition, according to an embodiment of the present invention, since surgery cue sheet data including an optimized surgical procedure is derived through a genetic algorithm, it can be utilized as learning data in a learning process such as deep learning.

98 is a schematic diagram of a system capable of performing robotic surgery according to an embodiment of the present invention.

Referring to FIG. 98 , the robotic surgery system includes a medical imaging device 10 , a server 100 , and a control unit 30 , a display 32 , and a surgical robot 34 provided in the operating room. According to an embodiment, the medical imaging device 10 may be omitted from the robotic surgery system according to the disclosed embodiment.

In one embodiment, the surgical robot 34 includes a photographing device 36 and a surgical tool 38 .

In one embodiment, the robotic surgery is performed by the user using the control unit 30 to control the surgical robot (34). In one embodiment, the robotic surgery may be performed automatically by the control unit 30 without the user's control.

The server 100 is a computing device including at least one processor and a communication unit.

The control unit 30 includes a computing device including at least one processor and a communication unit. In one embodiment, the controller 30 includes hardware and software interfaces for controlling the surgical robot 34 .

The photographing device 36 includes at least one image sensor. That is, the imaging device 36 includes at least one camera device, and is used to photograph an object, that is, a surgical site. In an embodiment, the imaging device 36 includes at least one camera coupled to a surgical arm of the surgical robot 34 .

In an embodiment, the image captured by the photographing device 36 is displayed on the display 340 .

In one embodiment, the surgical robot 34 includes one or more surgical tools 38 capable of performing cutting, clipping, fixing, grabbing operations of the surgical site, and the like. The surgical tool 38 is used in combination with the surgical arm of the surgical robot 34 .

The controller 30 receives information necessary for surgery from the server 100 or generates information necessary for surgery and provides it to the user. For example, the controller 30 displays generated or received information necessary for surgery on the display 32 .

For example, the user performs robotic surgery by controlling the movement of the surgical robot 34 by manipulating the controller 30 while viewing the display 32 .

The server 100 generates information necessary for robotic surgery by using the medical image data of the object photographed in advance from the medical imaging device 10 , and provides the generated information to the controller 30 .

The controller 30 provides the information received from the server 100 to the user by displaying it on the display 32 , or controls the surgical robot 34 using the information received from the server 100 .

In one embodiment, the means that can be used in the medical imaging equipment 10 is not limited, for example, CT, X-Ray, PET, MRI, etc., various other medical image acquisition means may be used.

When performing robotic surgery as described above, it is possible to acquire data including surgical images captured in the surgical process or various surgical information in the control process of the surgical robot. In one embodiment, in the present invention, the above-described surgical cue sheet data may be configured based on the surgical information (ie, surgical image) obtained in such a robotic surgery process, and a gene may be generated therefrom. In addition, in the present invention, the optimized surgical procedure as described in the embodiment of FIGS. 94 to 97 , that is, the optimized surgical cue sheet data may be derived and applied to the robotic surgery procedure of FIG. 98 . In this case, since the surgical robot can perform the operation according to the optimized surgical cue sheet data, the operation can be performed more accurately and effectively.

99 is a diagram schematically showing the configuration of an apparatus 400 for performing a surgical optimization method according to an embodiment of the present invention.

Referring to FIG. 99 , the processor 410 includes one or more cores (core, not shown) and a graphic processing unit (not shown) and/or a connection path (eg, a bus) for transmitting and receiving signals with other components. ) may be included.

The processor 410 according to an embodiment executes one or more instructions stored in the memory 420 , thereby performing the method of providing the surgical image described in relation to FIGS. 94 to 97 .

For example, the processor 410 generates a plurality of genes based on detailed surgical operations by executing one or more instructions stored in the memory 420, and performs virtual surgery on each of the plurality of genes to determine whether the surgery is optimized. It is possible to evaluate, apply a genetic algorithm by selecting at least one gene among a plurality of genes based on the evaluation result, create a new gene by applying the genetic algorithm, and derive an optimal surgical procedure based on the new gene. .

Meanwhile, the processor 410 includes a random access memory (RAM) and a read-only memory (ROM) that temporarily and/or permanently store a signal (or data) processed inside the processor 410 . , not shown) may be further included. In addition, the processor 410 may be implemented in the form of a system on chip (SoC) including at least one of a graphic processing unit, a RAM, and a ROM.

The memory 420 may store programs (one or more instructions) for processing and controlling the processor 410 . Programs stored in the memory 420 may be divided into a plurality of modules according to functions.

The surgery optimization method according to an embodiment of the present invention described above may be implemented as a program (or application) to be executed in combination with a computer, which is hardware, and stored in a medium.

With reference to FIGS. 100 to 104, a method for providing an optimized surgical tool according to an embodiment of the present invention will be described in detail.

100 is a schematic diagram of a system capable of performing robotic surgery according to an embodiment of the present invention.

Referring to FIG. 100 , the robotic surgery system includes a medical imaging device 10 , a server 100 , and a controller 30 , a display 32 , and a surgical robot 34 provided in the operating room. According to an embodiment, the medical imaging device 10 may be omitted from the robotic surgery system according to the disclosed embodiment.

In one embodiment, the surgical robot 34 includes a photographing device 36 and a surgical tool 38 .

In one embodiment, the robotic surgery is performed by the user using the control unit 30 to control the surgical robot (34). In one embodiment, the robotic surgery may be performed automatically by the control unit 30 without the user's control.

The server 100 is a computing device including at least one processor and a communication unit.

The control unit 30 includes a computing device including at least one processor and a communication unit. In one embodiment, the controller 30 includes hardware and software interfaces for controlling the surgical robot 34 .

The photographing device 36 includes at least one image sensor. That is, the imaging device 36 includes at least one camera device, and is used to photograph an object, that is, a surgical site. In an embodiment, the imaging device 36 includes at least one camera coupled to a surgical arm of the surgical robot 34 .

In an embodiment, the image captured by the photographing device 36 is displayed on the display 340 .

In one embodiment, the surgical robot 34 includes one or more surgical tools 38 capable of performing cutting, clipping, fixing, grabbing operations of the surgical site, and the like. The surgical tool 38 is used in combination with the surgical arm of the surgical robot 34 .

The controller 30 receives information necessary for surgery from the server 100 or generates information necessary for surgery and provides it to the user. For example, the controller 30 displays generated or received information necessary for surgery on the display 32 .

For example, the user performs robotic surgery by controlling the movement of the surgical robot 34 by manipulating the controller 30 while viewing the display 32 .

The server 100 generates information necessary for robotic surgery by using the medical image data of the object photographed in advance from the medical imaging device 10 , and provides the generated information to the controller 30 .

The controller 30 provides the information received from the server 100 to the user by displaying it on the display 32 , or controls the surgical robot 34 using the information received from the server 100 .

In one embodiment, the means that can be used in the medical imaging equipment 10 is not limited, for example, CT, X-Ray, PET, MRI, etc., various other medical image acquisition means may be used.

In the case of minimally invasive surgery, such as robotic surgery, laparoscopic surgery, or surgery using an endoscope, surgery is performed by inserting a surgical tool and a camera into the body of a patient (ie, a subject for surgery), that is, a surgical site. In this case, since organs, blood vessels, etc. are arranged inside the patient's body, and the internal space of the body is narrow, it is not easy to perform a surgical operation by inserting a surgical tool. Therefore, in the present invention, it is possible to minimize the occurrence of restrictions in the surgical process due to the arrangement state of organs and blood vessels present inside the patient's body, the length and structure of surgical tools, etc., and to perform the most convenient surgical operation for the medical staff. We want to provide a way to make it happen. It will be described in detail below.

101 is a flowchart schematically illustrating a method for providing an optimized surgical tool according to an embodiment of the present invention.

Each of the steps shown in FIG. 101 may be performed in time series by the server 100 or the controller 30 shown in FIG. 100 . Alternatively, it may be performed in a computing device provided separately from this. Hereinafter, each step is described as being performed by a computer for convenience of description, but the subject performing each step is not limited to a specific device, and all or part of it is performed by the server 20 or the control unit 30 or , may be performed in a separately provided computing device.

Referring to FIG. 101 , the method for providing an optimized surgical tool performed by a computer according to an embodiment of the present invention includes the steps of obtaining a virtual body model generated in accordance with the body state of a subject for surgery (S100), the virtual body model A step of simulating a surgical operation using a surgical tool within (S110), and deriving a configuration of a surgical tool suitable for applying the surgical operation performed by the surgical tool in the internal space of the patient's body based on the simulation result It may include step S120. Hereinafter, detailed description of each step is described.

The computer may acquire a virtual body model generated in accordance with the physical condition of the surgery subject (S100).

The virtual body model may be 3D modeling data generated based on medical image data (eg, medical images photographed through CT, PET, MRI, etc.) obtained by photographing the inside of the patient's body in advance. For example, it is modeled in conformity with the body of the subject to be operated on, and may be calibrated to the same state as the actual surgical state.

Medical staff can perform rehearsals or simulations using a virtual body model that is implemented in the same way as the physical state of the subject to be operated on, and through this, they can experience the same state as during the actual operation. In this case, the medical staff can freely perform surgical operations in the internal space of the patient's body through the virtual body model. For example, the virtual surgery can be performed according to the surgical operation pattern of medical staff without restrictions according to the internal characteristics of the patient's body (eg, organ arrangement, blood vessel condition, etc.) or the characteristics of surgical tools. And after performing this virtual surgery, based on the results, it is possible to derive the optimal actual surgical procedure by considering the characteristics of the patient's body or the characteristics of the surgical tool. In addition, when performing a virtual surgery using a virtual body model, data including a rehearsal or simulation action for the virtual body model can be obtained. For example, image data obtained by performing a virtual operation (ie, rehearsal or simulation) on the virtual body model may be obtained, or data recording a surgical operation performed on the virtual body model may be obtained.

The computer may simulate a surgical operation using a surgical tool in the virtual body model (S110).

The surgical tool may be configured to include an operation unit that directly performs a surgical operation on the surgical site, and an arm portion connected thereto and operated. For example, the operation unit is a part that can access the surgical site and perform surgical operations such as grabbing, cutting, moving, or suturing a target object, and may be composed of various instruments depending on the purpose of the surgical operation. . The arm part may be connected to the operating unit and operated according to the movement of the operating unit, or may operate to control the movement of the operating unit.

In the case of minimally invasive surgery such as the robot surgery or laparoscopic surgery of FIG. 100, only the operating part of the surgical tool is photographed by the camera inserted into the body of the subject to be operated and displayed on the screen. Accordingly, the operating state of the arm portion of the surgical tool is not provided through the screen.

In one embodiment, the computer may simulate a surgical operation using only the operation part of the surgical tool in the virtual body model. That is, the computer creates changes in the space inside the virtual body model among the composition of the surgical tools (e.g., causes physical or chemical changes in organs, blood vessels, tissues, etc. in the body, or inserts clips, gauze, saline, etc. Only the operation part that provides external objects) can be implemented and simulated inside the virtual body model. Therefore, in such a simulation, the virtual body model is performed without considering the characteristics of the arm part of the surgical tool, whether the internal organs of the body are affected by the operation of the arm part, or the entry position on the patient's body surface. By performing virtual surgery using only the operation part of the surgical tool within the surgery, the easiest and most familiar surgical operation pattern can be applied.

Here, the simulation (virtual surgery) for the virtual body model may be performed by a medical staff or may be performed by the computer itself. When performed by a medical staff, the computer receives a controller operation input by the medical personnel. In addition, when the computer performs its own simulation, the computer can perform virtual surgery on a specific patient based on the result of learning the existing surgical data.

The computer may derive the configuration of a surgical tool suitable for applying the surgical operation performed by the surgical tool in the internal space of the body of the surgical subject based on the simulation result (S120).

Specifically, the computer may acquire movement information of the surgical tool based on the simulation result through the virtual body model. The motion information may be information indicating a change in the position of a surgical tool that is generated as a surgical operation is performed, for example, may be a set of coordinate values indicating the position of each point of the surgical tool on the virtual body model coordinates. And, the computer may derive a configuration of a surgical tool suitable for applying movement information of the surgical tool in the internal space of the body of the surgical subject. That is, the computer can derive the optimal type of operation part and the structure of the arm part in applying the motion information obtained through the virtual body model to the inside of the body of the surgery subject.

In an embodiment, the computer may obtain motion information of the operating unit by simulating a surgical operation using only the operating part of the surgical tool through the virtual body model. The computer may determine the type of the operating unit optimized to apply a surgical operation to the operation target region of the operation target based on the motion information of the operating unit. For example, the computer may analyze the motion information of the operating unit to determine movement pattern information for a surgical operation performed by the operating unit. When the surgical operation performed by the operation unit is a cutting operation, the computer may recognize the cutting operation from the movement pattern information and derive the type (type) of the operation unit suitable for cutting. In addition, even if the same surgical operation is performed, the movement of the operation unit may appear differently depending on the type of operation or the operation target site. In this case, the computer may analyze the movement pattern information of the operation unit to derive the type of the operation unit most suitable for the operation or the operation site.

In addition, the computer may acquire the body internal information and body surface information of the surgery subject, and determine the structure of the cancer part based on the acquired information. The body internal information may include information on the arrangement of organs located in the internal space of the body of the surgery subject, and the body surface information may include information on the shape of the body surface of the surgery target.

The computer may acquire body internal information and body surface information in various ways. In an embodiment, body internal information or body surface information may be obtained from a virtual body model generated by applying medical image data about a patient to an undulation forming algorithm. The undulation formation algorithm is an algorithm that generates 3D modeling data in a steady state into 3D modeling data in a undulating state.

Unlike general open surgery, when performing laparoscopic surgery or robotic surgery, gas (ie, relief agent carbon dioxide) is injected into the body to form a space in which the surgical tool moves inside the body to form the patient's body in a undulating state. do. That is, when a patient is a target for laparoscopic surgery or robotic surgery, a process of modeling in a undulating state is required for the medical staff to simulate through the same virtual body model as during actual surgery. The computer acquires body surface information from the patient's body surface in the virtual body model formed in the undulating state by applying the undulations formation algorithm, and obtains the internal body information by extracting the organ arrangement information inside the body.

In one embodiment, in determining the structure of the arm part, the computer calculates the motion performance range of the arm part from the body internal information and body surface information of the subject to be operated on, and operates with the arm part based on the motion performance range of the arm part It is possible to derive a disposition relationship between parts. The computer may determine at least one of the length of the arm part, the presence or absence of the joint part, and motion information of the joint part according to the arrangement relationship with the operation part. In this case, the operation performance range of the arm refers to a range in which the operation of the arm part is possible according to the movement of the operation part based on the body internal information and body surface information of the subject to be operated on.

For example, since the computer can determine the organ arrangement state from the information inside the body of the subject to be operated on, when the movement of the arm part occurs according to the movement of the operation part, it can be determined whether the organ is affected by this. Accordingly, the computer can calculate the range of motion of the arm that can perform the motion without affecting the organs. In addition, the computer may derive a disposition relationship between the arm part and the operating part based on a positional relationship with the internal organ arrangement or body surface shape of the subject to be operated within the operation performance range of the arm part. That is, the computer may determine the optimized structure of the arm part based on a disposition relationship with the operating part within the operation performance range of the arm part. For example, the computer may calculate an arrangement relationship between the operating unit and the arm, such as an angle, a degree of inclination, a degree of bending, and the like. The computer can determine whether the arm part needs an articulation based on this arrangement relationship. The joint part may be a part that connects the operating part and the arm part, or connects the arm and the arm when the arm part includes a plurality of arms. When the arm part includes the joint part, the computer may determine the degree of movement such as the number of joints, the degree of freedom of the joint, the rotation angle, and the degree of bending.

In deriving the configuration of the surgical tool in an embodiment of the present invention, the computer can determine the configuration of the surgical tool in consideration of obtaining the optimal entry position on the body surface of the surgical subject into which the surgical tool is inserted, and considering the optimal entry position. have. In this case, the computer may use a predetermined entry position, or may obtain an optimal entry position by performing the method of FIG. 103, which will be described later. In one embodiment, the computer may derive the structure of the arm that can be inserted without affecting the internal organs or body surface shape when inserting the surgical tool to the optimal entry position. For example, the length of the arm part, the presence or absence of the joint part, motion information of the joint part, and the like may be derived.

102 is a flowchart schematically illustrating a method for providing an optimized surgical tool according to another embodiment of the present invention.

Each of the steps shown in FIG. 102 may be performed in time series by the server 100 or the controller 30 shown in FIG. 100 . Alternatively, it may be performed in a computing device provided separately from this. Hereinafter, each step is described as being performed by a computer for convenience of description, but the subject performing each step is not limited to a specific device, and all or part of it is performed by the server 20 or the control unit 30 or , may be performed in a separately provided computing device.

Referring to FIG. 102 , the method for providing an optimized surgical tool performed by a computer according to an embodiment of the present invention includes: acquiring a virtual body model generated in accordance with the physical condition of a subject for surgery (S200), the virtual body model A step of simulating a surgical motion corresponding to the actual surgical motion of the surgical subject within (S210), and deriving a surgical tool or a surgical robot suitable for applying the surgical motion in the internal space of the subject's body based on the simulation result (S220) may be included. Hereinafter, detailed description of each step is described.

The computer may acquire a virtual body model generated in accordance with the physical condition of the surgical subject (S200). This may be performed in the same manner as in step S100 of FIG. 101, and thus a detailed description thereof will be omitted.

The computer may simulate a surgical operation corresponding to the actual surgical operation of the surgical subject in the virtual body model (S210). This may be performed in the same manner as in step S110 of FIG. 101, and thus a detailed description thereof will be omitted.

The computer may derive a surgical tool or a surgical robot suitable for applying a surgical operation in the internal space of the patient's body based on the simulation result (S220).

In one embodiment, the computer may obtain motion information of the surgical tool from the simulation result, and analyze the surgical operation from the motion information. The computer may determine the type of a specific surgical robot optimized for the surgical target or the type of surgical tool included in the specific surgical robot based on the analyzed surgical motion.

That is, there may be various types of surgical robots depending on the type of surgery or the physical characteristics of the surgery target, and each company may have different types of surgical robots. Therefore, in the present invention, it is possible to obtain the movement information of the surgical tool through the virtual body model, derive the type of the most suitable surgical robot for the corresponding surgical operation, and recommend it to the medical staff. In addition, the movement characteristics of surgical tools (ie, the moving part and the arm part) may be different for each surgical robot, and there may be differences in the types of surgical tools possessed by each company's surgical robots. Therefore, in the present invention, it is possible to determine a specific surgical robot that is optimal for implementing the movement of the surgical tool obtained through the virtual body model. In addition, since a single surgical robot may include several surgical tools that perform the same surgical operation, the computer selects a specific optimal surgical tool to implement the movement of a surgical tool obtained through a virtual body model among various surgical tools in a specific surgical robot. The surgical instrument can be determined.

In deriving a surgical tool or a surgical robot in an embodiment of the present invention, the computer acquires an optimal entry position on the body surface of a subject for surgery into which the surgical tool is inserted, and selects the surgical tool or surgical robot in consideration of the optimal entry position. can decide In this case, the computer may use a predetermined entry position, or may obtain an optimal entry position by performing the method of FIG. 103, which will be described later. In one embodiment, the computer can derive a surgical tool or a surgical robot including the surgical tool that can be inserted without affecting the internal organs or body surface shape when inserting the surgical tool into the optimal entry position. have.

In addition, in one embodiment of the present invention, the computer may record and store the movement of the surgical tool in real time during surgery. Accordingly, as described above, the computer may determine at least one optimal surgical tool based on the simulation result, and may select one surgical tool from the user among the determined surgical tools. In this case, the computer may re-simulate the surgical procedure using the surgical tool selected by the user. Through this, an optimized actual surgical procedure (ie, actual surgical operation) can be derived.

103 is a flowchart schematically illustrating a method for providing an optimal entry position of a surgical tool according to an embodiment of the present invention.

Each of the steps shown in FIG. 103 may be performed in time series by the server 100 or the controller 30 shown in FIG. 100 . Alternatively, it may be performed in a computing device provided separately from this. Hereinafter, each step is described as being performed by a computer for convenience of description, but the subject performing each step is not limited to a specific device, and all or part of it is performed by the server 20 or the control unit 30 or , may be performed in a separately provided computing device.

Referring to FIG. 103 , the method for providing an optimal entry position of a surgical tool performed by a computer according to an embodiment of the present invention includes the steps of obtaining a virtual body model generated in accordance with the physical condition of a subject to be operated (S300), the virtual The operation performed by the operation unit in the internal space of the patient's body based on the simulation results and the step of simulating using the operating unit of the surgical tool for applying a surgical operation to the surgical operation target site in the body model in the body model (S310) It may include a step (S320) of calculating the optimal entry position on the body surface of the surgery subject to apply the motion. Hereinafter, detailed description of each step is described.

The computer may acquire a virtual body model generated in accordance with the physical state of the surgical subject (S300). This may be performed in the same manner as in step S100 of FIG. 101, and thus a detailed description thereof will be omitted.

The computer may simulate in the virtual body model using the operation part of the surgical tool that applies a surgical operation to the surgical target site of the surgical subject (S310). This may be performed in the same manner as in step S110 of FIG. 101, and thus a redundant description will be omitted.

In an embodiment, the computer may simulate a surgical operation using only the operating unit for the operation target in the virtual body model, without considering the configuration of the arm part that moves according to the surgical operation of the operating unit. The computer may acquire motion information of the operation unit from the simulation result. In this case, the motion information may be information indicating a change in the position of a surgical tool that is generated as a surgical operation is performed as described above, for example, a set of coordinate values indicating the position of each point of the surgical tool on the virtual body model coordinates. can

Therefore, in such a simulation, the virtual body model is performed without considering the characteristics of the arm part of the surgical tool, whether the internal organs of the body are affected by the operation of the arm part, or the entry position on the patient's body surface. By performing virtual surgery using only the operation part of the surgical tool within the surgery, the easiest and most familiar surgical operation pattern can be applied.

The computer may calculate an optimal entry position on the body surface of the surgery subject to apply the surgical operation performed by the operating unit in the internal space of the subject's body based on the simulation result (S320).

In an embodiment, the computer may extract an accessible range from the body surface of the surgery subject to which motion information of the operation unit may be applied in the internal space of the subject's body. In addition, the computer may calculate the optimal entry position from the possible entry range by reflecting the operation performance range of the arm portion in which movement occurs according to the surgical operation of the operation unit.

For example, the computer may extract an accessible range to which the motion information of the moving part can be applied from the entire body surface of the surgery subject. As an example, the accessible range is i) an area that the operating part of the surgical tool cannot reach when performing a surgical operation, or ii) a case where the operating part of the surgical tool that has reached a specific point cannot perform a specific surgical operation. It may be a location on a body surface excluding an area. As another example, the accessible range may be calculated based on the function of the surgical tool. For example, there are a gripping function, a cutting function, a supporting function (topography function), etc. depending on the type of surgical tool, and different surgical tools are used according to each function. Alternatively, there may be a case where one surgical tool (eg, Harmonic ACE) performs all functions. Accordingly, the computer may extract an accessible range to which the motion information of the operation unit according to each function can be applied based on the function of the surgical tool.

In addition, the computer may derive an optimal entry position by calculating a specific area or a specific point that satisfies the motion performance range of the arm within the extracted body surface entry range. That is, when the operation unit moves, the arm part connected thereto also moves. At this time, since the computer obtained the simulation result in the state that only the moving part was expressed through the virtual body model, the optimal entry position should be calculated by additionally considering the movement of the arm part.

For example, the computer can calculate the cancer part without affecting the organ or body surface based on the body internal information (ie, body internal organ placement state) and body surface information (ie body surface shape state) of the subject of surgery. It is possible to calculate the range of motion performance that can be operated at the maximum. The computer can determine the optimal entry position that does not affect the movement of the surgical tool in the internal space of the body by reflecting the range of motion within the accessible range of the body surface.

In addition, in an embodiment of the present invention, the computer can derive a configuration of a surgical tool suitable for performing a surgical operation by inserting the surgical tool into the optimal entry position. Since the process of deriving an optimal surgical tool has been described in detail through the embodiments of FIGS. 101 and 102 , the description will be omitted in this embodiment.

Meanwhile, in the above-described embodiments, the number of surgical tools has been described without considering the number of surgical tools for convenience of explanation, but the present invention is not limited thereto. The same process can be applied for

As an example, when using a surgical robot or tool A, tool B, or tool C during laparoscopic surgery, the computer operates in an area that cannot be reached when the operation part of tool A performs a surgical operation (that is, it is operated by the length limitation of the surgical tool) The area of the body surface where a point that cannot be reached when performing additional surgical operations occurs) may be excluded from the possible entry range. In addition, the computer may exclude a body surface area that collides with body organs or tissues in the process where the tool A enters and performs a surgical operation from the access range. In addition, if the computer fails to implement a surgical operation required at a specific position after the surgical tool enters at each body surface point within the accessible range, the computer may exclude the corresponding body surface point from the accessible range. Through this, the computer can calculate the possible access range for tool A. The computer may calculate the optimal entry position of each surgical tool by individually performing the process of calculating the accessible range for each surgical tool (eg, tool B and tool C). In addition, as described above, the computer may calculate the optimal entry position to which the function of each surgical tool can be applied by individually performing the process of calculating the possible access range for each function based on the function of the surgical tool.

As another example, if several surgical tools need to enter one optimal entry position, the computer may extract the optimal entry range for each surgical tool, and then determine the overlapping range of the plurality of optimal entry ranges as the optimal entry location. have. For example, when tool A is changed to tool D in the course of performing surgery, the computer may calculate an overlapping area between the accessible range for tool A and the accessible range for tool D as an optimal entry position candidate area. . Since the position where the surgical tool can enter is limited to a certain number (for example, 3), when the tool A is changed to the tool D, the same entry position has to be used, so the computer is A position that satisfies all the possible entry ranges of tool D can be determined as the final optimal entry position.

As another example, when the same surgical tool is used multiple times in the virtual body model, all surgical operations at one surgical tool entry position when the surgical operation of the surgical tool (that is, the range of motion) is wide. can be difficult to perform, the computer can divide the range in which the surgical tool is used (ie, the range of motion) into several groups reachable from multiple entry points on the body surface. For example, when performing laparoscopic surgery or robotic surgery by creating three entry points on the body surface, the computer divides the range of motion of the surgical tool into three or less groups. At this time, the computer divides the range of motion based on whether it is reachable from a plurality of accessible ranges selected by other surgical tools. In addition, a specific surgical tool having a wide range of motion (ie, the first surgical tool) is used simultaneously with another surgical tool (ie, the second surgical tool) and another surgical tool (ie, the second surgical tool) is required to enter. When the optimal entry position is determined, the computer determines the range of motion of the first surgical tool when used together with the second surgical tool at the optimal entry position of the first surgical tool (that is, the second surgical tool enters). It can be set to a range that cannot be accessed through the keyhole). In addition, when the first surgical tool is continuously used despite the change of the other surgical tools, the computer enters the same entry position in consideration of the user's convenience and the time required for surgery, and the operation must be performed. It can be set as a group.

As another example, in the process of calculating the entry positions of one or more surgical tools, the computer may exclude the camera entry range and the assist tool entry range from the accessible range of the patient's body surface. For example, the camera enters through the area around the navel to photograph the entire interior of the patient's abdomen through movement. To this end, the computer may perform a process of reducing the accessible range after excluding the area around the navel from the initially set accessible range.

In one embodiment, the computer may perform a process of calculating one or more optimal entry positions when using a plurality of surgical tools using the Monte Carlo method.

According to the embodiments of the present invention as described above, the medical staff determines the optimal entry position of the surgical tool by reflecting the results of the surgical simulation performed without considering the entry position of the surgical tool and the long-term jamming of the arm part of the surgical tool. , so that the medical staff can perform the most comfortable surgical operation.

In addition, according to embodiments of the present invention, by using the optimal entry position optimized for the patient's physical condition without using a general surgical tool entry position, the medical staff can perform actual surgery according to the patient's organ arrangement characteristics or the length of the surgical tool. In the process, it is possible to prevent a restriction in performing a specific operation.

In addition, according to embodiments of the present invention, when all surgical tools used in surgery are performed using a specific number of surgical tool entry positions, several surgical tool entry positions optimized for all surgical tools can be determined. have. In addition, it is possible to accurately set the entry position into which each surgical tool should enter from among a specific number of optimal entry positions.

In addition, according to embodiments of the present invention, when a plurality of surgical robots or a plurality of surgical tools for performing the same action in a specific surgical robot are available, the surgery most suitable for the patient's physical condition and the surgical operation of the medical staff By suggesting a robot or surgical tool, the medical staff can perform an efficient and fast operation.

In addition, according to embodiments of the present invention, when the surgical robot performs an operation by itself, it is possible to perform an optimized operation by calculating the optimal entry position of each surgical tool.

In addition, according to embodiments of the present invention, since the medical staff determines the surgical tool having the optimal structure by reflecting the results of the surgical simulation performed without considering the entry position of the surgical tool and the long-term jamming of the arm part of the surgical tool, The medical staff can conveniently operate the surgical tool according to their surgical operation pattern. Accordingly, the entire surgical procedure can be effectively performed, and surgical errors can be reduced.

In addition, according to embodiments of the present invention, as the operation is performed using the optimized surgical tool, the medical staff is limited in performing a specific surgical operation in the actual surgical procedure due to the characteristics of the patient's organ arrangement or surgical tool. can be prevented from doing

104 is a diagram schematically showing the configuration of an apparatus 200 for performing an optimized method for providing a surgical tool or a method for providing an optimal entry position of a surgical tool according to an embodiment of the present invention.

Referring to FIG. 104 , the processor 210 includes one or more cores (core, not shown) and a graphic processing unit (not shown) and/or a connection path (eg, a bus) for transmitting and receiving signals with other components. ) may be included.

The processor 210 according to an embodiment executes one or more instructions stored in the memory 220, thereby performing the method of providing the optimized surgical tool or the optimal entry position of the surgical tool described in relation to FIGS. 101 to 103. do.

As an example, the processor 210 obtains a virtual body model generated in accordance with the body state of the surgery subject by executing one or more instructions stored in the memory 220, and performs a surgical operation using a surgical tool within the virtual body model. , and based on the simulation result, it is possible to derive the configuration of the surgical tool suitable for applying the surgical operation performed by the surgical tool in the internal space of the patient's body.

As another example, the processor 210 executes one or more instructions stored in the memory 220 to obtain a virtual body model generated in accordance with the body state of the surgery subject, and the actual surgical operation of the surgery subject in the virtual body model. and a surgical operation corresponding to , and based on the simulation result, it is possible to derive a surgical tool or a surgical robot suitable for applying the surgical operation in the internal space of the patient's body.

As another example, the processor 210 executes one or more instructions stored in the memory 220 to obtain a virtual body model generated in accordance with the body state of the surgery subject, and within the virtual body model, the surgical target of the surgery subject. Simulation using the operation part of the surgical tool that applies a surgical operation to the site, and the surgical operation performed by the operation part in the internal space of the patient's body based on the simulation result. The optimal entry position on the body surface can be calculated.

On the other hand, the processor 210 is a RAM (Random Access Memory, not shown) and ROM (Read-Only Memory: ROM) for temporarily and / or permanently storing signals (or data) processed inside the processor 210 . , not shown) may be further included. Also, the processor 210 may be implemented in the form of a system on chip (SoC) including at least one of a graphic processing unit, a RAM, and a ROM.

The memory 220 may store programs (one or more instructions) for processing and controlling the processor 210 . Programs stored in the memory 220 may be divided into a plurality of modules according to functions.

The method for providing the optimal surgical tool or the optimal entry position of the surgical tool according to the embodiment of the present invention described above is implemented as a program (or application) to be executed in combination with a computer, which is hardware, and stored in a medium. can

The above-described program is C, C++, JAVA, machine language, etc. that a processor (CPU) of the computer can read through a device interface of the computer in order for the computer to read the program and execute the methods implemented as a program It may include code (Code) coded in the computer language of Such code may include functional code related to functions defining functions necessary for executing the methods, etc. can do. In addition, the code may further include additional information necessary for the processor of the computer to execute the functions or code related to memory reference for which location (address address) in the internal or external memory of the computer should be referenced. have. In addition, when the processor of the computer needs to communicate with any other computer or server located remotely in order to execute the functions, the code uses the communication module of the computer to determine how to communicate with any other computer or server remotely. It may further include a communication-related code for whether to communicate and what information or media to transmit and receive during communication.

The storage medium is not a medium that stores data for a short moment, such as a register, a cache, a memory, etc., but a medium that stores data semi-permanently and can be read by a device. Specifically, examples of the storage medium include, but are not limited to, ROM, RAM, CD-ROM, magnetic tape, floppy disk, and optical data storage device. That is, the program may be stored in various recording media on various servers accessible by the computer or in various recording media on the computer of the user. In addition, the medium may be distributed in a computer system connected by a network, and a computer readable code may be stored in a distributed manner.

In the above, various embodiments of the present invention that can be applied in the medical field have been described in detail with reference to FIGS. 1 to 104 . Hereinafter, a solution model capable of recognizing and solving a medical problem using the above-described embodiments of the present invention will be described with reference to FIG. 105 as an example.

105 shows an embodiment for explaining a medical solution model for recognizing and solving a medical problem according to the present invention.

Referring to FIG. 105 , a medical solution model for recognizing and solving a medical problem according to the present invention (hereinafter referred to as a medical solution model) may have a hierarchical structure.

As an embodiment, the lowest layer may be a surgical element recognition layer that recognizes surgical elements, which are the minimum units to be recognized medically in a medical surgery process. The middle layer may be a Surgical Module Layer that medically grasps meaning through each surgical element and determines a condition for making a medical diagnosis. The top layer may be a Surgical Solution Layer that recognizes medical problems that may occur in the entire medical surgical process and provides solutions for each problem.

The surgical element recognition layer may recognize surgical elements based on various surgical images captured in a medical surgical procedure and various surgical-related tools used in a medical surgical process. For example, the surgical element is a surgical phase (Phase), body part (eg, organ; Organ), event (Event), operation time (Time), surgical instrument (Instrument), camera (Camera), operation (Acton) , and other elements.

In an embodiment, the surgical element recognition layer may recognize at least one surgical element based on a surgical image obtained in a medical surgery process. In this case, not only the surgical elements recognized in the surgical image itself, but also the surgical elements may be recognized using the result of learning the surgical image. By utilizing the above-described embodiments of the present invention, it is possible to effectively recognize the surgical elements, which are the minimum units to be recognized medically in a medical surgery process.

The surgical element recognition layer may individually recognize the surgical elements. For example, when an organ is recognized from a surgical image, only a corresponding organ can be recognized, and when a surgical tool is recognized from a surgical image, only the corresponding surgical tool can be recognized.

In addition, the surgical element recognition layer may use another surgical element to recognize one surgical element. That is, the surgical element recognition layer may set a primitive level relation indicating a relationship between each surgical element, and recognize each surgical element using the primitive level relationship. For example, the primitive level relationship lists the additional surgical elements required to recognize the corresponding surgical element, and the relationship between these surgical elements (eg, state change, position change, shape change, color change, arrangement relationship, etc.) It may include specified information and the like. For example, when recognizing an event (for example, an event on whether or not bleeding occurs) from a surgical image, additional surgical elements such as organs, surgical tools, and surgical operations are further recognized based on the primitive level relationship for the event, The event can be recognized through the recognized additional surgical elements.

The surgical module layer may grasp a specific medical meaning or make a specific medical judgment based on each surgical element recognized through the surgical element recognition layer. For example, the surgical module layer, using at least one surgical element, bleeding loss estimation (Blood Loss Estimation), internal damage detection (Anatomy Injury Detection; for example, organ damage detection), tool misuse detection (Instrument Misuse Detection) It is possible to grasp medical meanings such as , Optimal Planning Suggestion, etc., or to judge a medical condition.

In other words, the surgical module layer can configure each surgical module according to the information needed in the medical surgery process or the medical problem to be solved (e.g., bleeding loss evaluation, internal damage detection, tool misuse detection, optimal surgical procedure suggestion, etc.). have. For example, the surgery module layer may constitute a bleeding loss evaluation module for determining the degree of bleeding of a subject for surgery in a medical surgery process. In addition, an internal damage detection module for determining to what extent damage has occurred to a specific organ during surgery may be configured. In addition, a surgical tool misuse detection module for determining whether the surgical tool is misused during surgery may be configured. In this case, each surgical module may selectively use at least one of the surgical elements recognized in the lower surgical element recognition layer.

In one embodiment, in selectively using at least one of the surgical elements, each surgical module may use a module level relationship (Module level Relation). The module-level relationship may mean designating and determining surgical elements to be recognized in the corresponding surgical module. For example, the module-level relationship may be that surgical elements to be recognized in the corresponding surgical module are determined based on the degree to which a specific meaning can be grasped from the surgical image (eg, the above-described representative recognition value; SAM).

The surgical solution layer can finally solve high-level medical problems by using the medical meaning or medical judgment identified through the surgical module layer. For example, the surgical solution layer uses at least one surgical module to generate a chart (Chart Generation), Complication Estimation, Surgical Performance Assessment, and a Surgical Q&A System (Q&A using Surgical Bot). System) and other medical problems can be solved.

In this case, the surgical solution layer may configure each solution module or system according to each medical problem (eg, chart generation, complication determination, surgical performance evaluation, surgical Q&A system, etc.). For example, the surgical solution layer may configure a chart generation solution module (or system) for recording all information generated in the surgical process or for recording the patient's condition. In addition, a complication determination solution module (or system) for predicting complications that may occur after a surgical procedure may be configured. In this case, each solution module (or system) may selectively use a medical meaning or medical judgment derived from a lower-level surgical module layer.

In an embodiment, in selectively using at least one of the surgical modules of the surgical module layer, each solution module (or system) of the surgical solution layer may use a solution level relation. The solution level relationship may mean that a surgical module that needs to be used in a corresponding solution module is determined and designated. For example, a medical staff may use a complication determination solution module to determine a patient's complications. In this case, the complication determination solution module may determine that a bleeding loss evaluation module and an internal body damage detection module are needed from a lower level based on the solution level relationship. In addition, it is possible to receive necessary information from the corresponding module of the lower layer, determine the patient's complications, and provide the determination result to the medical staff.

As described above, the medical solution model according to the embodiment of the present invention is configured to operate for each module from the lowest layer to the highest layer according to a medical problem. Therefore, even if a new medical problem occurs, a solution can be efficiently provided by configuring a new module for each layer or by configuring a new layer.

In addition, the medical solution model according to the embodiment of the present invention can be applied to various medical surgeries, and in particular, can be effectively utilized for minimally invasive surgery using a laparoscopy or an endoscope, as well as a surgical robot.

Hereinafter, with reference to the drawings, a detailed description of a method for analyzing surgery by a computer, a program, and an apparatus for analyzing surgery according to embodiments of the present invention will be described.

106 is a block diagram of a surgical image analysis apparatus according to an embodiment of the present invention.

Referring to Figure 106, the surgical image analysis apparatus according to an embodiment of the present invention, the surgical element recognition layer (10); and a surgical analysis layer 20 .

The surgical element recognition layer is a layer for recognizing elements involved in surgery and elements generated by surgery. Elements involved in the operation include a camera (eg, an endoscope used during laparoscopic surgery) for capturing a surgical image, a surgical tool, an organ, a blood vessel, and the like. In addition, the elements generated by the operation include an event (Event) such as operation of the surgical tool, bleeding, and the like. The surgical element recognition layer forms the lowest level in the surgical analysis system.

The surgical element recognition layer includes one or more surgical element recognition models that calculate a surgical recognition result as a surgical image is input. Hereinafter, a detailed description of the surgical element recognition model in each surgical element recognition layer will be described.

A plurality of surgical element recognition models may have a mutually connected relationship. That is, the recognition result of the surgical element recognition model B may be additionally used to calculate the recognition result of the surgical element recognition model A.

In one embodiment, it includes an organ recognition model for recognizing organs in the surgical image. That is, the organ recognition model plays a role of recognizing one or more organs present in each image frame in the surgical image. In addition, the organ recognition model may include a blood vessel model composed of arteries and veins.

The organ recognition model may recognize organs in a surgical image in various ways. For example, the organ recognition model may calculate an organ or blood vessel in which a surgical operation is performed based on a camera position obtained from a camera position recognition model to be described later or a surgical step from a surgical stage recognition model to be described later.

In another embodiment, it includes a surgical tool recognition model for recognizing a surgical tool (Instrument) in the surgical image. The surgical tool recognition model may recognize the type of surgical tool appearing in the image. For example, the surgical tool recognition model can recognize the surgical tools appearing in each frame according to the learning of the image of each surgical tool. Also, for example, the surgical tool recognition model may recognize a surgical tool performing a surgical operation inside the body as each surgical tool learns a plurality of surgical image data for performing an operation on an organ.

In another embodiment, the surgical tool operation (Instrument Action) recognition model; further includes. That is, the surgical tool motion recognition model plays a role of recognizing the meaning of the motion of the surgical tool performed in a specific surgical stage (ie, the motion to obtain a certain result).

Specifically, the surgical tool motion recognition model recognizes a basic surgical motion by learning a plurality of image frames included in the surgical image after acquiring a surgical image (or a specific section within the surgical image) requiring motion semantic recognition. The basic surgical operation means a basic operation such as cutting and grabbing using a specific surgical tool. Thereafter, the surgical tool motion recognition model extracts a set of continuous image frames from among the plurality of image frames based on the recognized surgical operation, and derives the meaning of the unit surgical operation through learning. The unit surgical operation refers to a surgical operation having a meaning for producing a specific result as an action for a specific organ in a surgical procedure.

To this end, the surgical tool motion recognition model includes: a basic surgical motion recognition module for recognizing a basic surgical motion that is a minimum unit of a surgical motion that does not reflect the meaning during surgery; and a surgical meaning recognition module for recognizing a unit surgical operation based on a continuous basic surgical operation.

In one embodiment, the basic surgical motion recognition module recognizes the basic surgical operation of the surgical tool by learning the surgical tool image, recognizing the surgical tool in successive frames, and calculating the state change of the surgical tool.

In addition, in one embodiment, as the operation meaning recognition module trains learning data matching the continuous basic operation operation and the unit operation operation, the basic operation recognition module recognizes the basic operation operation and the continuous basic operation image in the new operation image. When surgical motion data is input, a unit surgical motion is calculated. In addition, at this time, the learning data may further include organ information on which a continuous basic surgical operation is performed, and the organ information may be used as a result calculated by the organ recognition model in the surgical element recognition layer.

In addition, in another embodiment, it further includes a camera position recognition model for recognizing the position of the camera during minimally invasive surgery. Specifically, if the camera is an endoscope used in laparoscopic surgery, since the endoscope can move forward/backward, up/down/left/right, etc. in the abdominal cavity after the endoscope is inserted into the abdominal cavity, the camera position recognition model is It plays a role in calculating the position of the camera in the abdominal cavity in real time.

In one embodiment, the camera position recognition model sets the reference position of the camera based on one or more reference points of the reference object in the image captured by the camera, and calculates the relative position change based on the change of the object in the image. Calculate the camera position in real time.

Specifically, the method of calculating a camera position using an actual surgical image is to obtain a reference object from an actual surgical image captured by a camera in which a camera position recognition model enters the body, set a reference position for the camera, and the camera The amount of change in the position of the camera is calculated as . For example, the camera position recognition model calculates the amount of change in the position of the camera based on changes in the object in the real-time image acquired by the camera (eg, change in the size of the object, change in the position of the object, change in the object appearing in the image, etc.) Calculate. Thereafter, the camera position recognition model calculates the current position of the camera based on the change amount of the position of the camera with respect to the reference position. In this case, the actual surgical image may be a stereoscopic 3D image, and accordingly, the actual surgical image may be a three-dimensional image, that is, an image having a sense of depth. Therefore, it is possible to accurately grasp the position of the surgical tool in the three-dimensional space through the depth map of the actual surgical image.

The reference object is easy to detect features from an image, exists in a fixed position inside the body, has no or very little movement during surgery, does not change shape, is not affected by surgical tools, and is not affected by medical image data. It is possible to use an organ or a specific internal part that satisfies at least one of the conditions such as being able to be acquired even by (eg, data photographed by CT, PET, etc.). For example, a portion that moves very little during surgery, such as the liver and abdominal wall, and a portion that can be obtained from medical image data, such as the stomach, esophagus, and gallbladder, may be determined as the reference object.

In addition, when the position of the camera is moved, the surgical tool in the image captured by the camera as the reference object may be used as the reference object. In general, when performing robotic surgery, since movement of the camera and movement of the surgical tool do not occur at the same time, a surgical tool stopped at a specific position may be used as a reference object when the camera is moved.

In addition, the camera position recognition model may repeat the process of resetting the reference position of the camera based on one or more reference points of the reference object when the reference object appears in the surgical image. That is, when camera movement is frequent, an error may occur in the camera position as relative movement from the reference position is accumulated. can be done

In addition, the camera position recognition model may accumulate the camera position in the generated virtual body model to be the same as the actual patient's body. That is, the camera position recognition model may accumulate the relative position change with respect to the reference position of the camera after matching the real reference object in the real patient's body with the virtual reference object in the virtual body model. Through this, the camera position recognition model can record in what path the camera moved during the operation in the 3D virtual body model.

In addition, in another embodiment, the camera position recognition model, as the position of the camera is recognized, calculates the position of each surgical tool arranged relative to the position of the camera. That is, the camera position recognition model calculates the relative positions of one or more surgical tools from a specific position of the camera.

In addition, in another embodiment, it further includes a surgical tool position calculation model. As an embodiment of the surgical tool position calculation model, the position of the first point of the surgical tool in the external space of the surgical subject based on the sensing information obtained from the sensing device attached to the surgical tool to be inserted into the internal space of the surgical subject. , and through a virtual body model created in accordance with the physical condition of the subject to be operated on, the surgical tool's characteristic information is reflected based on the position of the first point of the surgical tool to present the surgical tool for the internal space of the surgical subject. 2 Calculate the position of the point. Through this, the surgical tool position calculation model provides the position information of the surgical tool in the actual internal body space of the surgical subject based on the position of the second point of the surgical tool with respect to the virtual body model.

As another embodiment of the surgical tool position calculation model, the coordinate system of the surgical robot system and the coordinate system of the virtual body model are matched, and movement data of the surgical tool movement of the surgical robot system is received and applied to the virtual body model. Calculate the real-time location of Specifically, the surgical tool position calculation model is based on the reference point of the surgical subject (eg, a specific position such as the navel of the patient's body, a marker displayed on the patient's body surface, or an identification mark projected by the surgical robot system on the body surface) to obtain the coordinate information of the surgical robot including the surgical tool with the The position of the surgical tool in the virtual body model corresponding to the position of the tool is calculated. That is, as the surgical tool position calculation model matches the coordinate system of the virtual body model with the real patient body that the surgical robot actually moves, the real-time location of the surgical tool can be obtained by applying the surgical tool movement of the surgical robot to the virtual body model. have.

In addition, in another embodiment, it further includes a surgical stage recognition model. The surgical stage recognition model calculates which detailed surgical stage corresponds to a process in which a current operation is performed or a specific section in a surgical image among the entire surgical process. The surgical stage recognition model can calculate the surgical stage of the surgical image in various ways. For example, the surgical stage recognition model stores a procedure for each type of surgery, and may recognize a specific operating phase in the procedure based on the area where the camera is located in the patient's body.

In addition, in another embodiment, the event recognition model that recognizes an event occurring in the surgical image, wherein the event is a non-ideal situation during surgery including bleeding; includes. The event recognition model (eg, bleeding recognition model) may include: a bleeding recognition module; and a bleeding amount calculation module.

The bleeding presence/absence recognition module may recognize whether a bleeding region exists in a surgical image based on deep learning-based learning, and recognize a bleeding location within an image frame. As the bleeding presence recognition module learns a plurality of images including bleeding, it can recognize whether a bleeding region is included in the new image.

In one embodiment, the bleeding presence recognition module may convert each pixel in the surgical image to a specific value based on a feature map including feature information, and specify a bleeding region based on a specific value of each pixel in the surgical image. . Each pixel in the surgical image may be converted into a specific value based on a predetermined weight depending on whether the region corresponds to the bleeding characteristic based on the feature map (ie, according to the degree of influence on the bleeding characteristic). For example, the bleeding presence recognition module applies Grad CAM (Gradient-weighted Class Activation Mapping) technology that inversely estimates the learning result of recognizing the bleeding area in the surgical image through CNN, and sets each pixel of the surgical image to a specific value. can be converted to The bleeding presence recognition module gives a high value (eg, a high weight) to a pixel recognized as corresponding to the bleeding area in the surgical image based on the feature map, and a low value to the pixel recognized as not corresponding to the bleeding area Each pixel value of the surgical image can be transformed by giving (eg, low weight). The bleeding presence/absence recognition module can emphasize the bleeding area in the surgical image more through the converted pixel value, and through this, the bleeding area can be segmented and the location of the corresponding area can be estimated. In addition, the bleeding presence recognition module can apply the Grad CAM technology, generate a heat map for each pixel in the surgical image based on the feature map, and convert each pixel into a probability value. The bleeding presence or absence recognition module may specify a bleeding region in the surgical image based on the converted probability value of each pixel. For example, the bleeding presence recognition module may determine a pixel area having a large probability value as the bleeding area.

The bleeding amount calculation module is characterized in that it calculates the bleeding amount in the bleeding area based on the location of the bleeding area. In an embodiment, the bleeding amount calculation module may calculate the bleeding amount by using pixel information of a bleeding area in the surgical image. For example, the bleeding amount calculation module may calculate the bleeding amount by using the number of pixels corresponding to the bleeding region in the surgical image, color information (eg, RGB values) of the pixels, and the like.

In another embodiment, when the surgical image is a stereoscopic image, the bleeding amount calculation module acquires depth information of a bleeding region in the surgical image based on a depth map of the surgical image, and bleeds based on the acquired depth information. The amount of bleeding may be calculated by estimating the volume corresponding to the region. When the surgical image is a stereoscopic image, since it has a three-dimensional effect, that is, depth information, the volume of the bleeding region in the three-dimensional space can be grasped. For example, the bleeding amount calculation module obtains pixel information (eg, the number of pixels, position of pixels, etc.) of the bleeding area in the surgical image, and calculates the depth value of the depth map corresponding to the acquired pixel information of the bleeding area. can The bleeding amount calculation module may calculate the bleeding amount by determining the volume of the bleeding area based on the calculated depth value.

In another embodiment, when gauze is included in the surgical image, the bleeding amount calculation module may calculate the amount of bleeding in the bleeding area using the gauze information. For example, the bleeding amount calculation module may use the number of gauze in the surgical image, color information (eg, RGB value) of the gauze, etc. to calculate the amount of bleeding occurring in the bleeding area.

In addition, in another embodiment, it further includes an operation time calculation/prediction model. The operation time calculation/prediction model includes: an operation time prediction module for calculating an expected time until completion of a surgical stage being performed during operation; and an operation time calculation module for calculating the time each step is performed after the operation is completed.

For example, the operation time calculation module extracts a time corresponding to a specific surgical phase after the operation is completed, and calculates the total required time of the specific surgical phase.

In addition, for example, the operation time prediction module is based on the surgical image of a specific surgical stage prior to a specific time point during operation (eg, a prediction reference time) and the required time until a specific time point of the specific surgical step, Calculate the remaining operating time for the expected surgical stage. Specifically, the surgery time prediction module acquires a preset surgical image including a surgical operation for a specific surgical step, and uses the preset surgical image and the surgery required based on the preset surgical image. It generates learning data, and predicts the operation time in the specific surgical stage by performing learning based on the learning data.

The surgical analysis layer is a layer that calculates an analysis result based on the surgical elements obtained from one or more surgical element recognition models. The surgical analysis layer is formed as an upper layer of the surgical element recognition layer. The surgery analysis layer includes one or more surgical analysis models for obtaining analysis results based on a combination of surgical recognition results, which is a combination of results provided by the one or more surgical element recognition models.

In one embodiment, the surgery analysis layer is a blood loss recognition model that calculates the degree of blood loss based on a combination of a surgical recognition result including a surgical organ type recognized by the surgical element recognition layer, a surgical operation, and an event occurring during surgery includes ; To this end, the blood loss recognition model may be connected to at least one of an event recognition model, an organ recognition model, and a surgical motion recognition model included in the surgical element recognition layer to receive a combination of surgical recognition results. For example, when the blood loss recognition model is used during surgery, the blood loss level may be calculated and provided to the medical staff when an event occurs.

In addition, in another embodiment, the surgery analysis layer, the organ damage detection model for calculating the degree of organ damage based on the combination of the surgical recognition result including the operation stage and operation time recognized by the surgical element recognition layer; includes; do. The organ damage detection model can calculate the level of organ damage by receiving a specific surgical stage, an operation performed based on a specific surgical tool, and an operating time from each surgical element recognition model.

The blood loss recognition model and the organ damage detection model are used to calculate analysis results in each surgical procedure during or after surgery.

In addition, in another embodiment, when analyzing the surgical result after surgery is completed, the surgical analysis layer includes a surgical tool recognized by the surgical element recognition layer, an organ operated with the surgical tool, and the surgical tool. It further includes; a surgical tool misuse detection model that detects the use of an incorrect surgical tool based on the detailed surgical step of the entire operation being performed, and an event that occurred in the detailed surgical step.

In addition, in another embodiment, the surgery analysis layer, an optimal surgical plan calculation model for calculating an optimal surgical plan based on the combination of the surgical recognition results obtained in the surgical element recognition layer; further includes.

In an embodiment, the optimal surgical plan calculation model calculates and provides an optimal surgical plan to be performed later by analyzing previously acquired surgical image data in real time.

In addition, in another embodiment, the optimal surgery plan calculation model calculates and provides a surgical plan for an actual patient based on a combination of surgical recognition results for virtual surgery performed on the patient's virtual body model before surgery.

Also, in another embodiment, the optimal surgery plan calculation model may include a function of calculating an optimal entry position during minimally invasive surgery. Specifically, when a medical staff performs a virtual surgery within a virtual body model, the virtual surgical system generates only the operating part of the surgical tool to perform a surgical simulation (ie, virtual surgery) without the influence of the arm part of the surgical tool. That is, the virtual surgery is performed according to the surgical operation pattern of the medical staff without restrictions according to the patient's internal characteristics (eg, organ arrangement, blood vessel condition, etc.) or the characteristics of the surgical tool. The optimal surgical plan calculation model calculates the optimal surgical tool entry position during actual surgery by considering the internal characteristics of the patient's body or the characteristics of the surgical tool based on the result after performing the virtual surgery. That is, as the movement of the surgical tool operation unit continuously calculates the candidate region for the entry position of the arm, the region that can be the intersection is extracted.

As an example, when using a surgical robot or tool A, tool B, or tool C during laparoscopic surgery, the computer operates in an area that cannot be reached when the operation part of tool A performs a surgical operation (that is, it is operated by the length limitation of the surgical tool) The area of the body surface where a point that cannot be reached when performing additional surgical operations occurs) may be excluded from the possible entry range. In addition, the computer may exclude a body surface area that collides with body organs or tissues in the process where the tool A enters and performs a surgical operation from the access range. In addition, if the computer fails to implement a surgical operation required at a specific position after the surgical tool enters at each body surface point within the accessible range, the corresponding body surface point may be excluded from the accessible range. Through this, the computer can calculate the possible access range for tool A. The computer may calculate the optimal entry position of each surgical tool by individually performing the process of calculating the accessible range for each surgical tool (eg, tool B and tool C). In addition, as described above, the computer may calculate the optimal entry position to which the function of each surgical tool can be applied by individually performing the process of calculating the possible access range for each function based on the function of the surgical tool.

As another example, if several surgical tools need to enter one optimal entry position, the computer may extract the optimal entry range for each surgical tool, and then determine the overlapping range of the plurality of optimal entry ranges as the optimal entry location. have. For example, when tool A is changed to tool D in the course of performing surgery, the computer may calculate an overlapping area between the accessible range for tool A and the accessible range for tool D as an optimal entry position candidate area. . Since the position where the surgical tool can enter is limited to a certain number (for example, 3), the same entry position must be used when the tool A is changed to the tool D. A position that satisfies all of the possible entry ranges of the tool can be determined as the final optimal entry position.

As another example, when the same surgical tool is used multiple times in the virtual body model, all surgical operations at one surgical tool entry position when the surgical operation of the surgical tool (that is, the range of motion) is wide. may be difficult to perform, the computer can divide the range in which the surgical tool is used (ie, the range of motion) into several groups reachable from multiple entry points on the body surface. For example, when performing laparoscopic surgery or robotic surgery by creating three entry points on the body surface, the computer divides the range of motion of the surgical tool into three or less groups. At this time, the computer divides the range of motion based on whether it is reachable from a plurality of accessible ranges selected by other surgical tools. In addition, a specific surgical tool having a wide range of motion (ie, the first surgical tool) is used simultaneously with another surgical tool (ie, the second surgical tool) and another surgical tool (ie, the second surgical tool) is required to enter. When the optimal entry position is determined, the computer determines the range of motion of the first surgical tool when used together with the second surgical tool at the optimal entry position of the first surgical tool (that is, the second surgical tool enters). It can be set to a range that cannot be accessed through the keyhole). In addition, when the first surgical tool is continuously used despite the change of the other surgical tools, the computer enters the same entry position in consideration of the user's convenience and the time required for surgery, and the operation must be performed. It can be set as a group.

The medical staff determines the optimal entry position of the surgical tool by reflecting the results of the surgical simulation performed without considering the entry position of the surgical tool and the long-term jamming of the arm part of the surgical tool, so that the medical staff can perform the most comfortable surgical operation. .

In addition, in another embodiment, the optimal surgery plan calculation model is based on the result of the medical staff performing the virtual surgery only with the operation part of the surgical tool from which the cancer part has been removed, and selects a surgical tool type suitable for use in surgery. It can be calculated and presented.

In addition, as an embodiment, the optimal surgery plan calculation model may perform training in various ways. For example, it is possible to obtain an image of a surgery performed by a plurality of medical staff and calculate an optimal surgery for each type of surgery through reinforcement learning.

In addition, for example, the optimal surgery plan calculation model does not use actual surgical data, but performs a virtual surgery with a virtual generated surgical procedure and then repeats the process of evaluating whether it is an optimized surgery. can create

Specifically, the optimal surgical plan calculation model is optimized by generating a plurality of genes corresponding to the surgical process based on a surgical process consisting of at least one detailed surgical operation, and performing virtual surgery on each of the plurality of genes. Evaluate surgery. Then, the optimal surgery plan calculation model selects at least one gene from among a plurality of genes based on the evaluation result, applies a genetic algorithm, generates a new gene by applying the genetic algorithm, and based on the new gene Deduce the optimal surgical procedure.

The optimal operation plan calculation model may calculate the fitness by performing virtual surgery on a new gene (child gene). In addition, the optimal surgical plan calculation model determines whether the fitness of a new gene (child gene) meets a preset condition, selects a new gene (child gene) that meets the condition, and applies a genetic algorithm such as crossbreeding, mutation, etc. can do. By applying this genetic algorithm to a new gene (child gene), it is possible to generate a new child gene again. That is, the computer repeatedly generates offspring genes from parent genes based on the fitness result to evaluate whether it is the optimal surgery, and finally obtains a gene including the optimal surgical procedure from among the generated offspring genes. have. For example, it is possible to select a gene having the highest fitness among the offspring genes and derive it through an optimized surgical procedure.

Each surgery analysis model in the surgery analysis layer may receive a combination of surgical recognition results in various forms.

In one embodiment, the surgical analysis model may be input in the form of code data connected by coding the results calculated from the surgical element recognition model. That is, each surgical analysis model can obtain code data that connects the code for the surgical recognition result at each time point, and then extract and input the code part required for analysis within each model.

In addition, in another embodiment, the surgical analysis model, for each of a plurality of image frames, based on the surgical recognition information calculated from the plurality of surgical element recognition model between the surgical element (surgical element) included in the surgical recognition information Relational representation information representing a relationship may be input. Surgical recognition information refers to information on surgical elements recognized from an image frame, for example, surgical tools, surgical motions, body parts, bleeding or not, surgical steps, operation time (eg, remaining operation time, operation time required, etc.), and camera information ( For example, the position, angle, direction, movement, etc. of the camera) may include at least one surgical element. For example, the relational expression may be in the form of a matrix in which rows and columns are listed as each surgical element, and a value for correlation between surgical elements is applied as a matrix value. Also, as an example, the surgical analysis model may be input after calculating a relational expression for each frame of the surgical image or a specific unit-segmented image.

In addition, in another embodiment, the surgical solution providing layer; further includes. The surgical solution providing layer includes one or more surgical solution models that provide an organized surgical product based on one or more analysis results obtained in the surgical analysis layer. That is, the surgical solution providing layer includes one or more surgical solution models, which are service types that medical staff can use immediately.

The surgical solution providing layer is formed as an upper layer of the surgery analysis layer, and receives the results calculated from each surgery analysis model in the surgery analysis layer.

In an embodiment, the surgical solution model includes a surgical evaluation model that calculates an evaluation result for surgery based on one or more analysis results obtained in the surgery analysis layer. For example, the surgery evaluation model may receive a blood loss result, a calculation result of an organ damage model, and a comparison result between an optimal surgical process and an actual surgery, and calculate an evaluation of the surgery performed by the medical staff.

Also, in another embodiment, the surgery solution model further includes a chart generation model for generating a chart for surgery based on one or more analysis results obtained in the surgery analysis layer. The chart generation model can automatically create a record of the process of performing the surgery, the results of the surgery, and the like. For example, the chart generation model combines the surgical elements obtained in the surgical process after surgery is completed, inputs it to each surgery analysis model, receives the analysis results of a plurality of surgery analysis models, and receives a chart (for example, surgery recording paper) can be automatically created (written).

In addition, in another embodiment, the operation complexity calculation model for calculating the complexity or difficulty of the operation based on one or more analysis results obtained in the operation analysis layer; further includes. For example, by using the operation complexity calculation model, the performance evaluation of the medical staff may be performed in consideration of the difficulty of the surgery performed by each medical staff.

In addition, for example, when evaluating the surgical result, the operation complexity calculated from the operation complexity calculation model may be reflected. That is, in the surgical solution providing layer, the surgery evaluation model may receive the complexity calculation result calculated from the surgery complexity calculation model and use it for surgery evaluation.

In addition, in another embodiment, as one or more analysis results obtained in the operation analysis layer are learned, a surgery Q&A model that calculates answers to questions about surgery; includes.

A detailed description of the case in which the surgical image analysis apparatus according to an embodiment of the present invention is composed of a surgical element recognition layer, a surgical analysis layer, and a surgical solution layer will be described.

As an embodiment, the lowest layer may be a surgical element recognition layer that recognizes surgical elements, which are the minimum units to be recognized medically in a medical surgery process. The middle layer may be a surgical analysis layer (Surgical Module Layer) that makes it possible to determine medical meaning through each surgical element and determine a condition for making a medical diagnosis. The top layer may be a Surgical Solution Layer that recognizes medical problems that may occur in the entire medical surgical process and provides solutions for each problem.

The surgical element recognition layer may recognize surgical elements based on various surgical images captured in a medical surgical procedure and various surgical-related tools used in a medical surgical process. For example, the surgical element is a surgical phase (Phase), body part (eg, organ; Organ), event (Event), operation time (Time), surgical instrument (Instrument), camera (Camera), operation (Acton) , and other elements.

In an embodiment, the surgical element recognition layer may recognize at least one surgical element based on a surgical image obtained in a medical surgery process. In this case, not only the surgical elements recognized in the surgical image itself, but also the surgical elements may be recognized using the result of learning the surgical image. By utilizing the above-described embodiments of the present invention, it is possible to effectively recognize the surgical elements, which are the minimum units to be recognized medically in a medical surgery process.

The surgical element recognition layer may individually recognize the surgical elements. For example, when an organ is recognized from a surgical image, only a corresponding organ can be recognized, and when a surgical tool is recognized from a surgical image, only the corresponding surgical tool can be recognized.

In addition, the surgical element recognition layer may use another surgical element to recognize one surgical element. That is, the surgical element recognition layer may set a primitive level relation indicating a relationship between each surgical element, and recognize each surgical element using the primitive level relationship. For example, the primitive level relationship lists the additional surgical elements required to recognize the corresponding surgical element, and the relationship between these surgical elements (eg, state change, position change, shape change, color change, arrangement relationship, etc.) It may include specified information and the like. For example, when recognizing an event (for example, an event on whether or not bleeding occurs) from a surgical image, additional surgical elements such as organs, surgical tools, and surgical operations are further recognized based on the primitive level relationship for the event, The event can be recognized through the recognized additional surgical elements.

The surgical analysis layer may grasp a specific medical meaning or make a specific medical judgment based on each surgical element recognized through the surgical element recognition layer. For example, the surgical analysis layer, by using at least one surgical element, blood loss estimation (Blood Loss Estimation), internal damage detection (Anatomy Injury Detection; for example, organ damage detection), tool misuse detection (Instrument Misuse Detection) It is possible to grasp medical meanings such as , Optimal Planning Suggestion, etc., or to judge a medical condition.

That is, the surgical analysis layer can configure each surgical module according to the information required in the medical surgery process or the medical problem to be solved (e.g., bleeding loss assessment, internal damage detection, tool misuse detection, optimal surgical procedure suggestion, etc.). have. For example, the surgery analysis layer may constitute a bleeding loss evaluation module for understanding the degree of bleeding of a subject for surgery in the course of medical surgery. In addition, an internal damage detection module for determining to what extent damage has occurred to a specific organ during surgery may be configured. In addition, a surgical tool misuse detection module for determining whether the surgical tool is misused during surgery may be configured. In this case, each surgical module may selectively use at least one of the surgical elements recognized in the lower surgical element recognition layer.

In one embodiment, in selectively using at least one of the surgical elements, each surgical module may use a module level relationship (Module level Relation). The module-level relationship may mean designating and determining surgical elements to be recognized in the corresponding surgical module. For example, the module-level relationship may be that surgical elements to be recognized in the corresponding surgical module are determined based on the degree to which a specific meaning can be grasped from the surgical image (eg, the above-described representative recognition value; SAM).

The surgical solution layer can finally solve high-level medical problems by using the medical meaning or medical judgment identified through the surgical analysis layer. For example, the surgical solution layer uses at least one surgical module to generate a chart (Chart Generation), Complication Estimation, Surgical Performance Assessment, and a Surgical Q&A System (Q&A using Surgical Bot). System) and other medical problems can be solved.

In this case, the surgical solution layer may configure each solution module or system according to each medical problem (eg, chart generation, complication determination, surgical performance evaluation, surgical Q&A system, etc.). For example, the surgical solution layer may configure a chart generation solution module (or system) for recording all information generated in the surgical process or for recording the patient's condition. In addition, a complication determination solution module (or system) for predicting complications that may occur after a surgical procedure may be configured. In this case, each solution module (or system) may selectively use a medical meaning or a medical judgment derived from a lower-level surgical analysis layer.

In an embodiment, in selectively using at least one of the surgical modules of the surgical analysis layer, each solution module (or system) of the surgical solution layer may use a solution level relation. The solution level relationship may mean that a surgical module that needs to be used in a corresponding solution module is determined and designated. For example, a medical staff may use a complication determination solution module to determine a patient's complications. In this case, the complication determination solution module may determine that a bleeding loss evaluation module and an internal body damage detection module are needed from a lower level based on the solution level relationship. In addition, it is possible to receive necessary information from the corresponding module of the lower layer, determine the patient's complications, and provide the determination result to the medical staff.

As described above, the medical solution model according to the embodiment of the present invention is configured to operate for each module from the lowest layer to the highest layer according to a medical problem. Therefore, even if a new medical problem occurs, a solution can be efficiently provided by configuring a new module for each layer or by configuring a new layer.

In addition, the medical solution model according to the embodiment of the present invention can be applied to various medical surgeries, and in particular, can be effectively utilized for minimally invasive surgery using a laparoscopy or an endoscope, as well as a surgical robot.

107 is a flowchart of a surgical analysis method by a computer according to an embodiment of the present invention.

Referring to FIG. 107 , the method for analyzing surgery by a computer according to an embodiment of the present invention includes: acquiring a surgical image by the computer (S200); Computer inputting the surgical image to one or more surgical element recognition model (S400); obtaining, by the computer, a combination of surgical recognition results calculated from each surgical element recognition model (S600; operation recognition result combination acquisition step); and obtaining, by the computer, the combination of the surgical recognition results into one or more surgical analysis models to obtain one or more analysis results (S800; surgical analysis result acquisition step). Hereinafter, detailed description of each step is described.

The computer acquires a surgical image (S200). The computer may acquire the surgical image in real time while the surgery is being performed by the medical staff. In addition, the computer may acquire the entire surgical image stored after the medical staff completes the operation.

For example, when a computer uses a surgical image acquired in real time, the computer utilizes the surgical image to provide necessary information during an operation of a medical staff. Also, for example, when the computer uses the entire image after surgery is completed, the computer uses the surgical image to perform a post-operative analysis on the operation.

The computer inputs the surgical image to one or more surgical element recognition models (S400). That is, the computer inputs the surgical image to one or more surgical element recognition models in order to recognize the surgical elements included in the surgical image.

The one or more surgical element recognition models are included in the surgical element recognition layer that forms the lowest layer in the surgical analysis system. In addition, when the surgical element recognition layer includes a plurality of surgical element recognition models, each surgical element recognition model is built in parallel in the surgical element recognition layer, and may have a mutually connected relationship. That is, the surgical element (ie, recognition result) calculated from the surgical element recognition model B formed in parallel in the process of performing the A surgical element recognition model can be received and used. A detailed description of each previously described surgical element recognition model will be omitted.

The computer acquires a combination of surgical recognition results calculated from each surgical element recognition model (S600; operation recognition result combination acquisition step). That is, the computer generates data combining each surgical element recognition result calculated from each surgical element recognition model.

The computer acquires one or more analysis results by inputting the combination of the surgical recognition results into one or more surgical analysis models (S800; surgical analysis result acquisition step). The one or more surgical analysis models are included in the surgical analysis layer on the surgical element recognition layer, and are selected according to a user's request. That is, each surgical analysis model in the surgical analysis layer has a connection relationship with one or more surgical element recognition models in the surgical element recognition layer based on the data required for analysis, so that the combination of surgical element recognition results required for analysis is can be entered into each surgical analysis model. A detailed description of each of the previously described surgical analysis models will be omitted.

In addition, in another embodiment, as in FIG. 108, the computer inputs one or more analysis results to a specific surgical solution model, and provides an organized surgical product based on the analysis results (S1000); further comprising; . The surgical solution model is included in the surgical solution providing layer, and the surgical solution providing layer is created in the surgical analysis system as an upper layer of the surgical analysis layer. In addition, each surgical solution model may receive one or more surgical analysis results as it is connected to one or more surgical analysis models. A detailed description of each of the previously described surgical solution models will be omitted.

The steps of a method or algorithm described in relation to an embodiment of the present invention may be implemented directly in hardware, as a software module executed by hardware, or by a combination thereof. A software module may contain random access memory (RAM), read only memory (ROM), erasable programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory, hard disk, removable disk, CD-ROM, or It may reside in any type of computer-readable recording medium well known in the art to which the present invention pertains.

As mentioned above, although embodiments of the present invention have been described with reference to the accompanying drawings, those skilled in the art to which the present invention pertains can realize that the present invention can be embodied in other specific forms without changing its technical spirit or essential features. you will be able to understand Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive.

Claims (13)

computer acquiring a surgical image;
inputting the surgical image into one or more surgical element recognition models by a computer;
A computer acquires a combination of surgical recognition results calculated from each surgical element recognition model, wherein one or more surgical element recognition models are included in the surgical element recognition layer, which is the lowest level in the surgical analysis system, a surgical recognition result combination acquisition step; and
The computer inputs the combination of the surgical recognition results into one or more surgical analysis models to obtain one or more analysis results, wherein the one or more surgical analysis models are included in the surgical analysis layer above the surgical element recognition layer. Including, a method for analyzing surgery by a computer; According to claim 1,
Each surgical analysis model in the surgical analysis layer,
Based on the data required for analysis, a method for analyzing surgery by a computer, characterized in that a connection relationship is established with one or more surgical element recognition models in the surgical element recognition layer. According to claim 1,
The surgical element recognition model,
an organ recognition model for recognizing organs in the surgical image;
a surgical tool recognition model for recognizing a surgical tool in the surgical image and a movement of the surgical tool; and
Recognizing an event occurring in the surgical image, wherein the event is a non-ideal situation during surgery including bleeding, an event recognition model; including, a method for analyzing surgery by a computer. According to claim 1,
The surgical analysis layer,
a blood loss recognition model for calculating the degree of blood loss based on a combination of a surgical recognition result including a surgical organ type recognized by the surgical element recognition layer, a surgical operation, and an event occurring during surgery; and
It includes a; organ damage detection model for calculating the degree of organ damage based on the combination of the surgical recognition result including the surgical stage and the surgical time recognized by the surgical element recognition layer;
The blood loss recognition model and the organ damage detection model are
Surgical analysis method by a computer, which is used to calculate analysis results in each surgical procedure during or after surgery. 5. The method of claim 4,
When analyzing the results of surgery after surgery is completed,
The surgical analysis layer,
The use of the wrong surgical tool based on the surgical tool recognized in the surgical element recognition layer, the organ for which the operation is performed with the surgical tool, the detailed surgical step during the entire operation performed with the surgical tool, and the event that occurred in the detailed surgical step Detecting the, surgical tool misuse detection model; further comprising, surgery analysis method by a computer. According to claim 1,
The surgical analysis layer,
Further comprising; an optimal surgical plan calculation model for calculating an optimal surgical plan based on the combination of surgical recognition results obtained from the surgical element recognition layer;
The optimal surgical plan calculation model is a method for analyzing surgery by a computer, characterized in that it calculates and provides an optimal surgical plan to be performed later by analyzing the previously acquired surgical image data in real time. According to claim 1,
The surgical analysis layer,
Further comprising; an optimal surgical plan calculation model for calculating an optimal surgical plan based on the combination of surgical recognition results obtained from the surgical element recognition layer;
The optimal surgical plan calculation model is a computerized surgery, characterized in that it calculates and provides a surgical plan for an actual patient based on a combination of surgical recognition results for virtual surgery performed on a virtual body model of a patient before surgery analysis method. According to claim 1,
The computer inputting one or more analysis results into a specific surgical solution model, and providing a surgical product organized based on the analysis results; further comprising, a method for analyzing surgery by a computer. 9. The method of claim 8,
The surgical solution model is,
a surgery evaluation model for calculating an evaluation result for surgery based on one or more analysis results obtained in the surgery analysis layer;
a chart generation model for generating a chart for surgery based on one or more analysis results obtained in the surgery analysis layer; and
Surgical complexity calculation model for calculating the complexity or difficulty of surgery based on one or more analysis results obtained in the surgery analysis layer; further comprising, a method for analyzing surgery by a computer. 9. The method of claim 8,
and a surgical Q&A model that calculates answers to questions about surgery by learning one or more analysis results obtained in the surgery analysis layer;
The step of providing the surgical product,
A method of analyzing surgery by a computer that calculates and provides answers as a medical staff inputs a question about a specific surgery. In combination with a computer, which is hardware, any one of claims 1 to 10 is stored in the medium to execute the method, the surgical analysis program by the computer. Including one or more surgical element recognition models that calculate a surgical recognition result as a surgical image is input, wherein the one or more surgical element recognition models are included in the surgical element recognition layer, which is the lowest level in the surgical analysis system, a surgical element recognition layer ; and
a surgical analysis layer comprising one or more surgical analysis models for obtaining an analysis result based on a combination of surgical recognition results, which is a combination of results provided by the one or more surgical element recognition models, formed as an upper layer of the surgical element recognition layer; Including, a surgical image analysis device. 13. The method of claim 12,
Surgical image analysis device further comprising a; comprising one or more surgical solution models that provide an organized surgical product based on one or more analysis results obtained in the surgical analysis layer.

Images