KR102013828B1 - Method and apparatus for predicting surgical duration based on surgical video - Google Patents

Abstract

Provided is a method for predicting a surgery time on the basis of a surgery image. The method comprises steps of: acquiring a predetermined surgery image including a surgical operation for a specific surgery step; generating learning data by using the predetermined surgery image and a surgery required time obtained on the basis of the predetermined surgery image; and performing learning on the basis of the learning data to predict the surgery time in the specific surgery step. According to the present invention, it is possible to derive an accurate prediction surgery time for each surgery step.

Description

Method and apparatus for predicting surgery time based on surgical image {METHOD AND APPARATUS FOR PREDICTING SURGICAL DURATION BASED ON SURGICAL VIDEO}

The present invention relates to a method and apparatus for predicting a surgery time based on a surgical image.

Medical surgery can be classified into open surgery, laparoscopic surgery, minimally invasive surgery (MIS) including radio surgery, radio surgery, and the like. Laparoscopic surgery refers to surgery performed by medical staff to see and touch the part to be treated. Minimally invasive surgery is also known as keyhole surgery, and laparoscopic surgery and robotic surgery are typical. In laparoscopic surgery, a small hole is made in a necessary part without opening, and a laparoscopic with a special camera is attached and a surgical tool is inserted into the body and observed through a video monitor. Microsurgery is performed using a laser or a special instrument. In addition, robot surgery is to perform minimally invasive surgery using a surgical robot. Furthermore, radiation surgery refers to surgical treatment with radiation or laser light outside the body.

In the case of such a medical operation, a surgical image is often obtained at the time of the actual operation and the surgery is performed based on this. Therefore, it is important to provide a variety of information through the surgical image obtained during the actual operation. In particular, it is necessary to provide information about the prediction of the operation time in the operation process. Thus, there is a need for a method that can predict the operation time using the surgical image acquired during the actual surgery.

The problem to be solved by the present invention is to provide a method and apparatus for predicting the operation time based on the surgical image.

The problem to be solved by the present invention is to provide a method and apparatus for predicting the operation time for each surgical step by performing the learning based on the surgical image.

The problem to be solved by the present invention is to provide a method and apparatus for estimating the remaining surgery time required to perform the remaining process remaining in the current surgical stage through the surgical image obtained in real time.

The problem to be solved by the present invention is to provide a method and apparatus for generating a variety of learning data to be used to predict the operation time based on the surgical image obtained through the actual operation of the patient.

Problems to be solved by the present invention are not limited to the above-mentioned problems, and other problems not mentioned will be clearly understood by those skilled in the art from the following description.

According to an embodiment of the present invention, a method for predicting a surgery time based on a surgery image performed by a computer includes: obtaining a preset surgery image including a surgery operation for a specific surgery stage, the preset surgery image, and Generating learning data using the surgery time obtained based on the predetermined surgery image, and performing the learning based on the learning data to predict the surgery time in the specific surgery stage.

In an embodiment of the present disclosure, the generating of the learning data may include obtaining a first surgery time based on the preset surgery image, and removing a predetermined section from the preset surgery image. (censored) generating a surgical image, acquiring a second surgery time based on the intermediate cut surgery image, and the preset surgery image based on the first surgery time and the second surgery time And generating at least one of the intermediately cut surgical images as the learning data.

In one embodiment of the present invention, the step of predicting the operation time, learning using the operation time required to perform the specific operation step based on each image frame and each image frame in the learning data. Performing, and predicting an average surgery time in the specific surgery step through the learning.

In one embodiment of the present invention, the step of performing the learning, when the predetermined surgical image is generated as the learning data, based on each image frame and the first surgery time in the predetermined surgical image Learning can be done.

In one embodiment of the present invention, the step of performing the learning, each of the image frame and the second surgery time in the intermediate-cut surgical image generated when the intermediate-cut surgical image You can learn on the basis.

In one embodiment of the present invention, further comprising the step of obtaining the actual surgery image by performing the specific surgery step in the actual surgery process, the step of predicting the operation time, the image of the current time point in the actual surgery image Acquiring an operation time required up to the present time required to perform the specific operation step based on a frame, and an average operation time required for the specific operation step predicted through the learning and an operation time required up to the current time point The method may include estimating the remaining surgery time required to perform the specific surgery step after the current time based on time.

In one embodiment of the present invention, the obtaining of the predetermined surgical image, it is possible to obtain a predetermined surgical image performing the specific surgical step, respectively from a plurality of patients.

In one embodiment of the present invention, the specific surgical step may be any one of the surgical steps belonging to a specific hierarchy in the surgical process consisting of a hierarchical structure according to the operation.

An apparatus according to an embodiment of the present invention includes a memory for storing one or more instructions, and a processor for executing the one or more instructions stored in the memory, wherein the processor executes the one or more instructions to perform a specific surgical step. Acquiring a predetermined surgical image including a surgical operation for the step, generating learning data using a surgery time required based on the predetermined surgical image and the predetermined surgical image, and the learning data Learning is performed based on the step of predicting the operation time in the specific surgery stage.

A computer program according to an embodiment of the present invention is combined with a computer, which is hardware, and stored in a computer-readable recording medium to perform a method of predicting an operation time based on the surgery image.

According to the present invention, it is possible to predict the operation time and the operation remaining time for each operation stage based on the surgical image. Providing the estimated time required for surgery and the time remaining for the operation to the medical staff, it is possible to more accurately understand the current situation of the operation is progressing through it, it is possible to proceed efficiently with the remaining surgery.

According to the present invention, it is possible to provide the time required for surgery and the operation remaining time in real time, through which an anesthetic dosage can be calculated. In addition, it is effective to determine the appropriate additional anesthesia dose according to the remaining surgery time.

According to the present invention, it is possible to generate a variety of learning data on the basis of the surgical image obtained through the actual surgical process of the patient, through which the learning for the prediction of the surgery time can be performed more effectively. In addition, by performing the learning using a variety of learning data it is possible to derive the correct prediction operation time for each surgical step.

Effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the following description.

1 is a schematic diagram of a system capable of performing robot surgery according to an embodiment of the present invention.
2 is a flowchart schematically illustrating a method for predicting a surgery time based on a surgery image according to an exemplary embodiment of the present invention.
3 is a view for explaining a process of generating learning data based on a surgical image according to an embodiment of the present invention.
4 is a diagram illustrating a process of performing learning based on learning data according to an embodiment of the present invention.
5 is a view schematically showing the configuration of an apparatus 200 for performing a method for predicting a surgery time based on a surgery image according to an embodiment of the present invention.

Advantages and features of the present invention and methods for achieving them will be apparent with reference to the embodiments described below in detail with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but can be embodied in various different forms, and the present embodiments only make the disclosure of the present invention complete, and those of ordinary skill in the art to which the present invention belongs. It is provided to fully inform the skilled worker of the scope of the invention, which is defined only by the scope of the claims.

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. In this specification, the singular also includes the plural unless specifically stated otherwise in the phrase. As used herein, "comprises" and / or "comprising" does not exclude the presence or addition of one or more other components in addition to the mentioned components. Like reference numerals refer to like elements throughout, and "and / or" includes each and all combinations of one or more of the mentioned components. Although "first", "second", etc. are used to describe various components, these components are of course not limited by these terms. These terms are only used to distinguish one component from another. Therefore, of course, the first component mentioned below may be a second component within the technical spirit of the present invention.

Unless otherwise defined, all terms used in the present specification (including technical and scientific terms) may be used in a sense that can be commonly understood by those skilled in the art. In addition, terms that are defined in a commonly used dictionary are not ideally or excessively interpreted unless they are specifically defined clearly.

As used herein, the term "part" or "module" refers to a hardware component such as software, FPGA, or ASIC, and the "part" or "module" plays certain roles. However, "part" or "module" is not meant to be limited to software or hardware. The “unit” or “module” may be configured to be in an addressable storage medium or may be configured to play one or more processors. Thus, as an example, a "part" or "module" may include components such as software components, object-oriented software components, class components, and task components, processes, functions, properties, Procedures, subroutines, segments of program code, drivers, firmware, microcode, circuitry, data, databases, data structures, tables, arrays, and variables. Functions provided within components and "parts" or "modules" may be combined into smaller numbers of components and "parts" or "modules" or into additional components and "parts" or "modules". Can be further separated.

As used herein, the term "computer" includes all the various devices capable of performing arithmetic processing to provide a result to a user. For example, a computer can be a desktop PC, a notebook, as well as a smartphone, a tablet PC, a cellular phone, a PCS phone (Personal Communication Service phone), synchronous / asynchronous The mobile terminal of the International Mobile Telecommunication-2000 (IMT-2000), a Palm Personal Computer (PC), a Personal Digital Assistant (PDA), and the like may also be applicable. In addition, when a head mounted display (HMD) device includes a computing function, the HMD device may be a computer. Also, the computer may correspond to a server that receives a request from a client and performs information processing.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a schematic diagram of a system capable of performing robot surgery according to an embodiment of the present invention.

According to FIG. 1, the robotic surgical system includes a medical imaging apparatus 10, a server 100, a control unit 30 provided in an operating room, a display 32, and a surgical robot 34. According to an embodiment, the medical imaging apparatus 10 may be omitted in the robot surgery system according to the disclosed embodiment.

In one embodiment, surgical robot 34 includes imaging device 36 and surgical instrument 38.

In one embodiment, the robot surgery is performed by the user controlling the surgical robot 34 using the control unit 30. In one embodiment, the robot surgery may be automatically performed by the controller 30 without the user's control.

The server 100 is a computing device including at least one processor and a communication unit.

The controller 30 includes a computing device including at least one processor and a communication unit. In one embodiment, the control unit 30 includes hardware and software interfaces for controlling the surgical robot 34.

The imaging device 36 includes at least one image sensor. That is, the imaging device 36 includes at least one camera device and is used to photograph an object, that is, a surgical site. In one embodiment, the imaging device 36 includes at least one camera coupled with a surgical arm of the surgical robot 34.

In an embodiment, the image photographed by the photographing apparatus 36 is displayed on the display 340.

In one embodiment, surgical robot 34 includes one or more surgical tools 38 that can perform cutting, clipping, fixing, grabbing operations, and the like, of the surgical site. Surgical tool 38 is used in conjunction with the surgical arm of the surgical robot 34.

The controller 30 receives information necessary for surgery from the server 100 or generates information necessary for surgery and provides the information to the user. For example, the controller 30 displays the information necessary for surgery, generated or received, on the display 32.

For example, the user performs the robot surgery by controlling the movement of the surgical robot 34 by manipulating the control unit 30 while looking at the display 32.

The server 100 generates information necessary for robotic surgery using medical image data of an object previously photographed from the medical image photographing apparatus 10, and provides the generated information to the controller 30.

The controller 30 displays the information received from the server 100 on the display 32 to provide the user, or controls the surgical robot 34 by using the information received from the server 100.

In one embodiment, the means that can be used in the medical imaging apparatus 10 is not limited, for example, other various medical image acquisition means such as CT, X-Ray, PET, MRI may be used.

As described above, when performing a robot operation, it is possible to obtain data including various surgical information in the surgical image taken during the surgical procedure or the control of the surgical robot. As described above, the present invention is to perform the learning using the surgical image or the surgical data that can be obtained in the surgical process as a learning material, and to provide a method that can be utilized in the surgical process of other patients through such learning.

Hereinafter, for convenience of description, it will be described that a "computer" performs a method of predicting a surgery time based on a surgery image according to an embodiment disclosed herein. "Computer" may mean the server 100 or the controller 30 of FIG. 1, but is not limited thereto and may be used to encompass a device capable of performing computing processing. For example, the computer may be a computing device provided separately from the device shown in FIG. 1.

In addition, the embodiments disclosed below may not be applicable only in connection with the robotic surgery system illustrated in FIG. 1, but may be applied to all kinds of embodiments that may acquire and utilize a surgical image in a surgical procedure. For example, in addition to robotic surgery, it can be applied in connection with minimally invasive surgery such as laparoscopic surgery or endoscopic surgery.

2 is a flowchart schematically illustrating a method for predicting a surgery time based on a surgery image according to an exemplary embodiment of the present invention.

Referring to Figure 2, the method for predicting the operation time based on the surgical image according to an embodiment of the present invention, the step of obtaining a predetermined surgical image including a surgical operation for a specific surgical step (S100), Generating learning data using a predetermined surgery image and a surgery time required based on the predetermined surgery image (S200), and performing a learning based on the learning data to perform a surgery time in the specific surgery stage. It may include the step (S300) to predict. Hereinafter, a detailed description of each step will be described.

The computer may acquire a predetermined surgical image including a surgical operation for a specific surgical step (S100).

When performing certain operations (eg, gastric cancer surgery, colorectal cancer surgery, etc.) on the patient, the medical staff may perform the actual surgery on the patient directly, as well as laparoscopes or endoscopes, including the surgical robot as described in FIG. Minimally invasive surgery may be performed. In this case, the computer may acquire a surgical image photographing a scene including a surgical operation performed in the surgical procedure, a surgical tool related thereto, a surgical site, and the like. In an embodiment, the computer may acquire a surgical image photographing a scene including a surgical site and a surgical tool that is currently undergoing surgery from a camera entering the patient's body.

Meanwhile, each surgery (eg, gastric cancer surgery, colorectal cancer surgery, etc.) may be performed in different surgical procedures, but the same type of surgery may be performed in the same or similar series of surgical procedures. That is, when performing a specific surgery (for example, the same type of surgery, such as gastric cancer surgery) for a plurality of patients, it is possible to proceed with a specific surgical procedure by performing the same or similar surgical operations for each patient.

In one embodiment, a particular surgery may consist of predefined surgical procedures (ie, surgical steps) according to classification criteria. For example, the computer may classify the surgical stages according to the time course of a specific surgery or classify the surgical stages corresponding to each surgical site based on the surgical site during the specific surgery. Alternatively, the computer may classify the surgical steps based on the position of the camera or the moving range of the camera during the specific surgery, or the surgical steps based on the change (eg replacement) of the surgical tool during the specific surgery. You can also classify them. The computer may classify each operation stage according to a specific classification criterion, and then define in advance the operation operations performed in each classified operation stage.

In addition, the surgical steps constituting a particular surgery may be classified into a hierarchical structure. In one embodiment, certain surgical procedures may be classified step by step from the lowest hierarchy (level) to the highest hierarchy (level) to form a hierarchical structure. Here, the lowest layer is composed of the smallest units representing the surgical procedure, and may include a minimum operation operation having a meaning as one minimum operation. For example, the computer may recognize a surgical operation that represents one constant motion pattern as a minimum operation motion such as cutting, grabbing, moving, etc., and configure it as the lowest hierarchy (ie, the lowest surgical step). In addition, the computer may be configured in a higher layer by grouping at least one minimal surgery operation according to a specific classification criteria (eg, time course, surgical site, camera movement, change of surgical tool, etc.). For example, when each of the minimum surgery operations indicating grabbing, moving, cutting, and the like of a surgical tool is performed in a predetermined order, and the predetermined order is connected, the minimum operation can be grouped into one higher layer when it has meaning as a specific operation. have. As another example, when each minimal surgical operation such as clipping, moving, or cutting is performed continuously, it may be recognized that this is a surgical operation for cutting blood vessels and may be grouped into one higher layer. As another example, when each minimal surgical operation such as catching, lifting, cutting, or kicking is performed in succession, it may be recognized that it is a surgical operation to remove fat and grouped into one higher layer. As such, when it is recognized as an upper operation operation (ie, an upper operation step) having a specific meaning from lower operation procedures (that is, lower operation steps), it may be classified into a higher layer and this process may be repeated. It can be done with Thus, a hierarchical structure can be finally formed up to the highest hierarchy (ie, the highest surgical stage) of a specific surgical procedure. For example, a particular surgery may be represented in a hierarchical structure, such as a tree, of the surgical steps constituting the specific surgery.

Therefore, the surgical image obtained in step S100 is obtained by performing a specific surgery on the patient, and may include image frames photographing any one of the surgical steps belonging to a specific layer in a specific surgical process having a hierarchical structure. have. Here, the specific layer may mean a middle layer other than the lowest layer and the highest layer in the hierarchical structure.

In addition, the surgical image obtained in step S100 may be composed of image frames including a series of predefined surgical operations for a particular surgical step of a particular surgery. In one embodiment, when a specific surgery of the same type is performed on a plurality of patients, the computer may obtain each surgical image corresponding to a specific surgery stage predefined in a specific surgery procedure from each of the plurality of patients. . In other words, each of the surgical images obtained from the plurality of patients may be set to image frames including predetermined surgical operations performed at a specific surgical stage. As such, in setting image frames including predetermined surgical operations performed in a specific surgical step on a surgical image obtained from a patient, a medical staff or a computer extracts only image frames corresponding to a specific surgical step from the surgical image. It may be generated in advance.

Hereinafter, for convenience of description, a surgical image set with image frames including predetermined surgical operations performed at a specific surgical step is referred to as a predetermined surgical image.

The computer may generate the learning data using the surgery time obtained based on the preset surgery image and the preset surgery image acquired in step S100 (S200).

In an embodiment, the computer may acquire the first surgery time based on the predetermined surgery image. The computer may generate a censored surgical image from which a predetermined section is removed from the preset surgical image, and acquire a second surgery time based on the censored surgical image. The computer may generate, as learning data, at least one of a predetermined surgery image and a middle-cut surgery image based on the first surgery time and the second surgery time. A detailed process thereof will be described with reference to FIG. 3.

On the other hand, even if a specific surgical step is applied to a plurality of patients in the same way, the variability may occur depending on the state of each patient or the operation method of each medical staff to perform the operation. Therefore, when applying a specific surgical step for each patient, the actual operation time required to perform this may be different for each patient. Thus, even if the same surgery is performed for each patient, it is difficult to predict the accurate surgery time or the remaining surgery time. In order to solve this problem, the present invention obtains a surgical image including the surgical operations performed in a specific surgical stage, and configured the learning data to perform the learning to more accurately predict the operating time required in each surgical stage Provide a method. In addition, the present invention provides a method for estimating the remaining surgery time based on the surgery image of the current time point obtained during the actual surgery.

3 is a view for explaining a process of generating learning data based on a surgical image according to an embodiment of the present invention.

Referring to FIG. 3, the computer may acquire predetermined surgical images from a plurality of patients (first patient to nth patient), respectively.

In this case, each of the predetermined surgical images obtained from the plurality of patients, even if the same operation steps are performed according to the patient's condition or the operation method of the medical staff instructing the operation as described above, the image frames included in each surgical image is different There can be. In addition, it is not easy for a medical staff or a computer to accurately determine the surgical operations defined in a specific surgical step from the surgical image and extract only the image frames corresponding thereto. That is, each preset surgical image obtained from a plurality of patients is composed of image frames for the same operation stage, but these image frames are not only the operation operations defined in the operation stage but also other operation stages (eg, It may also include a surgical operation (before or after the surgical stage). In addition, even if it is composed of the surgical operations defined in the operation stage, due to various variability in the operation may include additional surgical operations in addition to the essential surgical operation. Such a surgical image alone is difficult to determine the operation time required for a particular surgical step, and also is not suitable for applying to all patients in a batch.

Thus, the present invention configures the learning data for learning based on a predetermined surgical image obtained from the patient, and by learning this to accurately predict the operation time in the corresponding surgical image (ie, the corresponding surgical stage).

In one embodiment, the computer may generate learning data by applying a survival analysis technique. That is, the computer may apply the survival analysis technique to analyze the time required for the operation based on the surgical image and generate the learning data based on this.

For example, first, the computer may generate a predetermined surgical image and a middle-cut surgical image in which random sections are removed therefrom (S210). Here, the intermediately cut surgical image may be an image from which at least one image frame corresponding to an arbitrary section is removed from the predetermined surgical image. As described above, the predetermined surgery image may include an image frame of another surgery stage (eg, before or after surgery), or may include an additional surgery movement that is not a required surgery in the surgery stage. . As such, since the predetermined surgical image may not be an image including only the essential surgical operations of the corresponding surgical step, the computer generates a middle-cut surgical image in which a predetermined section is removed from the predetermined surgical image. For example, the arbitrary section may be determined through a random function, or the section including the front or rear image frame of the surgical image may be determined as the random section. Alternatively, if there is an image frame that is determined to be a different operation from a series of operations by analyzing the image frames in the surgical image, or if there is an image frame that is determined to be different from a predetermined operation to be performed in a specific operation step, the corresponding section May be determined as an arbitrary interval.

For example, as shown in FIG. 3, the computer may generate the first to n-th predetermined surgical images and the first to n-th cutaway surgical images respectively obtained from the first to nth patients. In FIG. 3, the midway cut surgical images are represented by image frames included in a box indicated by a dotted line.

Next, the computer acquires an operation time required to perform a specific surgery step based on a predetermined surgery image (hereinafter, referred to as a first surgery time), and is required to perform a specific surgery step based on the intermediately cut surgery image. It is possible to obtain the operation time required (hereinafter, the second operation time) (S220).

For example, as shown in FIG. 3, the computer may include a first surgery time (eg, an actual duration of FIG. 3) of the first to n-th predetermined surgery images, and a second surgery to the first to n-th intermediate surgery images. A time duration (eg, Censored Duration of FIG. 3) may be obtained and compared. In this case, each operation time may be a time required when all the image frames in each surgical image is performed. For example, it may be a playback time of each surgical image.

Next, the computer may generate learning data using at least one of a predetermined surgery image and / or a mid-cut surgery image based on the first surgery time and the second surgery time (S230).

For example, as shown in FIG. 3, the computer may include a first surgery time of the first predetermined surgery image (eg, Actual Duration of FIG. 3) and a second surgery time of the first censored surgery image (eg, FIG. By comparing the Censored Duration of 3, it is possible to determine the surgical image to be used as learning data of the two surgical images for the first patient. The computer may generate the learning data by mapping the surgery image (eg, the first predetermined surgery image) determined by the learning data from the first patient and the surgery time for the same.

According to an embodiment, in determining a surgical image to be used as learning data by comparing the first surgery time with the second surgery time, the computer may use the length of the surgery time or may use a predetermined criterion. Or you may select arbitrarily. For example, the computer may select a surgical image having a shorter time between the first surgery time and the second surgery time. Alternatively, the difference between the first surgery time and the second surgery time may be used. For example, when the difference between the first surgery time and the second surgery time is more than a predetermined reference value, the computer may select the surgery image having the first surgery time as learning data.

As shown in FIG. 3, the computer is based on a result of comparing the first surgery time of the first to n-th predetermined surgery images and the second surgery time of the first to n-th intermediate cut surgery images, respectively. By selecting one of a predetermined surgical image or a middle-cut surgical image, it may be configured as the learning data for a specific surgical step by mapping the corresponding operation time. For example, when a predetermined surgical image is selected, the computer may set the surgical status as “event” and record the surgical duration of the selected surgical image as learning data. Alternatively, when a mid-severed surgical image is selected, the computer may set the surgical status to "censored" and record the surgical duration of the selected surgical image as learning data. That is, the computer may generate a learning data set by using a predetermined surgical image or a middle-cut surgical image obtained from a plurality of patients undergoing surgery for a specific surgical stage.

Referring back to FIG. 2, the computer may predict the operation time in a specific surgery step by performing the learning based on the learning data generated in step S200 (S300).

In one embodiment, the computer may perform the learning by using each image frame in the learning data and the operation time required to perform a specific surgery step based on each image frame. The computer can learn the average time required for surgery at a particular stage of surgery. For example, when the preset surgical image is generated as learning data, the computer may perform the learning based on each image frame in the preset surgical image and the first surgery time thereof. Alternatively, in the case where the severed surgical image is generated as the learning data, the computer may perform the learning based on each image frame in the severed surgical image and the second surgery time. A detailed process thereof will be described with reference to FIG. 4.

4 is a diagram illustrating a process of performing learning based on learning data according to an embodiment of the present invention.

Referring to FIG. 4, the computer may acquire learning data and perform the learning using deep learning. In this case, the learning data may be generated through the steps S210 to S230, and may be generated based on a predetermined surgical image or a middle-cut surgical image.

In an embodiment, the computer may receive the training data and extract each image frame L ₁ to L _N from the training data (S310).

For example, when the input training data is a preset surgical image, the computer may extract each image frame included in the preset surgical image from the input training data. Alternatively, when the received learning data is a middle-cut surgical image, the computer may extract each image frame included in the half-cut surgical image.

The computer may learn from the extracted image frames by using a convolutional neural network (CNN), and as a result, may extract feature information about each image frame (S320).

For example, the computer may learn characteristics of the surgical image by inputting each image frame to at least one layer (eg, a convolution layer). As a result of this learning, the computer can infer what each image frame represents or represents.

The computer may perform learning using a recurrent neural network (RNN) for each image frame derived as a result of learning using the CNN (S330).

For example, the computer may receive learning data in units of frames and perform learning using an RNN (eg, an LSTM method). At this time, the characteristic information of each image frame may be input, and the learning may be performed using the operation state information (eg, event / censored) of the corresponding learning data and the operation time required for the corresponding learning data. In addition, the computer may perform the learning by connecting the at least one image frame of the previous time point with the image frame of the current time point. Therefore, the computer learns the relationship between each image frame based on the characteristic information of each image frame, so as to determine whether the operation includes a certain operation stage predefined in a specific operation stage or whether the operation includes operation operation of another operation stage. I can grasp the information.

The computer may predict a median surgical duration at a specific surgical stage through the learning process using the CNN and the RNN as described above (S340).

For example, as described above with reference to FIG. 3, the computer acquires a surgical image from each of a plurality of patients who have performed a specific surgical step and generates learning data based thereon. You can apply the same learning repeatedly as described in 4. Therefore, the computer repeatedly learns a large amount of learning data, thereby predicting an average time required for surgery at a specific stage of surgery.

According to an embodiment of the present invention, by performing the above-described step S100 ~ S300 can finally predict the operation time required to perform each surgical step in the surgical procedure. Accordingly, the computer can build a learning model for predicting the operation time of each operation stage.

In addition, in one embodiment of the present invention can be applied to the surgical process of the other patient by using the estimated operating time in each operation stage as described above.

In one embodiment, if there is a patient currently undergoing a real surgery, the computer may obtain a real surgery image of the patient to be operated in real time. In this case, the computer may extract an image frame of the current point of time from the acquired actual surgical image, and recognize the operation stage currently being performed based on the extracted image frame of the current point of view. Therefore, the computer may calculate the time required for the surgery from the current surgery stage to the current time based on the image frame of the current time. In addition, the computer can grasp the average operation time for the current operation stage through learning to predict the operation time of each operation stage as described above. Therefore, the computer uses the average operation time of the current operation stage predicted through the learning and the operation time until the current time calculated through the image frame of the current time that the actual operation is currently being performed. It is possible to estimate the remaining surgical time required to perform the surgical operations in the current surgical stage.

Therefore, in the present invention, it is possible not only to predict the time required for surgery at each operation stage, but also to accurately predict the remaining time remaining after the present time. In this way, by providing the correct operation time and the remaining operation time to the medical staff can be more accurately understand the current situation of the operation is progressing through it can be efficiently carried out the residual surgery process.

In addition, the correct operation time and the remaining time of the operation must be known in order to proceed with the operation of the patient. In particular, the anesthesia time during the operation is important, in the present invention can provide the operation time and the remaining operation time in real time, it is possible to calculate the correct anesthetic dosage. In addition, it is effective to determine the appropriate additional anesthesia dose according to the remaining surgery time.

5 is a view schematically showing the configuration of an apparatus 200 for performing a method for predicting a surgery time based on a surgery image according to an embodiment of the present invention.

Referring to FIG. 5, the processor 210 may include a connection passage (eg, a bus or the like) that transmits and receives signals with one or more cores (not shown) and a graphic processor (not shown) and / or other components. ) May be included.

The processor 210 according to an embodiment executes one or more instructions stored in the memory 220 to perform a method of predicting a surgery time based on the surgery image described with reference to FIGS. 2 to 4.

For example, the processor 210 may acquire a predetermined surgery image including a surgery operation for a specific surgery stage by executing one or more instructions stored in the memory 220, the preset surgery image and the preset surgery image. Generating the learning data using the operation time obtained based on the step, and performing the learning based on the learning data may be performed to predict the operation time in the specific surgery step.

On the other hand, the processor 210 is a random access memory (RAM) and a ROM (Read-Only Memory) for temporarily and / or permanently storing signals (or data) processed in the processor 210. , Not shown) may be further included. In addition, the processor 210 may be implemented in the form of a system on chip (SoC) including at least one of a graphic processor, a RAM, and a ROM.

The memory 220 may store programs (one or more instructions) for processing and controlling the processor 210. Programs stored in the memory 220 may be divided into a plurality of modules according to their functions.

The method of estimating the operation time based on the surgical image according to the embodiment of the present invention described above may be implemented as a program (or an application) to be executed in combination with a computer which is hardware and stored in a medium.

The above-described program includes C, C ++, JAVA, machine language, etc. which can be read by the computer's processor (CPU) through the computer's device interface so that the computer reads the program and executes the methods implemented as the program. Code may be coded in the computer language of. Such code may include functional code associated with a function or the like that defines the necessary functions for executing the methods, and includes control procedures related to execution procedures necessary for the computer's processor to execute the functions according to a predetermined procedure. can do. In addition, the code may further include memory reference code for additional information or media required for the computer's processor to execute the functions at which location (address address) of the computer's internal or external memory should be referenced. have. Also, if the processor of the computer needs to communicate with any other computer or server remotely in order to execute the functions, the code may be used to communicate with any other computer or server remotely using the communication module of the computer. It may further include a communication related code for whether to communicate, what information or media should be transmitted and received during communication.

The stored medium is not a medium for storing data for a short time such as a register, a cache, a memory, but semi-permanently, and means a medium that can be read by the device. Specifically, examples of the storage medium include, but are not limited to, a ROM, a RAM, a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, and the like. That is, the program may be stored in various recording media on various servers to which the computer can access or various recording media on the computer of the user. The media may also be distributed over network coupled computer systems so that the computer readable code is stored in a distributed fashion.

The steps of a method or algorithm described in connection with an embodiment of the present invention may be implemented directly in hardware, in a software module executed by hardware, or by a combination thereof. The software module may include random access memory (RAM), read only memory (ROM), erasable programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory, hard disk, removable disk, CD-ROM, or It may reside in any form of computer readable recording medium well known in the art.

In the above, embodiments of the present invention have been described with reference to the accompanying drawings, but those skilled in the art to which the present invention pertains may implement the present invention in other specific forms without changing the technical spirit or essential features. I can understand that. Therefore, it should be understood that the embodiments described above are exemplary in all respects and not restrictive.

Claims (10)

In the method for predicting the operation time based on the operation image performed by the computer,
Acquiring a predetermined surgical image including a surgical operation for a specific surgical stage;
Generating learning data using the surgery time obtained based on the preset surgery image and the preset surgery image; And
Predicting the operation time in the specific surgical step by performing the learning based on the learning data,
Generating the learning data,
Acquiring a first surgery time based on the preset surgery image;
Generating a censored surgical image from which a predetermined section is removed from the preset surgical image, and obtaining a second surgery time based on the censored surgical image; And
Generating at least one of the predetermined surgical image and the intermediate-cut surgical image based on the first surgery time and the second surgery time as the learning data,
The preset surgical image,
Method for predicting the operation time based on the surgical image, characterized in that the preset image frames consisting of a predetermined operation for the particular surgery stage. delete The method of claim 1,
Predicting the operation time,
Performing learning using an operation time required to perform the specific surgery step based on each image frame and each image frame in the learning data; And
Predicting the operation time based on the surgical image, characterized in that it comprises the step of estimating the average time required for surgery in the specific surgery step through the learning. The method of claim 3,
Performing the learning,
When the predetermined surgical image is generated as the learning data, the operation time is predicted based on the surgical image, wherein the learning is performed based on each image frame in the predetermined surgical image and the first surgery time. How to. The method of claim 3,
Performing the learning,
In the case where the intermediate-cut surgical image is generated as the learning data, the operation time is performed based on the surgical image, wherein the learning is performed based on each image frame and the second operation time in the intermediate-cut surgical image. How to predict. The method of claim 3,
The method may further include acquiring an actual surgical image by performing the specific surgery step in the actual surgery process.
Predicting the operation time,
Acquiring operation time required up to the current point in time to perform the specific operation step based on the image frame of the current point in the actual surgery image; And
Estimating the remaining surgery time required to perform the specific surgery step after the current time point based on the average surgery time in the specific surgery step predicted through the learning and the surgery time to the present time point. Method for predicting the operation time based on the surgical image, characterized in that. The method of claim 1,
Acquiring the predetermined surgical image,
Method for predicting the operation time based on the surgical image, characterized in that to obtain a predetermined surgical image performing the specific surgery step from a plurality of patients, respectively. The method of claim 1,
The specific surgery stage,
Method for predicting the operation time based on the surgical image, characterized in that any one of the surgical steps belonging to a specific layer in the surgical process consisting of a hierarchical structure according to the operation. Memory for storing one or more instructions; And
A processor for executing the one or more instructions stored in the memory,
The processor executes the one or more instructions,
Acquiring a predetermined surgical image including a surgical operation for a specific surgical stage;
Generating learning data using the surgery time obtained based on the preset surgery image and the preset surgery image; And
Performing the learning based on the learning data to predict the operation time in the specific surgery step,
Generating the learning data,
Acquiring a first surgery time based on the preset surgery image;
Generating a censored surgical image from which a predetermined section is removed from the preset surgical image, and obtaining a second surgery time based on the censored surgical image; And
Generating at least one of the predetermined surgical image and the intermediate-cut surgical image based on the first surgery time and the second surgery time as the learning data,
The preset surgical image,
Apparatus, characterized in that for pre-set the image frame consisting of a predefined operation for the particular surgical step. A computer program, coupled to a computer, which is hardware, stored on a recording medium readable by a computer so as to perform the method of claim 1.

Images