KR102628324B1 - Device and method for analysing results of surgical through user interface based on artificial interlligence - Google Patents

Abstract

Disclosed is a method and server for providing surgical process analysis based on a user interface.
A method for analyzing a surgical process based on a user interface according to an embodiment includes the steps of recognizing individual surgical steps using a surgical step learning model in all captured surgical images and separating a time region corresponding to each surgical step. Calculating the number of switching times and time required for each recognized surgical step, determining whether a bleeding event has occurred for each recognized surgical step, and determining surgical results for each surgical step based on the number of switching, time required, and bleeding event. It may include providing a user interface capable of analysis.

Description

Device and method for analyzing surgical results through artificial intelligence-based user interface {DEVICE AND METHOD FOR ANALYSING RESULTS OF SURGICAL THROUGH USER INTERFACE BASED ON ARTIFICIAL INTERLLIGENCE}

The present invention relates to a method and system for analyzing surgical procedures after surgery. More specifically, this is a system that recognizes individual phases of postoperative images through a learning model, counts the switching results of surgical images for each recognized phase, the time required, and the number of bleeding occurrences and provides them to the user. and methods.

Generally, when doctors establish a patient's surgical plan before and after surgery, they refer to two-dimensional medical images such as the patient's CT (Computed Tomographic) or MRI (Magnetic Resonance Imaging) photos. In this case, it is difficult to match the location of the lesion inside the patient's organ or the relationship with the blood vessel according to the location of the lesion to the 2D medical image, and there is no means of utilizing organ imaging information before surgery, so the inside of the organ during surgery is not available. There are many limitations in identifying the location of the lesion and the distribution of surrounding blood vessels.

In addition, in order to design a scenario to optimize the surgical process, it was necessary to refer to medical images taken in advance or receive advice from a highly experienced doctor. There was a problem in getting the right advice. In other words, it was difficult to use simply captured medical images for optimization and evaluation of the surgical process.

Therefore, the surgical process is optimized by minimizing unnecessary processes when performing surgery using 3D medical images (e.g., virtual images of 3D surgical tool movement and changes inside organs that occur due to tool movement). And the development of a method that can provide surgical assistance information based on this is required.

Additionally, deep learning has recently been widely used in the analysis of medical images. Deep learning is defined as a set of machine learning algorithms that attempt high-level abstractions (summarizing key content or functions in large amounts of data or complex data) through a combination of several nonlinear transformation techniques. Deep learning can be viewed in the larger framework as a field of machine learning that teaches computers about human thinking.

Public Patent Publication No. 10-2019-0080736, 2019.07.08.

The problem that the present invention aims to solve is to recognize individual surgical stages using a surgical phase learning model in surgery result images, and to create a UI that displays the number of switching times, time required, and whether a bleeding event occurred for each recognized surgical stage. It is provided to users.

In addition, a method is disclosed to determine whether bleeding occurred in each surgical step recognized in the surgical result image and to determine whether a surgical error occurred by measuring the bleeding area and amount of bleeding.

The problems to be solved by the present invention are not limited to the problems mentioned above, and other problems not mentioned can be clearly understood by those skilled in the art from the description below.

The method of analyzing the surgical process after surgery according to an embodiment of the present invention to solve the above-described problem is to recognize individual surgical steps using a surgical step learning model in the entire captured surgical image and to identify individual surgical steps corresponding to each surgical step. Separating the time domain; Calculating the number of switching times and time required for each recognized surgical step; Determining whether a bleeding event has occurred for each of the recognized surgical steps; And it may include providing a user interface that can analyze the surgical results for each surgical stage based on the number of switching, time required, and bleeding event.

Additionally, the step of calculating the number of switching may include counting the number of times the recognized surgical step is disconnected without being able to continue.

In addition, the surgical step learning model is machine-learned by defining at least one step required for surgery of the object as a label using surgical data and inputting learning images for each defined label, and the surgical step learning model is The step of recognizing the object's surgical stage is to extract feature information from the surgical image based on deep learning-based learning using CNN (Convolutional neural network), and to recognize the surgical step based on the extracted feature information. there is.

In addition, the step of determining whether the bleeding event occurs includes determining whether a bleeding area exists in the surgical image of the recognized surgical step based on deep learning-based learning; Determining the location of the bleeding area based on the determination result; And it may include calculating the amount of bleeding based on the location of the bleeding area.

In addition, the step of determining whether the bleeding event has occurred determines whether the bleeding requires stopping the surgery based on the calculated amount of bleeding, and if there is no need to stop the surgery, it is recognized as the first bleeding event. , if it is necessary to stop the surgery, a step of recognizing it as a second bleeding event may be further included.

Additionally, whether it is the first bleeding event or the second appearance event can be determined based on the number of switching times at the time bleeding occurs and the time required for the surgical step.

The computer program according to one embodiment of the present invention can be combined with a computer as hardware and stored in a computer-readable recording medium to perform the surgical process analysis method.

A surgical process analysis system according to an embodiment of the present invention includes medical imaging equipment for capturing surgical images; A display unit for providing the user with analysis results of the surgical process; and one or more processors; and a control unit including one or more memories storing instructions that cause the one or more processors to perform an operation when executed by the one or more processors, and the operation performed by the control unit is performed on the entire surgical image captured. An operation to recognize individual surgical steps using a step learning model and separate the time domain corresponding to each surgical step; Calculating the number of switching times and time required for each recognized surgical step; An operation to determine whether a bleeding event has occurred for each of the recognized surgical steps; And it may include a calculation that provides a user interface that can analyze the surgical results for each surgical stage based on the number of switching, time required, and bleeding event.

Additionally, the operation for calculating the number of switching may include counting the number of times the recognized surgical step is disconnected without being able to continue.

In addition, the calculation for determining whether the bleeding event has occurred determines whether the bleeding requires stopping the surgery based on the calculated amount of bleeding, and if there is no need to stop the surgery, it is recognized as the first bleeding event. , a case in which it was necessary to stop the surgery may include an operation for recognizing a second bleeding event.

According to one embodiment of the present invention, analysis results corresponding to each surgical step in the surgery result image are provided to the user to help determine surgical errors and allow the user to optimize the surgical process through feedback. there is.

In particular, by providing a user-selectable UI that displays the number of switching, time required, and whether or not a bleeding event has occurred, specialized for each surgical stage, it increases the user's surgical proficiency and at the same time prevents the occurrence of unwanted bleeding or damage to organs during surgery. Evaluation results can be provided efficiently to reduce the likelihood.

The effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description below.

1 is a diagram illustrating a robotic surgery system according to an embodiment.
Figure 2 is a flowchart for explaining a surgical process analysis method according to an embodiment.
Figure 3 is a flowchart for explaining a surgical step learning model according to an embodiment.
Figure 4 shows an example of divided surgical steps according to an embodiment.
Figure 5 shows an example showing the number of switching times and time required for each surgical stage according to an embodiment.
Figure 6 is a flowchart for explaining a method of evaluating bleeding for each surgical step according to an embodiment.
Figure 7 shows an example of displaying bleeding events for each surgical stage according to an embodiment.
Figure 8 is an example showing the results of analysis of a surgical procedure according to an embodiment.

The advantages and features of the present invention and methods for achieving them will become clear by referring to the embodiments described in detail below along with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below and may be implemented in various different forms. The present embodiments are merely provided to ensure that the disclosure of the present invention is complete and to provide a general understanding of the technical field to which the present invention pertains. It is provided to fully inform the skilled person of the scope of the present invention, and the present invention is only defined by the scope of the claims.

The terminology used herein is for describing embodiments and is not intended to limit the invention. As used herein, singular forms also include plural forms, unless specifically stated otherwise in the context. As used in the specification, “comprises” and/or “comprising” does not exclude the presence or addition of one or more other elements in addition to the mentioned elements. Like reference numerals refer to like elements throughout the specification, and “and/or” includes each and every combination of one or more of the referenced elements. Although “first”, “second”, etc. are used to describe various components, these components are of course not limited by these terms. These terms are merely used to distinguish one component from another. Therefore, it goes without saying that the first component mentioned below may also be a second component within the technical spirit of the present invention.

Unless otherwise defined, all terms (including technical and scientific terms) used in this specification may be used with meanings commonly understood by those skilled in the art to which the present invention pertains. Additionally, terms defined in commonly used dictionaries are not interpreted ideally or excessively unless clearly specifically defined.

In this specification, “image” may mean multi-dimensional data composed of discrete image elements (eg, pixels in a 2-dimensional image and voxels in a 3D image). For example, the image may include a medical image of an object acquired by a CT imaging device.

In this specification, “object” may be a person or an animal, or a part or all of a person or an animal. For example, the object may include at least one of organs such as the liver, heart, uterus, brain, breast, abdomen, and blood vessels.

In this specification, a “user” is a medical professional and may be a doctor, nurse, clinical pathologist, medical imaging expert, etc., and may be a technician who repairs a medical device, but is not limited thereto.

In this specification, “medical image data” refers to medical images captured with medical imaging equipment, and includes all medical images that can be implemented as a three-dimensional model of the subject's body. “Medical imaging data” may include computed tomography (CT) images, magnetic resonance imaging (MRI), and positron emission tomography (PET) images.

In this specification, “virtual body model” refers to a model created to match the actual patient's body based on medical image data. The “virtual body model” may be created by modeling medical image data as is in 3D, or it may be modeled and then corrected to resemble the actual surgery.

In this specification, “virtual surgery data” means data including rehearsal or simulation actions performed on a virtual body model. “Virtual surgery data” may be image data that has been rehearsed or simulated for a virtual body model in a virtual space, or may be data recorded about surgical operations performed on a virtual body model. Additionally, “virtual surgery data” may include learning data for training a surgery learning model.

In this specification, “actual surgical data” refers to data obtained as actual medical staff performs surgery. “Surgical data” may be image data taken of the surgical area during an actual surgical procedure, or may be data recorded about surgical movements performed during an actual surgical procedure.

In this specification, surgical phase refers to the basic steps performed sequentially in the overall surgery of a specific surgical type.

In this specification, “computer” includes all various devices that can perform computational processing and provide results to the user. For example, computers include not only desktop PCs and laptops (Note Books), but also smart phones, tablet PCs, cellular phones, PCS phones (Personal Communication Service phones), and synchronous/asynchronous computers. This may also include IMT-2000 (International Mobile Telecommunication-2000) mobile terminals, Palm Personal Computers (Palm PCs), and personal digital assistants (PDAs). Additionally, if a Head Mounted Display (HMD) device includes computing capabilities, the HMD device may be a computer. Additionally, a computer may be a server that receives requests from clients and performs information processing.

Hereinafter, embodiments of the present invention will be described in detail with reference to the attached drawings.

1 is a diagram illustrating a robotic surgery system according to an embodiment.

Referring to FIG. 1, a schematic diagram of a system capable of performing robotic surgery according to one embodiment is shown.

According to Figure 1, the robotic surgery system includes medical imaging equipment 10, a server 20, a control unit 30 provided in the operating room, an image capturing unit 36, a display 32, and a surgical robot 34. do. Depending on the embodiment, the medical image recording equipment 10 may be omitted from the robotic surgery system according to the embodiment.

In one embodiment, robotic surgery is performed by a user controlling the surgical robot 34 using the control unit 30. In one embodiment, robotic surgery may be automatically performed by the control unit 30 without user control.

The server 20 is a computing device that includes at least one processor, memory, and communication unit.

The control unit 30 includes a computing device including at least one processor, memory, and communication unit. In one embodiment, the control unit 30 includes hardware and software interfaces for controlling the surgical robot 34.

The image capture unit 36 includes at least one image sensor. That is, the image capture unit 36 includes at least one camera device and is used to photograph the surgical site. In one embodiment, the image capture unit 36 is used in combination with the surgical robot 34. For example, the image capture unit 36 may include at least one camera coupled to the surgical arm of the surgical robot 34.

In one embodiment, the image captured by the image capture unit 36 is displayed on the display 340.

The control unit 30 receives information necessary for surgery from the server 20, or generates information necessary for surgery and provides it to the user. For example, the control unit 30 displays generated or received information necessary for surgery on the display 32.

For example, the user performs robotic surgery by controlling the movement of the surgical robot 34 by manipulating the control unit 30 while looking at the display 32.

The server 20 generates information necessary for robotic surgery using medical image data of a subject (patient) previously imaged by the medical imaging equipment 10, and provides the generated information to the control unit 30.

The control unit 30 provides information received from the server 20 to the user by displaying it on the display 32, or controls the surgical robot 34 using the information received from the server 20.

In one embodiment, the means that can be used in the medical imaging equipment 10 are not limited, and for example, various other medical image acquisition means such as CT, X-ray, PET, and MRI can be used.

Below, for convenience of explanation, each step is described as being performed by a “computer,” but the subject performing each step is not limited to a specific device, and all or part of it is performed by the server 20 or the control unit 30. It can be done.

In one embodiment, surgical images captured by the medical imaging equipment 10 may be divided based on various criteria. As an example, a surgical image may be segmented based on the type of object included in the image. Segmentation methods based on the type of object require a computer to recognize each object.

Objects recognized in surgical images largely include the human body, objects introduced from the outside, and objects created independently. The human body includes body parts that are imaged by medical imaging (eg, CT) preceding surgery and body parts that are not imaged.

For example, body parts captured through medical imaging include organs, blood vessels, bones, tendons, etc., and these body parts can be recognized based on 3D modeling images generated based on medical imaging.

Specifically, the location, size, and shape of each body part are recognized in advance using a 3D analysis method based on medical images. The computer defines an algorithm that can determine the location of body parts corresponding to the surgical image in real time, and based on this, information on the location, size, and shape of each body part included in the surgical image without performing separate image recognition. can be obtained. Additionally, body parts that are not captured by medical imaging include the omentum, etc. Since these are not captured by medical imaging, they need to be recognized in real time during surgery. For example, the computer can determine the location and size of the omentum through an image recognition method, and if there is a blood vessel inside the omentum, the location of the blood vessel can also be predicted.

Objects brought in from outside include, for example, surgical tools, gauze, clips, etc. Because it has preset morphological characteristics, the computer can recognize it in real time through image analysis during surgery.

Objects created internally include, for example, bleeding occurring in body parts. This can be recognized in real time by a computer through image analysis during surgery.

The movement of organs or momentum contained in body parts, and the cause of objects being created internally, are all caused by the movement of objects introduced from the outside.

Therefore, in addition to recognizing each object, the surgical image can be divided into several surgical phases based on the movement of each object. In one embodiment, the surgical image may be divided based on the movement of an object introduced from the outside, that is, the action.

The computer determines the type of each object recognized in the surgical image, and determines the movement of each object based on a specific action, series of actions, and situations or results that occur according to the action, etc., predefined according to the type of each object. Actions can be recognized.

The computer can recognize the type of each action and further recognize the cause of each action. The computer can segment the surgical image based on the recognized actions, and can recognize each detailed surgical action to the type of overall surgery through step-by-step segmentation. For example, the computer extracts feature information from the captured surgical images based on a surgical step learning model that performs machine learning using a convolutional neural network, and analyzes images for each surgical step based on the feature information. You can segment it or recognize what surgical stage it is currently in.

Furthermore, the computer can determine the type of predefined surgery corresponding to the surgery image from the judgment of the action. When determining the type of surgery, information about the entire surgical process can be obtained. When a plurality of surgical processes exist for the same type of surgery, one surgical process may be selected according to the doctor's selection or based on actions recognized up to a specific point in time.

The computer can recognize and predict surgical steps based on the acquired surgical process. For example, when a specific step in a series of surgical processes is recognized, subsequent steps can be predicted or candidates for possible steps can be selected. Therefore, the error rate in surgical image recognition caused by omentum, etc. can be greatly reduced. Additionally, if the surgical image deviates significantly from the predictable surgical steps by more than a predetermined error range, it may be recognized that a surgical error situation has occurred. For example, if surgical step switching that deviates from the designated surgical process occurs frequently, it may be recognized that a surgical error situation has occurred.

In addition, based on image recognition of the surgical steps, the computer extracts and provides navigation information to the user about the major blood vessels corresponding to each surgical step and the blood vessels branching off from the major blood vessels according to blood flow, thereby assisting in effective surgery. can do.

Additionally, the computer can determine whether bleeding occurred due to a surgical error based on image recognition of the surgical stage.

Specifically, the computer can determine the location, time, and scale of each bleeding. Additionally, it is possible to determine whether the surgery should be stopped due to bleeding. Therefore, according to one embodiment, the computer can be used to provide data on error situations and bleeding situations as a surgical result report, exclude unnecessary movements or mistakes during the surgical process, and streamline the surgical process.

According to one embodiment, in order to provide a method for analyzing a surgical process after surgery, operations performed on a computer include: an operation for recognizing individual surgical steps in a captured surgical image using a surgical step learning model; Calculating the number of switching times and time required for each recognized surgical step; An operation to determine whether a bleeding event has occurred for each of the recognized surgical steps; And it may include an operation that provides a user interface that can analyze the surgical results for each surgical stage based on the number of switching, time required, and bleeding event.

Additionally, the operation for calculating the number of switching may include counting the number of times the recognized surgical step is disconnected without being able to continue.

In addition, the calculation for determining whether a bleeding event has occurred determines whether it is bleeding that requires stopping the surgery based on the calculated amount of bleeding, and if there is no need to stop the surgery, it is recognized as the first bleeding event, A case in which it is necessary to stop the surgery may include an operation for recognizing a second bleeding event.

Below, the method of analyzing the postoperative surgical process will be described in more detail with reference to the drawings.

Figure 2 is a flowchart for explaining a method of analyzing a surgical process after surgery according to an embodiment.

Each step shown in FIG. 2 is performed in time series by the server 20 or the control unit 30 shown in FIG. 1. Below, for convenience of explanation, each step is described as being performed by a “computer,” but the subject performing each step is not limited to a specific device, and all or part of it is performed by the server 20 or the control unit 30. It can be done.

In step S200, the computer according to one embodiment recognizes individual surgical steps using a surgical step learning model from all captured surgical images and separates time regions corresponding to each surgical step.

For example, using surgical data, at least one step required for surgery on the subject is defined as a label, and training images are input for each defined label to perform deep learning-based learning using a CNN (Convolutional neural network). , Feature information can be extracted from the surgical image from the learned surgical step learning model, and the surgical step can be recognized based on the extracted feature information.

For example, Figure 3 is a flowchart for explaining a surgical step learning model according to one embodiment.

Referring to Figure 3, in step S300, the computer selects the type of surgery. The type of surgery may be selected automatically by recognizing an object or object in the input image, or may be entered directly by the user.

Next, in step S310, the computer uses the surgery data to define the surgery with step-by-step labels.

Each surgical phase can be automatically divided by recognizing images of camera movements, movements of equipment such as robot arms, and organs, and the user can classify images that fit the standardized surgical process into learning images and learn them in advance. You can also do it.

For example, referring to Figure 4, when performing surgery on stomach cancer, approximately 21 surgical steps can be defined as shown in the table 400 of Figure 4.

Next, in step S320, the computer selects a learning image corresponding to the defined label. Learning images can be selected by automatically dividing the surgical images input by the computer, or the user can select and input learning images based on actual surgery data in advance.

Next, in step S330, the computer can perform machine learning using a convolutional neural network based on the selected training images. For example, if the frames of surgical images for each surgical stage defined in this way are learned using the Low-Fast-Net model, a highly accurate surgical stage learning model can be trained.

In this way, the trained surgical step learning model can be used to more accurately recognize surgical steps in surgical images to be analyzed after surgery.

For example, each surgical step can be segmented from the entire captured surgical image by recognizing individual surgical steps using a surgical step learning model and separating the time domain corresponding to each surgical step. there is.

Again in step S210 of Figure 2, the computer according to one embodiment calculates the number of switching and the time required for each recognized surgical step.

For example, if 21 surgical steps are defined for stomach cancer surgery, it is ideal for each step to proceed sequentially. However, it can be assumed that the reason these surgical steps are not connected and disconnected is because a mistake or error occurred during the surgery. Therefore, if surgical step switching that deviates from the designated surgical process occurs frequently or if a specific surgical step takes a relatively long time, it can be recognized that a surgical error situation has occurred.

For example, Figure 5 shows an example of the number of switching times and time required for each surgical step according to an embodiment of the workflow 500.

Referring to FIG. 5, it can be seen that in surgical stage 5 (for example, a stage in which partial omentectomy is required in gastric cancer surgery), the stages are not continuous and at least six switching times occur to proceed to another stage.

In the case of surgical stage 5 in gastric cancer surgery, a cutting machine called harmonic is generally used, but it can be assumed that switching occurred because an event such as the cutting machine being momentarily not recognized on the screen or the screen moving to another equipment occurred. You can. Therefore, it can be predicted that a surgical error such as bleeding occurred in surgical step 5, and the computer can include this in the surgical process analysis results and provide them to the user.

Referring again to FIG. 2, in step S220, the computer according to one embodiment determines whether a bleeding event has occurred for each recognized surgical step.

According to one embodiment, the computer can determine the location, time, and scale of each bleeding. Additionally, it is possible to determine whether there was interference with the surgery due to bleeding.

For example, Figure 6 is a flowchart for explaining a method of evaluating bleeding for each surgical step according to an embodiment.

Referring to FIG. 6, the computer can recognize whether an internal bleeding area exists on a frame-by-frame basis in the entire surgical image captured in step S100 based on deep learning-based learning (S600).

In one embodiment, the computer learns in advance to recognize whether each frame of the entire surgical image acquired in step S600 is a bleeding image containing a bleeding area (that is, to recognize whether a bleeding area exists in the surgical image). A learning model can be used.

At this time, when using a pre-trained learning model, the computer can first perform learning to recognize the presence or absence of a bleeding area in the surgical image using deep learning based on the surgical image dataset. The surgical image dataset may be a learning dataset in which labeling of surgical images is performed using various learning methods. For example, a surgical image dataset may be data learned using machine learning methods such as supervised learning, unsupervised learning, and reinforcement learning. Therefore, the computer acquires a surgical image dataset as learning data, and uses this to perform learning to recognize the presence or absence of a bleeding part (i.e., bleeding area) in the surgical image to create a learning model (e.g., a bleeding presence or absence recognition model). can be built in advance. At this time, the computer may build and store a learning model learned based on the surgical image dataset in advance, or may use a learning model built in another device.

Afterwards, when the computer acquires a new surgical image (i.e., the surgical image in step S600), the bleeding area in the new surgical image is identified using a learning model (e.g., bleeding presence/absence recognition model) learned based on the surgical image dataset. You can recognize whether it exists or not. In other words, the computer can quickly and effectively determine whether a newly acquired surgical image is a bleeding image containing a bleeding area or a non-bleeding image through a deep learning-based pre-built learning model (e.g., bleeding presence/absence recognition model). there is.

The computer can estimate the location of the bleeding area from the surgical image based on the surgical image recognition result of step S610 (that is, the recognition result of whether the surgical image is a bleeding image or a non-bleeding image) (S620).

In one embodiment, when the computer recognizes that the bleeding image has a bleeding area within the surgical image based on deep learning-based learning, it can specify the bleeding area within the surgical image and estimate the location of the specified bleeding area. . For example, a computer can extract feature information from a surgical image through deep learning-based learning, and recognize and specify the bleeding area within the surgical image based on the extracted feature information. At this time, the feature information is information representing the characteristics of the bleeding, and texture information such as color, shape, and texture that can specify the bleeding can be used.

The computer may calculate the amount of bleeding in the bleeding area based on the location of the bleeding area estimated in step S620 (S630).

In one embodiment, the computer can calculate the amount of bleeding using pixel information of the bleeding area in the surgical image. Alternatively, if the surgical image is a stereoscopic image, the computer obtains depth information of the bleeding area in the surgical image based on the depth map of the surgical image, and based on this, estimates the volume corresponding to the bleeding area to determine the amount of bleeding. can be calculated. Alternatively, if gauze is included in the surgical image, the computer can calculate the amount of bleeding in the bleeding area using gauze information.

In addition, the computer determines whether there is bleeding that interferes with the surgery based on the location of the bleeding area in the surgical image and the calculated amount of bleeding. If there is no intervention in the surgery, it recognizes it as the first bleeding event and continues the surgery. If there is interference, it can be recognized as a second bleeding event. For example, the first bleeding event may refer to bleeding that does not affect the surgery, such as a scratch, and the second bleeding event may refer to bleeding that requires stopping the surgery, such as damage to an artery. Additionally, the computer may recognize a case in which bleeding collected tissue or bleeding stained tissue is discovered as a second bleeding event. Alternatively, the case where blood-stained tissue is recognized may be defined as a third bleeding event, and the case where blood-stained tissue is recognized may be defined as the fourth bleeding event.

As described above, the surgical image is divided into surgical stages as it is input into the surgical stage recognition model, and as the surgical image is input into the bleeding recognition model, frames in which the first and second bleeding events exist in the surgical image The area within the frame is extracted. Then, the number of switching and the time required are calculated based on the surgical stage segmentation results, and the number of bleeding events within each surgical stage is calculated by linking the segmentation results of each surgical stage and the bleeding recognition results.

In other words, the above-described surgical stage recognition and bleeding recognition can be performed in parallel.

Meanwhile, whether it is the first bleeding event or the second appearance event can be determined based on the number of switches at the time bleeding occurs and the time required for the surgical step.

Figure 7 shows an example of displaying bleeding events for each surgical stage according to an embodiment.

Referring to FIG. 7, the step 701 in which bleeding does not occur, the step 703 in which the first bleeding event occurs, and the step in which the second bleeding event 702 occurs can be schematized and displayed in a highly visible form. Of course, it is not necessarily limited to this configuration, and the occurrence and duration of bleeding events for each surgical stage can be shown in various forms such as a linear graph, bar graph, or circular graph.

For example, in Figure 7, it can be determined that the second bleeding event occurred the longest in surgical stage 5 and surgical stage 15.

Referring again to Figure 2, in step S230, the computer according to one embodiment provides a user interface that can analyze the surgical results for each surgical step based on the number of switching, time required, and bleeding event.

Therefore, the user can receive data on error situations and bleeding situations at each stage of surgery as a surgical result report and use it to exclude unnecessary movements or mistakes during the surgical process and streamline the surgical process.

Figure 8 is an example showing the results of analysis of a surgical procedure according to an embodiment.

Referring to FIG. 8, the user interface 800 according to one embodiment includes a video playback area 801, an area 802 that displays a summary of the entire surgical procedure, and when the video is played, analysis data for the corresponding part is exposed. It may include an area 803 that provides video-related additional functions and an area 804 that provides video-related additional functions.

First, in the video playback area 801, the video for the selected surgical step can be played at 0.5 to 7 times the speed. Additionally, the surgical instrument and bleeding point can be recognized to indicate whether a first bleeding event or a second bleeding event has occurred.

In the area 802 that displays a summary of the entire surgical process, the portion for each surgical step in the total surgical time can be displayed in a predetermined color. For example, if a user selects a specific surgical step, only the surgical process at that step can be viewed.

Additionally, bleeding points can be displayed throughout the entire surgical time, and a video of that portion can be played based on the user's selection.

When a video is played, in an area 803 that exposes analysis data for the corresponding portion, the user may be shown graphs related to the number of switching times, time required, and bleeding events, as well as respective analysis results.

In the area 804 that provides video-related additional functions, a video clip function that saves only the portion desired by the user may be provided.

Additionally, according to one embodiment of the present invention, the surgical process analysis results can be provided to the user in the form of a report.

For example, as shown in the charts shown in FIGS. 5 and 7, analysis results that the user can check through the user interface 800 can be automatically analyzed and provided in the form of a report.

The steps of the method or algorithm described in connection with embodiments of the present invention may be implemented directly in hardware, implemented as a software module executed by hardware, or a combination thereof. The software module may be RAM (Random Access Memory), ROM (Read Only Memory), EPROM (Erasable Programmable ROM), EEPROM (Electrically Erasable Programmable ROM), Flash Memory, hard disk, removable disk, CD-ROM, or It may reside on any type of computer-readable recording medium well known in the art to which the present invention pertains.

Above, embodiments of the present invention have been described with reference to the attached drawings, but those skilled in the art will understand that the present invention can be implemented in other specific forms without changing its technical idea or essential features. You will be able to understand it. Therefore, the embodiments described above should be understood in all respects as illustrative and not restrictive.

10: Medical imaging equipment
20: Server
30: control unit
32: display
34: Surgical robot
36: Video recording department

Claims (10)

In a surgical result analysis device using an artificial intelligence-based user interface,
display; and
Includes a control unit,
The control unit,
Based on the actions of objects in the entire surgical video captured, individual surgical steps are recognized, the type of surgery is determined, and information about the entire surgical process is obtained.
Calculates surgery-related information for each recognized surgical stage,
Outputting a UI (User Interface) that can analyze surgical results for each surgical stage based on the surgery-related information on the display,
The surgery-related information includes information on at least one of the number of switching times for each surgical stage, the time required, and whether a bleeding event occurs,
The control unit,
If there are more than a predetermined number of switching operations that deviate from the established surgical process, it is recognized as a surgical error situation,
A device that extracts and provides navigation information about blood vessels corresponding to each surgical step. According to paragraph 1,
The control unit,
A device that calculates the number of switching, time required, and bleeding events for each recognized surgical step. According to paragraph 1,
The control unit,
When the surgery-related information includes information about the number of switching, the device calculates the number of switching by counting the number of times the recognized surgical step is not continued and is disconnected. According to paragraph 1,
The control unit recognizes the individual surgical steps from the entire surgical image using a surgical step learning model,
The surgical step learning model is a device that uses surgical data to define at least one step required for surgery on an object as a label, and is machine-learned by inputting learning images for each defined label. According to paragraph 4,
The control unit,
When recognizing at least one step required for surgery of the object using the surgical step learning model, feature information is extracted from the surgical image based on deep learning-based learning using CNN (Convolutional neural network), and the extraction An apparatus for recognizing the at least one step based on characteristic information. According to paragraph 1,
The control unit,
Separate time regions corresponding to each of the recognized individual surgical steps,
When determining whether the bleeding event has occurred, determine whether the bleeding event has occurred on a frame-by-frame basis in the entire surgical video captured based on deep learning-based learning,
A device that connects whether the bleeding event occurs to the recognition result of the surgical step using the separated time domain. According to paragraph 1,
The control unit,
When the surgery-related information includes information about whether the bleeding event has occurred, when determining whether the bleeding event has occurred, determine whether the bleeding interferes with the surgery based on the calculated bleeding amount,
Cases in which there was no interference with the surgery are recognized as the first bleeding event.
A device that recognizes a second bleeding event if there is interference with the surgery. In clause 7,
The control unit,
A device that determines whether the bleeding event is the first bleeding event or the second appearing event based on the number of switching at the time bleeding occurs and the time required for the surgical step. According to paragraph 1,
The UI is,
A first area where images for the selected surgical step are played,
a second area displaying a summary of the entire surgical procedure; and
The device includes at least one of a third area that displays analysis data of a corresponding portion of the area reproduced in the first area. In a method performed by the control unit of the device,
Recognizing individual surgical steps based on the actions of objects in the entire surgical image captured, determining the type of surgery, and obtaining information about the entire surgical process;
Calculating surgery-related information for each recognized surgical step; and
Comprising the step of providing a UI (User Interface) that can analyze the surgical results for each surgical step based on the surgery-related information,
The surgery-related information includes information on at least one of the number of switching times for each surgical stage, the time required, and whether a bleeding event occurs,
The control unit,
If there are more than a predetermined number of switching operations that deviate from the established surgical process, it is recognized as a surgical error situation,
A method of analyzing surgical results through an artificial intelligence-based user interface that extracts and provides navigation information about blood vessels corresponding to each surgical step.