KR102655325B1 - Apparatus and method for collecting learning data for training of surgical tool and organ recognition - Google Patents

Abstract

A learning data collection method and program for surgical tool recognition learning training are provided. The learning data collection method for the surgical tool recognition training is a method executed by a computer, comprising: the computer acquiring at least a plurality of surgical data among actual surgical data, virtual surgical data, and random surgical data; And comprising the step of configuring and providing the plurality of acquired surgical data into a learning data set.

Description

Learning data collection device and method for surgical tool and organ recognition learning training {APPARATUS AND METHOD FOR COLLECTING LEARNING DATA FOR TRAINING OF SURGICAL TOOL AND ORGAN RECOGNITION}

The present invention relates to a learning data collection device and method for surgical tool and organ recognition learning training.

In recent surgical operations, the proportion of open surgeries has decreased significantly, and the proportion of minimally invasive surgeries, which can improve recovery speed by minimizing patient complications, is greatly increasing.

Among these, the rate of minimally invasive surgery using robots is increasing rapidly. Additionally, due to the nature of laparoscopic surgery, in which a specialist performs the surgery while viewing images input from a camera, in minimally invasive surgery, all surgical procedures that occur inside the patient are recorded on video.

Therefore, if the surgical process is recognized by considering the surgical video as a kind of surgical record, not only evaluation and analysis of the surgery, but also real-time guidance during the surgery, as well as automated surgery can be possible.

In order to analyze and evaluate automated surgical procedures using surgical images, position measurement, movement recognition, and organ recognition for surgical tools are essential. Surgical data, that is, the surgical process, position measurement and movement of surgical tools, recognition of organs, etc., can be acquired and used as learning data.

Republic of Korea Patent Publication No. 10-2013-0100758 (2013.09.11)

The problem that the present invention aims to solve is to provide learning data of surgical tools and organs for accurate surgical tool and organ recognition learning.

Additionally, the problem that the present invention aims to solve is to provide learning data that can solve class imbalance.

The problems to be solved by the present invention are not limited to the problems mentioned above, and other problems not mentioned can be clearly understood by those skilled in the art from the description below.

The learning data collection method for surgical tool and organ recognition training according to an embodiment of the present invention to solve the above-described problem is a method executed by a computer, where the computer uses actual surgical data, virtual surgical data, and random surgical data. It includes acquiring at least a plurality of surgical data and configuring the acquired plurality of surgical data into a learning data set, wherein the actual surgical data includes the type of surgery actually performed and the location of the tool used. It includes actual surgical tool data including information on the surgical tools used and actual surgical long-term data about the organs of the surgery actually performed, and the virtual surgical data is used in a virtual surgical environment and a virtual surgical environment. It includes virtual surgical tool data and virtual surgical organ data for organs in a virtual surgical environment, and the random surgical data may be virtual data generated from domain randomization.

The learning data collection method for surgical tool recognition learning training according to an embodiment of the present invention to solve the above-described problem involves surgical tool recognition learning training for a specific surgical tool used for a specific surgery based on the learning data set. It may further include performing and generating a surgical tool recognition learning model based on the surgical tool recognition learning training performed above.

The actual surgical tool data is bounded within the actual surgical image by distinguishing at least one of the head, wrist, and body of the tool, or processed as a polygon on a pixel basis, or the tool's Each feature may include at least one of bounding within the actual surgical image or processing the polygon on a pixel basis.

The virtual surgical environment is a 3D surgical environment created by turning it into 3D using specific actual surgical image data, and the virtual surgical tool data and the virtual surgical organ data are derived by being utilized by the user within the 3D surgical environment. It may be virtual surgical tools and surgical organ information.

The random surgery data is virtual data generated from domain randomization, and is randomly generated by randomly changing the setting conditions of surgical tools, surgical target organs, and cameras, and randomly generated specific surgery, specific surgical target organs, and It may include data on at least one of the random surgical tools used in a particular surgery.

The learning data may be obtained by maintaining class balance as a uniform amount of data of a specific surgical tool and a specific surgical organ for a specific surgery.

In addition, the present invention further includes the step of generating semantic image data for the obtained surgical data through a segmentation model after acquiring the surgical data, and the step of providing the learning data set configuration includes the step of generating semantic image data for the obtained surgical data through a segmentation model, and the step of providing the learning data set includes the plurality of surgical data. Data and the semantic image data corresponding to each of the plurality of surgical data may be provided by configuring a learning data set.

Here, in the semantic image data generating step, when the plurality of surgical data includes the actual surgical data, actual semantic image data is generated based on the actual surgical data through the segmentation model, and the plurality of surgical data If the virtual surgery data is included in the virtual surgery data, virtual semantic image data is generated based on the virtual surgery data through the segmentation model, and if the random surgery data is included in the plurality of surgery data, through the segmentation model. Random semantic image data can be generated based on the random surgery data.

In addition, the semantic image data includes semantic surgical tool data for the tool or semantic surgical long-term data for the organ included in each of the plurality of surgical data, and the semantic surgical tool data or the semantic surgical long-term data , a location range of the tool or the organ, and a name of the tool or the organ.

The learning data collection method for surgical tool recognition training according to an embodiment of the present invention to solve the above-described problem is a method executed by a computer, comprising the step of the computer acquiring actual surgical data. Extracting the type and required amount of data lacking annotations that cause class imbalance, the required amount corresponding to the type of the extracted data from at least one of virtual surgery data and random surgery data. A step of acquiring data as balanced data, wherein the actual surgical data includes actual surgical tool data including the type of surgery actually performed, location information of the tool used, and information on the surgical tool used, The virtual surgery data includes a virtual surgery environment and virtual surgical tool data used in the virtual surgery environment, and the random surgery data may be virtual data generated from domain randomization.

A learning data collection method for surgical tool recognition training according to an embodiment of the present invention to solve the above-described problem is a specific surgical tool used for a specific surgery using the obtained balance data and the actual surgical data as learning data. It may include performing surgical tool recognition learning training and generating a surgical tool recognition learning model based on the performed surgical tool recognition learning training.

When the specific surgery is gastrectomy, the specific surgical tools include at least one of robotic surgical tools, laparoscopic surgical tools, and auxiliary surgical tools, and the specific surgical organs include the stomach, liver, gallbladder, spleen, and pancreas. And when the specific surgery is cholecystectomy, the specific surgical tool may include at least one of a laparoscopic tool and an auxiliary surgical tool, and the specific surgical organ may include a gallbladder and a liver.

A learning data collection program for surgical tool recognition training according to another embodiment of the present invention to solve the above-described problem may be combined with a computer as hardware and stored to execute any one of the above methods.

Other specific details of the invention are included in the detailed description and drawings.

According to the present invention, class imbalance can be resolved by obtaining data that is difficult to obtain from actual surgery data as virtual data.

In addition, according to the present invention, a more accurate learning model can be created by generating a large amount of various repetitive data and utilizing this data.

The effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description below.

FIG. 1A is a diagram illustrating a device 10 for collecting learning data for surgical tool and organ recognition training according to the present invention.
Figure 1B is a flowchart of a learning data collection method for surgical tool and organ recognition training according to an embodiment of the present invention.
Figure 2a is a diagram illustrating an annotation of surgical tools and organs in an image according to the present invention.
Figure 2b is a diagram to explain generating actual semantic image data based on actual surgical data through a segmentation model according to the present invention.
Figure 2c is a diagram illustrating the creation of virtual semantic image data based on virtual surgical data through a segmentation model according to the present invention.
FIG. 2D is a diagram illustrating generating random semantic image data based on random surgical data through a segmentation model according to the present invention.
Figure 3 is a flowchart of a method for generating a surgical tool recognition learning model using a learning data set according to the present invention.
Figure 4 is a flowchart of a method for obtaining insufficient data in actual surgery data as balanced data according to the present invention.
Figure 5 is a graph showing the distribution of annotations obtained from actual surgery data according to the present invention.
Figure 6 is a graph showing the distribution of annotations changed by acquiring virtual surgery data and/or random surgery data according to the present invention.
Figure 7 is a graph explaining the distribution of annotations obtained from actual surgery data (laparoscopic cholecystectomy and stomach cancer video) according to the present invention.
Figure 8 is a graph illustrating the distribution of annotations obtained from laparoscopic cholecystectomy and gastrectomy for actual surgery data (gastric cancer surgery data) and virtual surgery data (3D patient model) according to the present invention.
Figure 9 is a flowchart of a method for generating a surgical tool and organ recognition learning model based on the acquired actual surgical data and balance data.

The advantages and features of the present invention and methods for achieving them will become clear by referring to the embodiments described in detail below along with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below and may be implemented in various different forms. The present embodiments are merely provided to ensure that the disclosure of the present invention is complete and to provide a general understanding of the technical field to which the present invention pertains. It is provided to fully inform the skilled person of the scope of the present invention, and the present invention is only defined by the scope of the claims.

The terminology used herein is for describing embodiments and is not intended to limit the invention. As used herein, singular forms also include plural forms, unless specifically stated otherwise in the context. As used in the specification, “comprises” and/or “comprising” does not exclude the presence or addition of one or more other elements in addition to the mentioned elements. Like reference numerals refer to like elements throughout the specification, and “and/or” includes each and every combination of one or more of the referenced elements. Although “first”, “second”, etc. are used to describe various components, these components are of course not limited by these terms. These terms are merely used to distinguish one component from another. Therefore, it goes without saying that the first component mentioned below may also be a second component within the technical spirit of the present invention.

Unless otherwise defined, all terms (including technical and scientific terms) used in this specification may be used with meanings commonly understood by those skilled in the art to which the present invention pertains. Additionally, terms defined in commonly used dictionaries are not interpreted ideally or excessively unless clearly specifically defined.

In this specification, 'computer' includes all various devices capable of performing computational processing. For example, computers include not only desktop PCs and laptops (Note Books), but also smart phones, tablet PCs, cellular phones, PCS phones (Personal Communication Service phones), and synchronous/asynchronous computers. This may also include IMT-2000 (International Mobile Telecommunication-2000) mobile terminals, Palm Personal Computers (Palm PCs), and personal digital assistants (PDAs).

Hereinafter, embodiments of the present invention will be described in detail with reference to the attached drawings.

FIG. 1A is a diagram illustrating an apparatus 10 for collecting learning data for surgical tool recognition and long-term learning training according to the present invention.

Figure 1B is a flowchart showing the learning data collection process for surgical tool recognition and long-term learning training according to an embodiment of the present invention.

Figure 2a is a diagram illustrating an annotation of surgical tools and organs in an image according to the present invention.

Figure 2b is a diagram to explain generating actual semantic image data based on actual surgical data through a segmentation model according to the present invention.

Figure 2c is a diagram illustrating the creation of virtual semantic image data based on virtual surgical data through a segmentation model according to the present invention.

FIG. 2D is a diagram illustrating generating random semantic image data based on random surgical data through a segmentation model according to the present invention.

Hereinafter, with reference to FIGS. 1A to 2D, the apparatus 10 for collecting learning data for surgical tool and organ recognition training according to the present invention will be described.

The device 10 can provide learning data of surgical tools and organs for accurate surgical tool recognition and organ learning. Additionally, the device 10 may provide learning data that can resolve class imbalance.

Accordingly, the device 10 can resolve class imbalance by obtaining data that is difficult to obtain from actual surgery data as virtual data. In addition, the device 10 can generate a lot of repetitive and diverse data, and use this to create a more accurate learning model.

The device 10 may include various devices that can perform computational processing and provide results to the user.

It can be in the form of a computer. More specifically, the computer may include various devices that can perform computational processing and provide results to the user.

For example, computers include not only desktop PCs and laptops (Note Books), but also smart phones, tablet PCs, cellular phones, PCS phones (Personal Communication Service phones), and synchronous/asynchronous computers. This may also include IMT-2000 (International Mobile Telecommunication-2000) mobile terminals, Palm Personal Computers (Palm PCs), and personal digital assistants (PDAs). Additionally, if a Head Mounted Display (HMD) device includes computing capabilities, the HMD device may be a computer.

Additionally, a computer may be a server that receives requests from clients and performs information processing.

Additionally, the device 10 may include an acquisition unit 110, a memory 120, and a processor 130. Here, device 10 may include fewer or more components than those shown in FIG. 1A.

The acquisition unit 110 may include one or more modules that enable wireless communication between the device 10 and an external server (not shown) or between the device 10 and a communication network (not shown).

Here, the acquisition unit 110 may include one or more modules that connect the device 10 to one or more networks.

The acquisition unit 110 may acquire at least a plurality of surgical data among actual surgical data, virtual surgical data, and random surgical data. Here, the acquisition unit 110 may acquire the actual surgery data, virtual surgery data, and random surgery data from the external server (not shown).

A communication network (not shown) can transmit and receive various information between the device 10 and an external server (not shown). Various types of communication networks (not shown) may be used, such as WLAN (Wireless LAN), Wi-Fi, Wibro, Wimax, HSDPA (High Speed Downlink Packet Access), etc. Wireless communication methods or wired communication methods such as Ethernet, xDSL (ADSL, VDSL), HFC (Hybrid Fiber Coax), FTTC (Fiber to The Curb), and FTTH (Fiber To The Home) can be used.

Meanwhile, the communication network (not shown) is not limited to the communication methods presented above, and may include all types of communication methods that are widely known or will be developed in the future in addition to the communication methods described above.

Memory 120 may store data supporting various functions of device 10. The memory 120 may store a number of application programs (application programs or applications) running on the device 10, data for operating the device 10, and commands. At least some of these application programs may exist for basic functions of the device 10. Meanwhile, the application program may be stored in the memory 120, installed on the device 10, and driven by the processor 130 to perform the operation (or function) of the device 10.

Additionally, the memory 120 may store the actual surgery data, the virtual surgery data, and the random surgery data acquired by the acquisition unit 110 for each patient.

Additionally, the memory 120 may be provided with a plurality of processes for collecting learning data for surgical tool and organ recognition training according to the present invention. Here, the plurality of processes will be described later when explaining the operation of the processor 130.

In addition to operations related to the application program, the processor 130 may generally control the overall operation of the device 10. The processor 130 processes signals, data, information, etc. input or output through the components discussed above, or runs an application program or a plurality of processes stored in the memory 120, thereby providing or processing appropriate information or functions to the user. can do.

Additionally, the processor 130 may control at least some of the components examined with FIG. 1A in order to run an application program or a plurality of processes stored in the memory 120. Furthermore, the processor 130 may operate in combination with at least two or more of the components included in the device 10 to drive the application program or the plurality of processes.

Referring to FIG. 1B, the learning data collection method for surgical tool and organ recognition learning training of the present invention is a method executed by the processor 130 of the device 10, where the processor 130 collects at least a plurality of surgical data. A step of acquiring (S110), after acquiring the surgical data, a step of generating semantic image data based on the plurality of surgical data obtained through a segmentation model (S120), the plurality of acquired surgical data and the semantic It may include a step (S130) of configuring and providing image data as a learning data set.

The processor 130 may acquire at least a plurality of surgical data (S110).

Here, the processor 130 may acquire at least a plurality of surgical data among actual surgical data, virtual surgical data, and random surgical data based on a first process among the plurality of processes.

Here, the actual surgery data may include actual surgical tool data including the type of surgery actually performed, location information of the tools used, and information on the surgical tools used, and actual organ data about the organs of the surgery actually performed. there is.

Additionally, the actual surgery data may be obtained from actual surgery image data, and the actual surgery image data may be image data acquired during an actual surgery.

In addition, as described above, the actual surgery data includes the type of surgery in which the surgery was performed and the location information of the tools used during the actual surgery. The actual surgical tool data, which is information on the surgical tools used at this time, and information on the surgical organs operated on It can include long-term data from actual surgeries.

The processor 130 may generate the actual surgery data based on the actual surgery image data through a segmentation model. Here, the segmentation model will be described in detail later.

Here, the actual surgical tool data is bounded within the actual surgical image by distinguishing at least one of the head, wrist, and body of the tool through the segmentation model, or is converted into a polygon in pixel units. It may contain at least one of the things being processed.

Alternatively, the actual surgical tool data may include at least one of being bounded within the actual surgical image for each tool feature through the segmentation model or polygonally processed on a pixel basis.

Here, the characteristics of the tool include all characteristics such as the type of tool and the size of the tool, and can be classified and bound for each feature through the segmentation model, or polygonized on a pixel basis. This polygon processing may be a method of drawing an outline of a specific part on a pixel-by-pixel basis.

In addition, in the actual surgery organ data, at least one of the pixels containing visually distinguished organs in the actual surgery image may be polygonized through the segmentation model. Alternatively, the actual surgical organ data may include at least one of bounding a specific portion containing the organ within the actual surgical image through the segmentation model.

Bounding within the actual surgery image may be annotating the part to be specified within the actual surgery image, that is, the part corresponding to the information to be learned, in the form of a bounding box through the segmentation model. .

Alternatively, polygon processing within the actual surgery image may be polygon processing a portion to be specified within the actual surgery image, that is, a portion corresponding to information to be learned, on a pixel-by-pixel basis through the segmentation model.

Here, the annotation may be a correct answer label including a bounding box or a specific polygon-processed part.

Referring to Figure 2a, the method of annotating surgical tools by displaying them as bounding boxes within a surgical image is not only performed within the actual surgical image, but also involves performing annotation within an image in a 3D surgical environment or 3D virtual space. The same can be applied.

Classifying the surgical tool into the head, joint, and body and processing it into a bounding box is to obtain accurate position information of the tool. By specifying the position of the head of the surgical tool and the position of the joint and/or body, the surgical tool Location and direction information can be obtained.

However, depending on the kinematics, surgical tools may be divided into only the head and body, or they may be divided into the head, joints, and body.

Surgical tools may include, by way of example, laparoscopic surgical tools and robotic surgical tools, as well as needles, specimen bags, and surgical tubes frequently used during surgical procedures. In addition to the examples listed above, all tools used during surgery are included. You can.

Virtual surgery data may include a virtual surgery environment, virtual surgery tool data used in the virtual surgery environment, and virtual surgery organ data. At this time, the virtual surgical environment is a 3D surgical environment created by turning it into 3D using specific actual surgical image data, and the virtual surgical tool data and virtual surgical organ data are virtual surgical environments derived from use by the user within the 3D surgical environment. It may be information on surgical tools.

The processor 130 may generate the virtual surgery data based on the virtual surgery environment through a segmentation model. Here, the segmentation model will be described in detail later.

This virtual surgical environment may be an environment implemented through a virtual body model. Here, the virtual surgical environment may be a digital twin virtual environment for effective labeling distribution for surgical tools and organ training, such as:

Additionally, a virtual body model can be created using abdominal computed tomography images considering the surgical situation, labels for segmenting the five reconstructed organs, and skin reflecting the patient's pneumoperitoneum for 3D reconstruction.

Additionally, a virtual surgical environment can be created by reconstructing a 3D model using commercial software tools 3D Max, Zbrush, and Substance Painter, using catalogs of robotic and laparoscopic instruments and actual measurements in the case of surgical tools.

Additionally, a virtual surgical environment can be created by constructing a virtual camera environment with the same internal/external parameters as the endoscope used in robotic surgery to reproduce the camera viewpoint of the surgery.

A 3D surgical environment created in 3D using actual surgical video data can be created by projecting from 2D information corresponding to a specific camera viewpoint.

Here, actual surgical image data is, for example, computed tomography (CT), nuclear magnetic resonance computed tomography (NMR-CT), and positron emission tomography (PET). ), CBCT (Cone Beam CT, CBCT), Electron BeamTomography (EBT), X-ray image, and Magnetic Resonance Imaging (MRI), but limited to the above examples. However, it can include all image data taken during surgery, including videos.

The virtual surgical tool data and virtual surgical organ data may be information on surgical tools and surgical organs that are arbitrarily used by the user within a configured 3D surgical environment.

The processor 130 performs learning training by annotating virtually used surgical tools or surgical organs with a bounding box through the segmentation model or processing them into polygons on a pixel basis and using them as learning data, even if the surgical tools are not used in actual surgery. can be performed.

However, in the case of virtual surgical tool data and virtual surgical organ data, the technique of translation between images using a generative adversarial network can be applied for learning.

The generative adversarial network will be described later along with the explanation of the deep neural network.

Random surgery data is virtual data generated from domain randomization, and is randomly generated by randomly changing the setting conditions of surgical tools, organs to be operated, and cameras, and randomly generated specific surgery, organs to be operated on, and specific organs to be operated on. It may include data on at least one of the random surgical tools used in surgery.

The processor 130 may generate the random surgery data based on the virtual surgery environment through a segmentation model. Here, the segmentation model will be described in detail later.

That is, the processor 130 annotates surgical tools or surgical organs as bounding boxes or polygon-processes them on a pixel basis through the segmentation model for random surgery data, which is virtual data generated from domain randomization. You can.

The 3D virtual space implemented to acquire random surgery data may include a 3D virtual space of random surgery, and may include a 3D surgery environment created identically to the 3D surgery environment included in the above-described virtual surgery data. there is.

For organs subject to surgery, the color, texture, deformation, and location of the organ can be set randomly. For surgical tools, the type, texture, kinematic motion, and location of the surgical tool can be set randomly. And in the case of cameras, the camera's field of view (FOV), location, type, and distortion can be set randomly.

That is, the processor 130 can randomly set all surgical target organs, surgical tools, camera conditions, etc., regardless of the surgical tools commonly used in existing actual surgeries, and convert randomly acquired information into data.

For example, when acquiring random surgical data, one or more robotic tools (e.g., Harmonic Ace, Maryland Bipolar Forceps, Cadiere Forceps, Stapler, Medium-Large Clip Applier, Small Clip Applier, etc.), one or more laparoscopic tools (e.g., Atraumatic Grasp Applier, etc.) , Electric hook, Curved Atraumatic Grasper, Suction-Irrigation, Clip Applier (metal), Scissors, Overholt, Ligasure, etc.) and one or more organs. However, it is not limited to the above embodiment, and 3D models can be produced for various robotic tools and various laparoscopic tools.

Alternatively, when acquiring random surgical data, tools used in surgery for gastrectomy (e.g., atraumatic grasper, Cadiere forceps, curved atraumatic grasper, harmonic ace, Maryland bipolar forceps, medium-large clip applier, Overholt, scissors, small clip applier, stapler, and suction-irrigation) and one or more laparoscopic tools (e.g., atraumatic grasper, clip applier (hem-o-lok), clip applier (metal), curved atraumatic grasper, electrichook, Ligasure, Overholt, scissors, and suction-irrigation ), you can create a 3D model. However, it is not limited to the above embodiment, and 3D models can be produced for various robotic tools and various laparoscopic tools.

The processor 130 may generate semantic image data based on at least a plurality of surgical data acquired through a segmentation model (S120).

Here, the partition model may include a semantic partition model and an instance partition model. The semantic segmentation model may include at least one of DeepLabV2, HRNet, and HRNet+OCR. Additionally, the instance segmentation model may include at least one of Casecade R-CNN, HRNet, and HRNet+OCR.

Here, the semantic segmentation model can infer all pixels existing in the image, and the instance segmentation model can infer instance information within the bounding box at the pixel level.

More specifically, the processor 130 may generate the semantic image data from frames and corresponding label maps each extracted from the actual surgery data, the virtual surgery data, and the random surgery data through the segmentation model.

Here, the semantic image data may include semantic surgical tool data for tools or semantic surgical organ data for organs included in each of the plurality of surgical data.

In addition, the semantic surgical tool data or the semantic surgical organ data may include at least one of being bound to the tool or the organ included in each of the plurality of surgical data, or being polygon-processed on a pixel-by-pixel basis.

And, the semantic surgical tool data or the semantic surgical organ data includes the location of the tool or organ, the name of the tool or organ, and the state of the tool or organ for each bounded or polygon-processed part (as an example) , tool: state in which the joint is folded, organ: state in which part is cut, etc.).

Specifically, looking at FIG. 2B, when the plurality of surgical data includes the actual surgical data, the processor 130 may generate actual semantic image data based on the actual surgical data through the segmentation model.

Additionally, referring to FIG. 2C, when the plurality of surgical data includes the virtual surgical data, the processor 130 may generate virtual semantic image data based on the virtual surgical data through the segmentation model.

Additionally, referring to FIG. 2D, when the plurality of surgical data includes the random surgical data, the processor 130 may generate random semantic image data based on the random surgical data through the segmentation model.

The processor 130 may configure the acquired plurality of surgical data and the semantic image data into a learning data set and provide it (S130). Here, the subject receiving the learning data set may include users, computers, systems, etc. that utilize the learning data.

More specifically, the processor 130 may provide the plurality of surgical data and the semantic image data for each of the plurality of surgical data as a learning data set based on a third process among the plurality of processes.

Accordingly, since actual surgery images are lacking and only tools used in actual surgery are likely to be used, the learning performance can be improved by adding additional virtual surgery data and random surgery data, and additionally adding semantic image data. .

As an example, the processor 130 may configure the actual surgery data and semantic image data for the actual surgery data into a learning data set and provide it.

Figure 3 is a flowchart of a method for generating a surgical tool recognition learning model using the learning data set of the present invention.

Figure 3 further includes the step of performing surgical tool recognition learning training in the method of Figure 1b (S150) and generating a surgical tool recognition learning model (S170).

The step of performing surgical tool recognition learning training (S150) is to perform surgical tool recognition learning training on a specific surgical tool used for a specific surgery based on the learning data set.

At this time, the learning data set is obtained from the above-described actual surgery data, virtual surgery data, and/or random surgery data, and the acquired learning data set is at least one of the surgery type, organ size, organ condition, and surgery subject age. It can be divided into one.

In addition, the learning data may be obtained by maintaining class balance as a uniform amount of data on a specific surgical tool for a specific surgery.

Specific surgical tools may include, by way of example, laparoscopic surgical tools and robotic surgical tools, as well as needles, specimen bags, and surgical tubes frequently used during surgical procedures. In addition to the examples listed above, tools used during surgery may include: All can be included.

Specific examples include laparoscopic tools and auxiliary surgical tools used in cholecystectomy: Atraumatic grasper, Electric hook, Curved Atraumatic Grasper, Suction-Irrigation, Clip Applier (metal), Scissors, Overholt, Ligasure, Clip Applier (Ham) -o-Lok), Specimen Bag may be applicable. Here, in the case of the cholecystectomy, the surgical organs may include the gallbladder and the liver.

In the case of gastrectomy, it may include robotic tools, laparoscopic tools, and auxiliary tools, such as Harmonic Ace, Maryland Bipolar Forceps, Cadiere Forceps, Medium-Large Clip Applier, Small Clip Applier, Curved Atraumatic Grasper, Stapler, Suction, Suction- This may include Irrigation, Scissors, Atraumatic Grasper, Overholt, Guaze Needle, Endotip, Drain Tube, Needle, Needle Holder, and Specimen Bag. Here, in the case of the gastrectomy, surgical organs may include the stomach, liver, gallbladder, spleen, and pancreas.

Specifically, the robotic tools may include Harmoic Ace, Maryland Bipolar Forceps, Cadiere Forceps, Medium-large Clip Applier, Small Clip Applier, etc., and the laparoscopic tools may include Curved Atraumatic Grasper, Stapler, Suction, etc. Surgical auxiliary tools may include Guaze Needle, Endotip, Specimenbag, Drain Tube, etc.

The step of generating a surgical tool recognition learning model (S170) is to generate a surgical tool and organ recognition learning model based on the performed surgical tool recognition learning training.

A deep neural network (DNN) according to embodiments of the present invention refers to a system or network that builds one or more layers in one or more computers and makes decisions based on a plurality of data. For example, a deep neural network may be implemented as a set of layers including a convolutional pooling layer, a locally-connected layer, and a fully-connected layer. A convolutional pooling layer or a local connection layer may be configured to extract features within the image. The fully connected layer can determine correlations between features of the image. In some embodiments, the overall structure of the deep neural network may be a convolutional pooling layer followed by a local connection layer, and a local connection layer followed by a fully connected layer. A deep neural network can include various judgment criteria (i.e., parameters), and new judgment criteria (i.e., parameters) can be added through analysis of input images.

The deep neural network according to embodiments of the present invention is a structure called a convolutional neural network suitable for image analysis, and is a feature extraction layer that autonomously learns the features with the greatest discriminative power from given image data. ) and a prediction layer that learns a prediction model to produce the highest prediction performance based on the extracted features can be configured in an integrated structure.

The feature extraction layer is a convolution layer that creates a feature map by applying multiple filters to each area of the image and spatially integrates the feature map to extract features that are invariant to changes in position or rotation. It can be formed in a structure in which the pooling layer that allows extraction is alternately repeated several times. Through this, various levels of features can be extracted, from low-level features such as points, lines, and surfaces to complex and meaningful high-level features.

The convolution layer obtains a feature map by taking a non-linear activation function as the inner product of the filter and the local receptive field for each patch of the input image. In comparison, CNN is characterized by using filters with sparse connectivity and shared weights. This connection structure reduces the number of parameters to learn, makes learning through the backpropagation algorithm efficient, and ultimately improves prediction performance.

The integration layer (Pooling Layer or Sub-sampling Layer) creates a new feature map using the local information of the feature map obtained from the previous convolution layer. In general, the newly created feature map by the integration layer is reduced to a smaller size than the original feature map. Representative integration methods include max pooling, which selects the maximum value of the corresponding area within the feature map, and There is average pooling, which calculates the average value of an area. The feature map of the integrated layer can generally be less affected by the location of arbitrary structures or patterns present in the input image than the feature map of the previous layer. In other words, the integration layer can extract features that are more robust to local changes such as noise or distortion in the input image or previous feature map, and these features can play an important role in classification performance. The role of another integration layer is to reflect features of a wider area as you go up to higher learning layers in the deep structure. As feature extraction layers accumulate, local features are reflected in lower layers and moved up to higher layers. Increasingly, more abstract features can be generated that reflect the features of the entire image.

In this way, the features finally extracted through repetition of the convolutional layer and the integration layer are classified into fully connected layers such as classification models such as Multi-layer Perception (MLP) or Support Vector Machine (SVM). -connected layer) and can be used for classification model learning and prediction.

However, the structure of the deep neural network according to embodiments of the present invention is not limited to this, and may be formed as a neural network of various structures.

The generative adversarial network learning model is an unsupervised learning model that consists of two models: a discriminator (D), which is responsible for classification, and a generator (G), which generates data from random noise.

The generative adversarial network learning model is a model in which the generator G and the discriminator D confront each other and improve each other's performance. The discriminator D strives to judge only the original data as true, and the generator determines that the discriminator D is false. This is a model in which the performance of both models increases together by generating fake data to prevent this from happening.

Generator G tries to find out the probability distribution of the original data and reproduces the distribution to make fake data no different from real data, and discriminator D distinguishes whether the data to be determined is real data or data created by generator G. The task is to estimate the probability for each.

The formula for the generative adversarial network learning model is as shown in Equation 1 below.

[Equation 1]

Figure 4 is a flowchart of a method for obtaining insufficient data in actual surgery data as balanced data.

Figure 5 is a graph showing the distribution of annotations obtained from actual surgical data according to the present invention.

Figure 6 is a graph showing the distribution of annotations changed by acquiring virtual surgery data and/or random surgery data according to the present invention.

Figure 7 is a graph explaining the distribution of annotations obtained from actual surgery data (laparoscopic cholecystectomy and stomach cancer video) according to the present invention.

Figure 8 is a graph illustrating the distribution of annotations obtained from laparoscopic cholecystectomy and gastrectomy for actual surgery data (gastric cancer surgery data) and virtual surgery data (3D patient model) according to the present invention.

Referring to Figure 4, the learning data collection method for surgical tool recognition training of the present invention includes the steps of acquiring actual surgical data (S210), extracting the type and required amount of data lacking annotation (S230), and It includes a step (S250) of obtaining data corresponding to the type of extracted data as balance data.

The actual surgery data, virtual surgery data, and random surgery data in FIGS. 4 to 9 are the same as those described above in FIGS. 1A to 3.

The step of extracting the type and required amount of data lacking annotations (S230) is to extract the type and required amount of data lacking annotations that cause class imbalance from the acquired actual surgery data.

The tools used during actual surgery may have restrictions/limitations, there may be movements that are not performed in actual surgery, and the tools and surgical movements may be completely different for each actual surgery case, so all tools are used during actual surgery. and it is difficult to obtain data on movement. For this reason, when only actual surgery data is acquired, class imbalance may occur.

In the present invention, class imbalance that occurs only with actual surgery data is resolved by acquiring virtual surgery data and/or random surgery data. Accordingly, the type and required amount of data causing class imbalance can be extracted from the acquired actual surgery data, and the required amount of data type can be obtained from the virtual surgery data and/or random surgery data.

In the step (S230) of extracting the type and required amount of data lacking annotation, data corresponding to the type of extracted data is obtained as balance data in the required amount from at least one of virtual surgery data and random surgery data.

Balanced data refers to additional data obtained from virtual surgery data and/or random surgery data to resolve class imbalance that occurs only with actual surgery data.

Types of data lacking annotations may include at least one of surgical operations, surgical tool types, specific surgical positions, and surgical tool use positions.

In this embodiment, the actual surgery data includes actual surgical tool data including the type of surgery actually performed, location information of the tool used, and information on the surgical tool used, and the virtual surgery data includes the virtual surgery data. It includes virtual surgical tool data used in the environment and virtual surgical environment, and the random surgical data is virtual data generated from domain randomization, and each data is described above in FIGS. 1A to 3. Same as the content.

Referring to FIG. 5, FIG. 5 shows the distribution of actual surgical data classified by each surgical tool, including actual surgical data for cholecystectomy (top graph of FIG. 5) and actual surgical data for gastrectomy (top graph of FIG. 5). Bottom graph) It can be seen that the acquired data itself is unbalanced, causing class imbalance.

Referring to Figure 6, Figure 6 shows the distribution of annotations changed by acquiring virtual surgery data and/or random surgery data, which are values of data that combine actual surgery data and balanced data, and are classified by each surgical tool. will be. It can be seen that the sum of actual surgical data and balanced data for cholecystectomy (top graph of FIG. 6) and the sum of actual surgical data and balanced data for gastrectomy (bottom graph of FIG. 6) are balanced.

Therefore, as a result, according to the present invention, the problem of class imbalance is solved, and learning results using the corresponding data are derived as more accurate results.

Referring to Figure 7, Figure 7 can show the distribution of annotations obtained from actual surgery data (laparoscopic cholecystectomy and stomach cancer videos).

Here, the number of annotations can be adjusted in a logarithmic scale. It can be seen that both cholecystectomy and gastrectomy, which are actual surgery data, cause serious imbalance problems in several tools.

Referring to Figure 8, Figure 8 can show the annotation distribution obtained from laparoscopic cholecystectomy and gastrectomy for actual surgery data (gastric cancer video) and 3D patient model.

Compared to the distribution of annotations obtained from actual surgery data (e.g., stomach cancer surgery data), the class imbalance problem is that both of the above two surgeries (laparoscopic cholecystectomy and gastrectomy) are class imbalance in the distribution of real surgery data, virtual surgery data, and random surgery data. You can confirm that the problem has been alleviated.

Figure 9 is a flowchart of a method for generating a surgical tool recognition learning model based on the acquired actual surgical data and balance data.

Referring to Figure 9, the method of generating a surgical tool recognition learning model further includes performing surgical tool recognition learning training in the steps of Figure 6 (S270) and generating a surgical tool recognition learning model (S290). do.

The step of performing surgical tool recognition learning training (S270) is to perform surgical tool recognition learning training on a specific surgical tool used for a specific surgery using the acquired balance data and actual surgical data as learning data.

The step of generating a surgical tool recognition learning model (S290) is to generate a surgical tool recognition learning model based on the performed surgical tool recognition learning training.

The step of performing surgical tool recognition learning training in FIG. 9 (S270) and the step of generating a surgical tool recognition learning model (S290) include the step of performing surgical tool recognition learning training in FIG. 3 (S150) and surgical tool recognition learning. Only the model creation step (S170) and the learning data are different, and the execution method is applied the same.

The steps of the method or algorithm described in connection with embodiments of the present invention may be implemented directly in hardware, implemented as a software module executed by hardware, or a combination thereof. The software module may be RAM (Random Access Memory), ROM (Read Only Memory), EPROM (Erasable Programmable ROM), EEPROM (Electrically Erasable Programmable ROM), Flash Memory, hard disk, removable disk, CD-ROM, or It may reside on any type of computer-readable recording medium well known in the art to which the present invention pertains.

Above, embodiments of the present invention have been described with reference to the attached drawings, but those skilled in the art will understand that the present invention can be implemented in other specific forms without changing its technical idea or essential features. You will be able to understand it. Therefore, the embodiments described above should be understood in all respects as illustrative and not restrictive.

10: device
110: Department of Communications
120: memory
130: processor

Claims (13)

A learning data collection method for surgical tool and organ recognition learning training performed by a device, comprising:
Obtaining at least a plurality of surgical data among actual surgery data, virtual surgery data, and random surgery data; and
Comprising the step of configuring and providing the obtained plurality of surgical data into a learning data set,
The actual surgical data includes actual surgical tool data including the type of surgery actually performed, location information of the tool used, and information on the surgical tool used, and actual surgical organ data about the organ of the surgery actually performed. ,
The virtual surgery data includes a virtual surgical environment, virtual surgical tool data used in the virtual surgical environment, and virtual surgical organ data for organs in the virtual surgical environment,
The random surgery data is virtual data generated from domain randomization,
The learning data collection method is,
Extracting the type and required amount of data lacking annotations that cause class imbalance from the acquired actual surgery data; and
Comprising the step of obtaining data corresponding to the type of the extracted data by the required amount as balance data from at least one of virtual surgery data and random surgery data,
The actual surgical tool data is bounded within the actual surgical image by distinguishing at least one of the head, wrist, and body of the tool, or processed as a polygon on a pixel basis, or the tool's Includes at least one of being bounded within the actual surgery image for each feature or processed into the polygon on a pixel basis,
Bounding within the actual surgery image means that the part corresponding to the information to be learned within the actual surgery image is annotated in the form of a bounding box through a segmentation model, and the polygon processing is learned within the actual surgery image. The part corresponding to the desired information is polygonally processed on a pixel basis through the segmentation model,
The actual surgery organ data is at least one of the pixels containing visually distinct organs in the actual surgery image is polygonized, or a specific portion containing the organ in the actual surgery image is bounded. It contains one,
The learning data collection method is,
After acquiring the plurality of surgical data, generating semantic image data for the plurality of surgical data obtained through the segmentation model,
The semantic image data is generated based on frames and corresponding label maps each extracted from the actual surgery data, the virtual surgery data, and the random surgery data through the segmentation model,
The segmentation model includes a semantic segmentation model capable of inferring all pixels present in an image and an instance segmentation model capable of inferring instance information in a bounding box at the pixel level. A learning data collection method for surgical tool and organ recognition training. . According to paragraph 1,
Performing surgical tool recognition learning training on a specific surgical tool used for a specific surgery based on the learning data set; and
Further comprising the step of generating a surgical tool recognition learning model based on the surgical tool recognition learning training performed above,
Learning data collection method for surgical instrument and organ recognition learning training. delete According to paragraph 1,
The virtual surgical environment is a 3D surgical environment created based on specific actual surgical image data,
The virtual surgical tool data and the virtual surgical organ data are virtual surgical tool information and surgical organ information derived from use by the user within the 3D surgical environment,
Learning data collection method for surgical instrument and organ recognition learning training. According to paragraph 1,
The random surgery data is,
This is virtual data generated from domain randomization, and is randomly generated by randomly changing the setting conditions of surgical tools, surgical target organs, and cameras, and randomly generated specific surgery, specific surgical target organs, and random data used in a specific surgery. Containing data for at least one of the surgical instruments,
Learning data collection method for surgical instrument and organ recognition learning training. According to paragraph 1,
The learning data is,
Data on specific surgical tools and specific surgical organs for specific surgeries are each obtained as a uniform amount of data with class balance,
Learning data collection method for surgical instrument and organ recognition learning training. According to paragraph 1,
The step of providing the learning data set configuration is,
Providing the plurality of surgical data and the semantic image data corresponding to each of the plurality of surgical data as a learning data set,
Learning data collection method for surgical instrument and organ recognition learning training. delete In clause 7,
The semantic image data is,
Containing semantic surgical tool data for the tool or semantic surgical organ data for the organ included in each of the plurality of surgical data,
The semantic surgical tool data or the semantic surgical organ data includes at least one of a location range of the tool or the organ, and a name of the tool or the organ,
Learning data collection method for surgical instrument and organ recognition learning training. delete delete According to paragraph 2,
If the specific surgery above is a gastrectomy,
The specific surgical tool includes at least one of a robotic surgical tool, a laparoscopic surgical tool, and an auxiliary surgical tool,
The specific surgical organs include the stomach, liver, gallbladder, spleen and pancreas,
If the specific surgery above is a cholecystectomy,
The specific surgical tool includes at least one of a laparoscopic tool and an auxiliary surgical tool,
The specific surgical organs include the gallbladder and liver,
Learning data collection method for surgical instrument and organ recognition learning training. A surgical tool coupled to a hardware computer and stored in a computer-readable recording medium to execute the method of any one of claims 1, 2, 4 to 7, 9, and 12, and A learning data collection program for long-term cognitive learning training.

Images