KR20220021423A - Apparatus and method for collecting learning data for training of surgical tool and organ recognition - Google Patents

Abstract

Provided are a method for collecting learning data for a surgical tool recognition learning training and a program therefor. As the method executed by a computer, the method for collecting the learning data for the surgical tool recognition learning training comprises: a step of acquiring, by the computer, at least a plurality of surgical data among the actual surgical data, the virtual surgical data, and the random surgical data; and a step of composing and providing the acquired plurality of surgical data as a learning dataset. Therefore, the present invention is capable of solving a class imbalance.

Description

Apparatus and method for collecting learning data for surgical tool and organ recognition learning training

The present invention relates to a learning data collection apparatus and method for surgical tool and organ recognition learning training.

In recent surgical operations, the proportion of open surgery has greatly decreased, and the proportion of minimally invasive surgery, which can improve recovery speed by minimizing patient complications, is increasing significantly.

Among them, the rate of minimally invasive surgery using robots is increasing even more rapidly. In addition, due to the characteristics of laparoscopic surgery, in which a specialist views an image input from a camera, minimally invasive surgery records all procedures of the operation occurring inside the patient as an image.

Therefore, if the surgical image is regarded as a kind of surgical record and the surgical process is recognized, not only the evaluation and analysis of the operation, but also real-time guidance during the operation as well as the automated operation may be possible.

In order to analyze and evaluate the automated surgical process using surgical images, position measurement, movement recognition, and organ recognition of surgical tools are essential. Surgery data, that is, surgical process, position measurement and movement of surgical tools, recognition of organs, etc. may be acquired and used as learning data.

Republic of Korea Patent Publication No. 10-2013-0100758 (2013.09.11)

The problem to be solved by the present invention is to provide an accurate surgical tool and learning data of a surgical tool and an organ for learning to recognize an organ.

In addition, the problem to be solved by the present invention is to provide learning data capable of solving class imbalance.

The problems to be solved by the present invention are not limited to the problems mentioned above, and other problems not mentioned will be clearly understood by those skilled in the art from the following description.

The learning data collection method for surgical tool and organ recognition learning training according to an embodiment of the present invention for solving the above-described problems is a method executed by a computer, and the computer uses real surgical data, virtual surgery data, and random surgery data Comprising the steps of acquiring at least a plurality of surgical data and providing the plurality of acquired surgical data as a learning data set, wherein the actual surgical data includes the type of surgery actually performed and the location of the tool used. It includes real surgical tool data including information and information on the used surgical tools and real surgical organ data for organs of actually performed surgery, and the virtual surgical data is used in a virtual surgical environment and a virtual surgical environment. It includes virtual surgical tool data and virtual surgical organ data for organs in a virtual surgical environment, and the random surgical data may be virtual data generated from domain randomization.

The learning data collection method for surgical tool recognition learning training according to an embodiment of the present invention for solving the above-described problems is a surgical tool recognition learning training for a specific surgical tool used for a specific surgery based on the learning data set and generating a surgical tool recognition learning model based on the performed surgical tool recognition learning training.

The actual surgical tool data is bound within the actual surgical image by dividing at least one of a head, a wrist, and a body of the tool, or is processed as a polygon in a pixel unit, or of the tool. For each feature, it may include at least one of bounding within the actual surgical image or processing the polygon in units of pixels.

The virtual surgical environment is a 3D surgical environment produced by 3D using specific real surgical image data, and the virtual surgical tool data and the virtual surgical organ data are utilized and derived from the user within the 3D surgical environment. virtual surgical tools and surgical organ information.

The random surgery data is virtual data generated from domain randomization, and by randomly changing the setting conditions of a surgical tool, a target organ, and a camera, a specific surgery, a specific surgery target organ, and It may include data on at least one of random surgical tools used in a specific surgery.

The learning data may be obtained by balancing a class as data of a specific surgical tool and a specific surgical organ for a specific surgery, respectively, as uniform amounts of data.

In addition, the present invention further comprises the step of generating semantic image data for the acquired surgical data through a segmentation model after acquiring the surgical data, and the providing of the learning data set configuration includes the plurality of surgeries Data and the semantic image data corresponding to each of the plurality of surgical data may be provided as a learning data set.

Here, the semantic image data generating step, when the actual surgical data is included in the plurality of surgical data, generates real semantic image data based on the actual surgical data through the segmentation model, and the plurality of surgical data When the virtual surgical data is included in the segmentation model, virtual semantic image data is generated based on the virtual surgical data through the segmentation model, and when the random surgical data is included in the plurality of surgical data, through the segmentation model Random semantic image data may be generated based on the random surgical data.

In addition, the semantic image data includes semantic surgical tool data for the tool or semantic surgery organ data for the organ included in each of the plurality of surgical data, and the semantic surgical tool data or the semantic surgery organ data is , a location range of the tool or the organ, and at least one of a name of the tool or the organ.

The learning data collection method for the surgical tool recognition learning training according to an embodiment of the present invention for solving the above-mentioned problems is a method executed by a computer, the computer acquiring actual surgical data The acquired actual surgical data extracting a type and a required amount of data lacking an annotation for generating a class imbalance in at least one of virtual surgery data and random surgery data corresponding to the type of data extracted by the required amount Acquiring data as balance data, wherein the actual surgical data includes actual surgical tool data including information on the type of surgery actually performed, location information of a used tool, and information on a used surgical tool, The virtual surgical data includes a virtual surgical environment and virtual surgical tool data used in the virtual surgical environment, and the random surgical data may be virtual data generated from domain randomization.

The learning data collection method for surgical tool recognition learning training according to an embodiment of the present invention for solving the above-described problems is a specific surgical tool used for a specific surgery by using the acquired balance data and the actual surgical data as learning data It may include the steps of performing a surgical tool recognition learning training for the and generating a surgical tool recognition learning model based on the performed surgical tool recognition learning training.

When the specific operation is gastrectomy, the specific surgical tool includes at least one of a robotic surgical tool, a laparoscopic surgical tool, and an auxiliary surgical tool, and the specific surgical organ includes a stomach, liver, gallbladder, spleen, and pancreas and, when the specific operation is cholecystectomy, the specific surgical tool may include at least one of a laparoscopic tool and an auxiliary surgical tool, and the specific surgical organ may include a gallbladder and a liver.

The learning data collection program for surgical tool recognition learning training according to another embodiment of the present invention for solving the above-described problems may be combined with a computer, which is hardware, and stored to execute any one of the methods.

Other specific details of the invention are included in the detailed description and drawings.

According to the present invention, by acquiring data that is difficult to obtain from actual surgical data as virtual data, class imbalance can be solved.

In addition, according to the present invention, it is possible to generate a more accurate learning model by generating a lot of repeated various data and utilizing it.

Effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the following description.

1A is a view for explaining an apparatus 10 for collecting learning data for a surgical tool and organ recognition learning training according to the present invention.
1B is a flowchart of a learning data collection method for training for learning a surgical tool and organ recognition according to an embodiment of the present invention.
FIG. 2A is a diagram exemplarily illustrating annotation of a surgical tool and an organ within an image according to the present invention.
2B is a diagram for explaining the generation of actual semantic image data based on actual surgical data through the segmentation model according to the present invention.
2C is a diagram for explaining the generation of virtual semantic image data based on virtual surgical data through the segmentation model according to the present invention.
2D is a diagram for explaining the generation of random semantic image data based on random surgical data through the segmentation model according to the present invention.
3 is a flowchart of a method for generating a surgical tool recognition learning model using a learning data set according to the present invention.
4 is a flowchart of a method for acquiring insufficient data as balance data in actual surgical data according to the present invention.
5 is a graph showing the distribution of annotations obtained from actual surgical data according to the present invention.
6 is a graph showing the distribution of annotations changed by acquiring virtual surgical data and/or random surgical data according to the present invention.
7 is a graph illustrating the distribution of annotations obtained from actual surgical data (laparoscopic cholecystectomy and gastric cancer video) according to the present invention.
8 is a graph explaining the distribution of annotations obtained from laparoscopic cholecystectomy and gastrectomy for actual surgical data (gastric cancer surgery data) and virtual surgery data (3D patient model) according to the present invention.
9 is a flowchart of a method of generating a surgical tool and organ recognition learning model based on the acquired actual surgical data and balance data.

Advantages and features of the present invention and methods of achieving them will become apparent with reference to the embodiments described below in detail in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in various different forms, and only the present embodiments allow the disclosure of the present invention to be complete, and those of ordinary skill in the art to which the present invention pertains. It is provided to fully understand the scope of the present invention to those skilled in the art, and the present invention is only defined by the scope of the claims.

The terminology used herein is for the purpose of describing the embodiments and is not intended to limit the present invention. In this specification, the singular also includes the plural unless specifically stated otherwise in the phrase. As used herein, “comprises” and/or “comprising” does not exclude the presence or addition of one or more other components in addition to the stated components. Like reference numerals refer to like elements throughout, and "and/or" includes each and every combination of one or more of the recited elements. Although "first", "second", etc. are used to describe various elements, these elements are not limited by these terms, of course. These terms are only used to distinguish one component from another. Accordingly, it goes without saying that the first component mentioned below may be the second component within the spirit of the present invention.

Unless otherwise defined, all terms (including technical and scientific terms) used herein will have the meaning commonly understood by those of ordinary skill in the art to which this invention belongs. In addition, terms defined in a commonly used dictionary are not to be interpreted ideally or excessively unless specifically defined explicitly.

As used herein, the term 'computer' includes various devices capable of performing arithmetic processing. For example, computers include desktop PCs and notebooks (Note Books) as well as smart phones, tablet PCs, cellular phones, PCS phones (Personal Communication Service phones), synchronous/asynchronous A mobile terminal of International Mobile Telecommunication-2000 (IMT-2000), a Palm Personal Computer (PC), a Personal Digital Assistant (PDA), and the like may also be applicable.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1A is a view for explaining an apparatus 10 for collecting learning data for surgical tool recognition and long-term learning training according to the present invention.

1B is a flowchart illustrating a learning data collection process for surgical tool recognition and long-term learning training according to an embodiment of the present invention.

FIG. 2A is a diagram exemplarily illustrating annotation of a surgical tool and an organ within an image according to the present invention.

2B is a diagram for explaining the generation of actual semantic image data based on actual surgical data through the segmentation model according to the present invention.

2C is a diagram for explaining the generation of virtual semantic image data based on virtual surgical data through the segmentation model according to the present invention.

2D is a diagram for explaining the generation of random semantic image data based on random surgical data through the segmentation model according to the present invention.

Hereinafter, with reference to FIGS. 1A to 2D , an apparatus 10 for collecting learning data for a surgical tool and organ recognition learning training according to the present invention will be described.

The device 10 may provide learning data of surgical tools and organs for accurate surgical tool recognition and organ learning. In addition, the device 10 may provide training data capable of resolving class imbalance.

Accordingly, the device 10 may solve the class imbalance by acquiring data that is difficult to obtain from actual surgical data as virtual data. In addition, the device 10 may generate a lot of repeated various data and use it to generate a more accurate learning model.

The device 10 may include all of a variety of devices capable of providing a result to a user by performing arithmetic processing.

It can be in the form of a computer. More specifically, the computer may include all of a variety of devices capable of providing a result to a user by performing arithmetic processing.

For example, computers include desktop PCs and notebooks (Note Books) as well as smart phones, tablet PCs, cellular phones, PCS phones (Personal Communication Service phones), synchronous/asynchronous A mobile terminal of International Mobile Telecommunication-2000 (IMT-2000), a Palm Personal Computer (PC), a Personal Digital Assistant (PDA), and the like may also be applicable. In addition, when a head mounted display (HMD) device includes a computing function, the HMD device may be a computer.

In addition, the computer may correspond to a server that receives a request from a client and performs information processing.

In addition, the device 10 may include an acquisition unit 110 , a memory 120 , and a processor 130 . Here, the device 10 may include fewer or more components than those illustrated in FIG. 1A .

The acquisition unit 110 may include one or more modules that enable wireless communication between the device 10 and an external server (not shown) or between the device 10 and a communication network (not shown).

Here, the acquisition unit 110 may include one or more modules for connecting the device 10 to one or more networks.

The acquisition unit 110 may acquire at least a plurality of surgical data among actual surgical data, virtual surgical data, and random surgical data. Here, the acquisition unit 110 may acquire the actual surgery data, virtual surgery data, and random surgery data from the external server (not shown).

A communication network (not shown) may transmit/receive various information between the device 10 and an external server (not shown). The communication network (not shown) may use various types of communication networks, for example, WLAN (Wireless LAN), Wi-Fi, Wibro, Wimax, HSDPA (High Speed Downlink Packet Access), etc. of wireless communication method or wired communication method such as Ethernet, xDSL (ADSL, VDSL), HFC (Hybrid Fiber Coax), FTTC (Fiber to The Curb), FTTH (Fiber To The Home) can be used.

On the other hand, the communication network (not shown) is not limited to the communication method presented above, and may include all types of communication methods that are well known or to be developed in the future in addition to the above communication methods.

The memory 120 may store data supporting various functions of the device 10 . The memory 120 may store a plurality of application programs (or applications) driven in the device 10 , data for operation of the device 10 , and commands. At least some of these application programs may exist for basic functions of the device 10 . Meanwhile, the application program may be stored in the memory 120 , installed on the device 10 , and driven by the processor 130 to perform an operation (or function) of the device 10 .

In addition, the memory 120 may store the actual surgery data, the virtual surgery data, and the random surgery data acquired by the acquisition unit 110 for each patient.

In addition, the memory 120 may include a plurality of processes for collecting learning data for a surgical tool and organ recognition learning training according to the present invention. Here, the plurality of processes will be described later when an operation of the processor 130 is described.

In addition to the operation related to the application program, the processor 130 may generally control the overall operation of the device 10 . The processor 130 processes signals, data, information, etc. input or output through the above-described components or drives an application program or a plurality of processes stored in the memory 120 to provide or process appropriate information or functions to the user. can do.

In addition, the processor 130 may control at least some of the components discussed with reference to FIG. 1A in order to drive an application program or a plurality of processes stored in the memory 120 . Furthermore, the processor 130 may operate by combining at least two or more of the components included in the device 10 to drive the application program or the plurality of processes.

Referring to FIG. 1B , the method for collecting learning data for surgical tool and organ recognition learning training of the present invention is a method executed by the processor 130 of the device 10, wherein the processor 130 includes at least a plurality of surgical data A step of acquiring (S110), after acquiring the surgical data, generating semantic image data based on the acquired plurality of surgical data through a segmentation model (S120), the acquired plurality of surgical data and the semantic It may include a step (S130) of providing image data by configuring it as a learning data set.

The processor 130 may acquire at least a plurality of surgical data (S110).

Here, the processor 130 may acquire at least a plurality of surgical data among actual surgical data, virtual surgical data, and random surgical data based on a first process among the plurality of processes.

Here, the actual surgical data may include actual surgical tool data including the type of surgery actually performed, location information of the used tool, and information on the used surgical tool, and actual organ data on the organ of the actually performed surgery. there is.

Also, the actual surgical data may be obtained from the actual surgical image data, and the actual surgical image data may be image data obtained during an actual operation or during an actual operation.

In addition, as described above, the actual surgical data includes the type of operation in which the corresponding operation was performed and the location information of the tool used during the actual operation. Actual surgical organ data may be included.

The processor 130 may generate the actual surgical data based on the actual surgical image data through a segmentation model. Here, the split model will be described later in detail.

Here, the actual surgical tool data is bound within the actual surgical image by dividing at least one of a head, a joint, and a body of the tool through the segmentation model, or a polygon in a pixel unit. It may include at least one of those to be processed.

Alternatively, the actual surgical tool data may include at least one of bounding within the actual surgical image for each feature of the tool through the segmentation model or processing polygons in units of pixels.

Here, the features of the tool include all features such as the type of tool and the size of the tool, and may be divided and bound for each feature through the division model, or may be polygon-processed in units of pixels. Such polygon processing may be a method of tracing an outline for a specific portion in units of pixels.

In addition, the actual surgery organ data may be polygon-processed at least one or more of the pixels including the organs visually distinguished in the actual surgery image through the segmentation model. Alternatively, the actual surgery organ data may include at least one of bounding a specific part including the organ in the actual surgery image through the segmentation model.

Bounding in the actual surgical image may be annotating the part to be specified in the actual surgical image, that is, the part corresponding to the information to be learned, in the form of a bounding box through the segmentation model. .

Alternatively, the polygon processing in the actual surgical image may be a pixel-by-pixel processing of a part to be specified in the actual surgical image, that is, a part corresponding to information to be learned through the segmentation model.

Here, the annotation may be a correct answer label including a bounding box or a polygon-treated specific part.

Referring to FIG. 2A , a method of annotating a surgical tool by displaying it as a bounding box in a surgical image is not only performed within the actual surgical image, but also performed in a 3D surgical environment or an image of a 3D virtual space. The same can be applied.

The processing of the bounding box by dividing the surgical tool into a head, a joint, and a body is to obtain accurate position information of the tool. It is possible to obtain location and direction information.

However, the surgical tool may be divided into only a head and a body, or may be divided into a head, a joint, and a body according to kinematic characteristics.

Surgical tools may include, for example, laparoscopic surgical tools and robotic surgical tools, as well as needles, specimen bags, and surgical tubes frequently used in the surgical process, and all tools used during surgery other than the examples listed above are included. can

The virtual surgery data may include virtual surgical tool data and virtual surgical organ data used in the virtual surgical environment and the virtual surgical environment. At this time, the virtual surgical environment is a 3D surgical environment produced by 3D using specific real surgical image data, and the virtual surgical tool data and virtual surgical organ data are virtual surgical environments derived from users within the 3D surgical environment. of surgical tools.

The processor 130 may generate the virtual surgical data based on the virtual surgical environment through the segmentation model. Here, the split model will be described later in detail.

Such a virtual surgical environment may be an environment implemented through a virtual body model. Here, the virtual surgical environment may be the following digital twin virtual environment for effective labeling distribution for surgical tools and long-term education.

In addition, the virtual body model can be created by using a label for segmenting five reconstructed organs using a computed tomography image of the abdomen in consideration of the surgical situation and using the skin reflecting the patient's peritoneum for 3D reconstruction.

In addition, a virtual surgical environment can be created by reconstructing a 3D model using commercially available software tools 3D Max, Zbrush, and Substance Painter using catalogs and actual measurements of robotic and laparoscopic instruments in the case of surgical tools.

In addition, the virtual surgical environment may be created by building a virtual camera environment with the same internal/external parameters as the endoscope used in robotic surgery to reproduce the camera viewpoint of the surgery.

A 3D surgical environment produced by 3D using actual surgical image data can be produced by projecting from 2D information corresponding to a specific camera viewpoint.

Here, the actual surgical image data is exemplarily computed tomography (CT), nuclear magnetic resonance computed tomography (NMR-CT), positron emission tomography (PET). ), CBCT (Cone Beam CT, CBCT), Electron Beam Tomography (EBT), X-ray image, Magnetic Resonance Imaging (MRI), but limited to the above examples It is not possible to include all image data taken during surgery, including video.

The virtual surgical tool data and the virtual surgical organ data may be information of a surgical tool and a surgical organ that are arbitrarily used by a user within the configured 3D surgical environment.

The processor 130 annotates a surgical tool or a surgical organ that is used virtually even if it is a surgical tool that is not used in actual surgery as a bounding box through the segmentation model, or by processing polygons in pixel units to learn training using learning data can be performed.

However, in the case of virtual surgical tool data and virtual surgical organ data, a technique of image-to-image translation using a generative adversarial neural network may be applied and utilized for learning.

The generative adversarial network will be described later along with the description of the deep neural network.

Random surgical data is virtual data generated from domain randomization, and by randomly changing the setting conditions of a surgical tool, a surgical target organ, and a camera, a specific surgery, a specific surgery target organ, and a specific It may include data about at least one of the random surgical tools used in surgery.

The processor 130 may generate the random surgical data based on the virtual surgical environment through a segmentation model. Here, the split model will be described later in detail.

That is, the processor 130 annotates a surgical tool or a surgical organ as a bounding box through the segmentation model for random surgical data, which is virtual data generated from domain randomization, or to process polygons in units of pixels. can

The 3D virtual space implemented to acquire random surgical data may include a 3D virtual space of an arbitrary surgery, and may include a 3D surgical environment manufactured in the same way as the 3D surgical environment included in the aforementioned virtual surgical data. there is.

In the case of an organ to be operated, the color, texture, deformation, and position of the organ can be set randomly. And, in the case of a camera, the field of view (FOV), position, type, and distortion of the camera can be randomly set.

That is, the processor 130 may randomly set all surgical target organs, surgical tools, camera conditions, and the like, irrespective of surgical tools generally used in the existing actual surgery, and convert the randomly acquired information into data.

For example, when acquiring random surgical data, one or more robotic tools (eg, Harmonic Ace, Maryland Bipolar Forceps, Cadiere Forceps, Stapler, Medium-Large Clip Applier, Small Clip Applier, etc.), one or more laparoscopic tools (eg, Atraumatic grasper) , Electric hook, Curved Atraumatic Grasper, Suction-Irrigation, Clip Applier (metal), Scissors, Overholt, Ligasure, etc.) and one or more organs can produce 3D models. However, the present invention is not limited to the above embodiment, and 3D models may be produced for various robot tools and various laparoscopic tools.

Alternatively, when acquiring random surgical data, tools used for gastrectomy surgery (eg, atraumatic grasper, Cadiere forceps, curved atraumatic grasper, harmonic ace, Maryland bipolar forceps, medium-large clip applier, Overholt, scissors, small clip applier, stapler, and suction-irrigation) and one or more laparoscopic tools (e.g., atraumatic grasper, clip applier (hem-o-lok), clip applier (metal), curved atraumatic grasper, electrichook, Ligasure, Overholt, scissors, and suction-irrigation) ) can create a 3D model. However, the present invention is not limited to the above embodiment, and 3D models may be produced for various robot tools and various laparoscopic tools.

The processor 130 may generate semantic image data based on at least a plurality of surgical data acquired through the segmentation model (S120).

Here, the segmentation model may include a semantic segmentation model and an instance segmentation model. The semantic segmentation model may include at least one of DeepLabV2, HRNet, and HRNet+OCR. In addition, the instance partitioning model may include at least one of Casecade R-CNN, HRNet, and HRNet+OCR.

Here, the semantic segmentation model may infer all pixels existing in the image, and the instance segmentation model may infer instance information in the bounding box at the pixel level.

In more detail, the processor 130 may generate the semantic image data from each of the frames extracted from the actual surgical data, the virtual surgical data, and the random surgical data through the segmentation model and a label map corresponding thereto.

Here, the semantic image data may include semantic surgical tool data for tools included in each of the plurality of surgical data or semantic surgical organ data for organs.

In addition, the semantic surgery tool data or the semantic surgery organ data may include at least one of bound to the tool or the organ included in each of the plurality of surgical data, or polygon-processed in units of pixels.

And, the semantic surgery tool data or the semantic surgery organ data is the location of the tool or organ, the name of the tool or organ, the state of the tool or the organ for each bounded or polygon-processed part (for example, , a tool: a state in which the joint part is folded, an organ: a state in which a part is cut, etc.).

Specifically, referring to FIG. 2B , when the actual surgical data is included in the plurality of surgical data, the processor 130 may generate actual semantic image data based on the actual surgical data through the segmentation model.

In addition, referring to FIG. 2C , when the virtual surgical data is included in the plurality of surgical data, the processor 130 may generate virtual semantic image data based on the virtual surgical data through the segmentation model.

In addition, referring to FIG. 2D , when the random surgical data is included in the plurality of surgical data, the processor 130 may generate random semantic image data based on the random surgical data through the segmentation model.

The processor 130 may configure and provide a plurality of acquired surgical data and the semantic image data as a learning data set (S130). Here, the subject receiving the learning data set may include a user, a computer, a system, etc. utilizing the corresponding learning data.

In more detail, the processor 130 may configure the plurality of surgical data and the semantic image data for each of the plurality of surgical data as a learning data set based on a third process among the plurality of processes to provide the training data set.

Accordingly, since there is a shortage of actual surgical images and only tools used in actual surgery are highly likely to be used, the learning performance can be improved by additionally adding virtual surgical data and random surgical data, and additionally adding semantic image data. .

As an example, the processor 130 may configure the actual surgical data and semantic image data data for the actual surgical data as a learning data set and provide them.

3 is a flowchart of a method for generating a surgical tool recognition learning model using the learning data set of the present invention.

Figure 3 further includes the step of performing the surgical tool recognition learning training in the method of Figure 1b (S150) and generating the surgical tool recognition learning model (S170).

The step of performing the surgical tool recognition learning training (S150) is to perform the surgical tool recognition learning training with respect to a specific surgical tool used for a specific surgery based on the learning data set.

At this time, the learning data set is obtained from the above-described actual surgery data, virtual surgery data, and/or random surgery data, and the obtained learning data set is at least one of the surgical type, the size of the organ, the condition of the organ, and the age of the operation target. It can be divided into one.

In addition, the learning data may be acquired by balancing the class as data of a specific surgical tool for a specific surgery, respectively, as a uniform amount of data.

Specific surgical tools may include, by way of example, not only laparoscopic surgical tools and robotic surgical tools, but also needles, specimen bags, surgical tubes, etc. frequently used in the surgical process. In addition to the examples listed above, tools used during surgery are can all be included.

As a specific example, in the case of laparoscopic tools and auxiliary surgical tools used for cholecystectomy, Atraumatic grasper, Electric hook, Curved Atraumatic Grasper, Suction-Irrigation, Clip Applier (metal), Scissors, Overholt, Ligasure, Clip Applier (Ham) -o-Lok), Specimen Bag may be applicable. Here, in the case of the cholecystectomy, the surgical organ may include a gallbladder and a liver.

In the case of gastrectomy, it may include robotic tools, laparoscopic tools, auxiliary tools, Harmonic Ace, Maryland Bipolar Forceps, Cadiere Forceps, Medium-Large Clip Applier, Small Clip Applier, Curved Atraumatic Grasper, Stapler, Suction, Suction- Irrigation, Scissors, Atraumatic Grasper, Overholt, Guaze Needle, Endotip, Drain Tube, Needle, Needle Holder, Specimen Bag may be applicable. Here, in the case of gastrectomy, the surgical organ may include a stomach, a liver, a gallbladder, a spleen, and a pancreas.

Specifically, the robot tool may include Harmoic Ace, Maryland Bipolar Forceps, Cadiere Forceps, Medium-large Clip Applier, Small Clip Applier, etc., and the laparoscopic tool may include Curved Atraumatic Grasper, Stapler, Suction, etc., and the Surgical auxiliary tools may include Guaze Needle, Endotip, Specimenbag, Drain Tube, and the like.

The step of generating the surgical tool recognition learning model ( S170 ) is to generate a surgical tool and organ recognition learning model based on the performed surgical tool recognition learning training.

A deep neural network (DNN) according to embodiments of the present invention refers to a system or network that constructs one or more layers in one or more computers and performs judgment based on a plurality of data. For example, a deep neural network may be implemented with a set of layers including a convolutional pooling layer, a locally-connected layer, and a fully-connected layer. A convolutional pooling layer or a local connection layer may be configured to extract features in an image. A fully connected layer can determine the correlation between features of an image. In some embodiments, the overall structure of the deep neural network may consist of a convolutional pooling layer followed by a local access layer, and a fully connected layer connected to the local access layer. The deep neural network may include various judgment criteria (ie, parameters), and may add new judgment criteria (ie, parameters) through input image analysis.

The deep neural network according to embodiments of the present invention is a structure called a convolutional neural network suitable for image analysis, and a feature extraction layer that learns by itself the feature with the greatest discriminative power from given image data. ) and a prediction layer that learns a predictive model to obtain the highest predictive performance based on the extracted features may have an integrated structure.

The feature extraction layer spatially integrates the convolution layer, which creates a feature map by applying a plurality of filters to each region of the image, and the feature map to obtain features that are invariant to changes in position or rotation. It may be formed in a structure in which a pooling layer that can be extracted is alternately repeated several times. Through this, various level features can be extracted from low-level features such as points, lines, and planes to complex and meaningful high-level features.

The convolutional layer obtains a feature map by taking a nonlinear activation function on the dot product of the filter and the local receptive field for each patch of the input image. In comparison, CNNs are characterized by using filters with sparse connectivity and shared weights. Such a connection structure reduces the number of parameters to be learned, makes learning through the backpropagation algorithm efficient, and consequently improves prediction performance.

The integration layer (Pooling Layer or Sub-sampling Layer) generates a new feature map by using local information of the feature map obtained from the previous convolutional layer. In general, the newly created feature map by the integration layer is reduced to a smaller size than the original feature map. Representative integration methods include Max Pooling, which selects the maximum value of the corresponding region in the feature map, and the corresponding feature map in the feature map. There is an average pooling method that obtains the average value of a region. In general, the feature map of the integrated layer can be less affected by the location of arbitrary structures or patterns present in the input image than the feature map of the previous layer. That is, the integration layer can extract features that are more robust to regional changes such as noise or distortion in the input image or previous feature map, and these features can play an important role in classification performance. Another role of the integration layer is to reflect the features of a wider area as you go up to the upper learning layer in the deep structure. More and more abstract features can be generated that reflect the features of the entire image.

In this way, the features finally extracted through iteration of the convolutional layer and the integration layer are fully connected to a classification model such as a multi-layer perception (MLP) or a support vector machine (SVM). -connected layer) and can be used for classification model training and prediction.

However, the structure of the deep neural network according to the embodiments of the present invention is not limited thereto, and may be formed of a neural network of various structures.

The generative adversarial neural network learning model is unsupervised learning and consists of two models: a discriminator D (Discriminator) responsible for classification and a generator G (Generator) that generates data from random noise.

The generative adversarial neural network learning model is a model in which the generator G and the discriminator D oppose each other and improve each other's performance. It is a model in which the performance of both models increases together by generating fake data to prevent this from happening.

Generator G tries to find out the probability distribution of the original data and reproduces the distribution so that the fake data does not differ from the real data. Estimating the probability for each.

The formula of the generative adversarial neural network learning model is as Equation 1 below.

[Equation 1]

4 is a flowchart of a method for acquiring insufficient data as balance data in actual surgical data.

5 is a graph showing the distribution of annotations obtained from actual surgical data according to the present invention.

6 is a graph showing the distribution of annotations changed by acquiring virtual surgical data and/or random surgical data according to the present invention.

7 is a graph illustrating the distribution of annotations obtained from actual surgical data (laparoscopic cholecystectomy and gastric cancer video) according to the present invention.

8 is a graph explaining the distribution of annotations obtained from laparoscopic cholecystectomy and gastrectomy for actual surgical data (gastric cancer surgery data) and virtual surgery data (3D patient model) according to the present invention.

Referring to FIG. 4, the learning data collection method for the surgical tool recognition learning training of the present invention includes the steps of acquiring actual surgical data (S210), extracting the type and required amount of data lacking annotation (S230) and and acquiring data corresponding to the type of extracted data as balanced data (S250).

The actual surgery data, virtual surgery data, and random surgery data in FIGS. 4 to 9 are the same as those described above in FIGS. 1A to 3 .

The step of extracting the type and required amount of data lacking annotation ( S230 ) is to extract the type and required amount of data lacking annotation that causes class imbalance in the acquired actual surgical data.

Since the tools used in actual surgery may have limitations/limitations, there may be movements that are not performed in actual surgery, and the tools and surgical operations may be completely different for each actual surgical case, all tools are used in actual surgery and it is difficult to obtain data on the operation. For this reason, when only actual surgical data is acquired, class imbalance may occur.

In the present invention, class imbalance that occurs only with actual surgical data is solved by acquiring virtual surgical data and/or random surgical data. Accordingly, by extracting the type and required amount of data generating class imbalance from the acquired actual surgical data, the required data type as much as the required amount may be obtained from the virtual surgery data and/or random surgery data.

The step of extracting the type and required amount of data lacking annotation ( S230 ) is to obtain the data corresponding to the type of extracted data as balanced data by the required amount from at least one of the virtual surgery data and the random surgery data.

Balanced data refers to additional data obtained from virtual surgery data and/or random surgery data in order to resolve class imbalance caused only by actual surgical data.

The type of data lacking annotations may include at least one of a surgical operation, a surgical tool type, a specific surgical location, and a surgical tool use location.

In this embodiment, the actual surgical data includes real surgical tool data including the type of surgery actually performed, location information of a used tool, and information on a used surgical tool, and the virtual surgery data includes a virtual surgery It includes the virtual surgical tool data used in the environment and the virtual surgical environment, and the random surgical data is virtual data generated from domain randomization, and each data is described above in FIGS. 1A to 3 . Same as content

Referring to FIG. 5, FIG. 5 shows the distribution of actual surgical data classified by each surgical tool. bottom graph), it can be seen that the obtained data itself is unbalanced, causing class imbalance.

Referring to FIG. 6, FIG. 6 shows the distribution of annotations changed by acquiring virtual surgical data and/or random surgical data, which are values of data obtained by summing actual surgical data and balance data, classified by each surgical tool. will be. It can be seen that the sum of actual surgical data and balance data for cholecystectomy (upper graph in Fig. 6) and the sum of actual operational data and balance data for gastrectomy (lower graph in Fig. 6) are balanced.

Therefore, as a result, according to the present invention, the problem of class imbalance is solved, and a learning result using the corresponding data is derived as a more accurate result.

Referring to FIG. 7 , FIG. 7 may represent the distribution of annotations obtained from actual surgical data (laparoscopic cholecystectomy and gastric cancer video).

Here, the number of annotations may be adjusted on a logarithmic scale. It can be confirmed that both cholecystectomy and gastrectomy, which are actual surgical data, cause serious imbalance problems in some tools.

Referring to FIG. 8, FIG. 8 may show the distribution of annotations obtained from laparoscopic cholecystectomy and gastrectomy for actual surgical data (gastric cancer video) and 3D patient model.

Compared with the distribution of annotations obtained from actual surgical data (eg, gastric cancer surgery data), the class imbalance problem is class imbalance in both of the above two surgeries (laparoscopic cholecystectomy, gastrectomy) in the actual surgical data, virtual surgery data, and random surgical data distribution. It can be seen that the problem has been alleviated.

9 is a flowchart of a method of generating a surgical tool recognition learning model based on the acquired actual surgical data and balance data.

Referring to FIG. 9 , the method of generating a surgical tool recognition learning model further includes performing surgical tool recognition learning training in the step of FIG. 6 ( S270 ) and generating a surgical tool recognition learning model ( S290 ) do.

The step of performing the surgical tool recognition learning training (S270) is to perform the surgical tool recognition learning training with respect to a specific surgical tool used for a specific surgery using the acquired balance data and actual surgical data as learning data.

The step of generating the surgical tool recognition learning model ( S290 ) is to generate a surgical tool recognition learning model based on the performed surgical tool recognition learning training.

The step of performing the surgical tool recognition learning training of FIG. 9 ( S270 ) and the step of generating the surgical tool recognition learning model ( S290 ), the step of performing the surgical tool recognition learning training of FIG. 3 ( S150 ) and the surgical tool recognition learning Only the step of generating the model ( S170 ) and the training data are different, and the performing method is applied the same.

The steps of a method or algorithm described in relation to an embodiment of the present invention may be implemented directly in hardware, as a software module executed by hardware, or by a combination thereof. A software module may include random access memory (RAM), read only memory (ROM), erasable programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory, hard disk, removable disk, CD-ROM, or It may reside in any type of computer-readable recording medium well known in the art to which the present invention pertains.

As mentioned above, although embodiments of the present invention have been described with reference to the accompanying drawings, those skilled in the art to which the present invention pertains know that the present invention may be embodied in other specific forms without changing the technical spirit or essential features thereof. you will be able to understand Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive.

10: device
110: communication department
120: memory
130: processor

Claims (13)

A learning data collection method for surgical tool and organ recognition learning training performed by a device, the method comprising:
Acquiring at least a plurality of surgical data of actual surgical data, virtual surgery data, and random surgery data; and
Comprising the step of providing the obtained plurality of surgical data to configure as a learning data set,
The actual surgical data includes actual surgical tool data including the type of surgery actually performed, location information of the used tool, and information on the used surgical tool, and actual surgical organ data for the organ of the actually performed surgery. ,
The virtual surgical data includes a virtual surgical environment, virtual surgical tool data used in the virtual surgical environment, and virtual surgical organ data for organs in the virtual surgical environment,
The random surgical data is virtual data generated from domain randomization,
Learning data collection methods for training surgical instruments and organ recognition learning. According to claim 1,
performing surgical tool recognition learning training for a specific surgical tool used for a specific surgery based on the learning data set; and
Further comprising the step of generating a surgical tool recognition learning model based on the performed surgical tool recognition learning training,
Learning data collection methods for training surgical instruments and organ recognition learning. According to claim 1,
The actual surgical tool data is bound within the actual surgical image by dividing at least one of a head, a wrist, and a body of the tool, or polygon-processed in units of pixels,
Or comprising at least one of bounding within the actual surgical image for each feature of the tool or processing the polygon in units of pixels,
The actual surgery organ data is at least one of at least one of the pixels including the visually distinguished organs in the actual surgery image is polygon-processed, or a specific part including the organ in the actual surgery image is bound. which includes one,
Learning data collection methods for training surgical instruments and organ recognition learning. According to claim 1,
The virtual surgical environment is a 3D surgical environment created based on specific actual surgical image data,
The virtual surgical tool data and the virtual surgical organ data are virtual surgical tool information and surgical organ information derived by being utilized by a user within the 3D surgical environment,
Learning data collection methods for training surgical instruments and organ recognition learning. According to claim 1,
The random surgical data is,
As virtual data generated from domain randomization, by randomly changing the setting conditions of a surgical tool, a surgical target organ, and a camera, a specific surgery randomly generated, a specific surgery target organ, and random used in a specific surgery comprising data about at least one of the surgical instruments;
Learning data collection methods for training surgical instruments and organ recognition learning. According to claim 1,
The learning data is
Data of a specific surgical tool and a specific surgical organ for a specific surgery are each obtained with a class balance as a uniform amount of data,
Learning data collection methods for training surgical instruments and organ recognition learning. According to claim 1,
After acquiring the plurality of surgical data, generating semantic image data for the acquired plurality of surgical data through a segmentation model; further comprising,
The step of providing the learning data set configuration,
Providing the plurality of surgical data and the semantic image data corresponding to each of the plurality of surgical data as a learning data set,
Learning data collection methods for training surgical instruments and organ recognition learning. 8. The method of claim 7,
The semantic image data generation step includes:
When the actual surgical data is included in the plurality of surgical data,
Generate actual semantic image data based on the actual surgical data through the segmentation model,
When the virtual surgical data is included in the plurality of surgical data,
Generate virtual semantic image data based on the virtual surgical data through the segmentation model,
When the random surgical data is included in the plurality of surgical data,
Generating random semantic image data based on the random surgical data through the segmentation model,
Learning data collection methods for training surgical instruments and organ recognition learning. 9. The method of claim 8,
The semantic image data is
including semantic surgical tool data for the tool or semantic surgical organ data for the organ included in each of the plurality of surgical data,
The semantic surgery tool data or the semantic surgery organ data includes at least one of a location range of the tool or the organ, and a name of the tool or the organ.
Learning data collection methods for training surgical instruments and organ recognition learning. A learning data collection method for surgical tool recognition learning training performed by a device, the method comprising:
acquiring actual surgical data;
extracting a type and a required amount of data lacking an annotation for generating a class imbalance from the acquired actual surgical data;
Comprising the step of acquiring data corresponding to the type of data extracted by the required amount from at least one of virtual surgery data and random surgery data as balanced data,
The actual surgical data includes actual surgical tool data including information on the type of surgery actually performed, location information of a used tool, and information on a used surgical tool,
The virtual surgical data is to include a virtual surgical environment and virtual surgical tool data used in the virtual surgical environment,
The random surgical data is virtual data generated from domain randomization,
Learning data collection methods for training surgical instruments and organ recognition learning. 11. The method of claim 10,
performing surgical tool recognition learning training on a specific surgical tool used for a specific surgery using the acquired balance data and the actual surgical data as learning data; and
Comprising the step of generating a surgical tool recognition learning model based on the performed surgical tool recognition learning training,
Learning data collection methods for training surgical instruments and organ recognition learning. 12. The method of claim 2 or 11,
If the specific operation is gastrectomy,
The specific surgical tool includes at least one of a robotic surgical tool, a laparoscopic surgical tool, and an auxiliary surgical tool,
The specific surgical organs include stomach, liver, gallbladder, spleen and pancreas,
If the specific operation is cholecystectomy,
The specific surgical tool includes at least one of a laparoscopic tool and an auxiliary surgical tool,
The specific surgical organ, including the gallbladder and liver,
Learning data collection methods for training surgical instruments and organ recognition learning. 13. A learning data collection program for surgical tools and organ recognition learning training, stored for executing the method of any one of claims 1 to 12 in combination with a computer which is hardware.

Images