WO2019164274A1

WO2019164274A1 - Training data generation method and device

Info

Publication number: WO2019164274A1
Application number: PCT/KR2019/002092
Authority: WO
Inventors: 이종혁; 형우진; 양훈모; 김호승; 허성환; 최민국; 이재준
Original assignee: (주)휴톰
Priority date: 2018-02-20
Filing date: 2019-02-20
Publication date: 2019-08-29

Abstract

A relay server for constructing surgical information is provided. The relay server is for relaying communication between clients, and receives actual surgical information generated on the basis of an actual surgical image from a first client, transmits the received actual surgical information to a second client, receives virtual surgical information generated by performing a virtual surgery on the basis of the actual surgical information from the second client, and transmits the virtual surgical information to the first client.

Description

Method and device for generating learning data

The present invention relates to a method and apparatus for generating learning data.

Medical surgery can be classified into open surgery, laparoscopic surgery, minimally invasive surgery (MIS) including radio surgery, radio surgery, and the like. Laparoscopic surgery refers to surgery performed by medical staff to see and touch the part to be treated. Minimally invasive surgery is also known as keyhole surgery, and laparoscopic surgery and robotic surgery are typical. In laparoscopic surgery, a small hole is made in a necessary part without opening, and a laparoscopic with a special camera is attached and a surgical tool is inserted into the body and observed through a video monitor. Microsurgery is performed using a laser or a special instrument. In addition, robot surgery is to perform minimally invasive surgery using a surgical robot. Furthermore, radiation surgery refers to surgical treatment with radiation or laser light outside the body.

In the case of such a medical operation, a surgical image is often obtained at the time of the actual operation and the surgery is performed based on this. Therefore, it is important to provide a variety of information through the surgical image obtained at the time of surgery.

In addition, the development of technologies that can provide information to assist the surgeon in the surgery process is required. In order to provide information to assist the operation, it is important to recognize the operation performed in the operation process or various surgical information and to understand the meaning of the recognized information. Therefore, the development of a technology that allows the computer to recognize the operation or a variety of surgical information from the surgical image is required.

In recent years, deep learning has been widely used to analyze medical images. Deep learning is defined as a set of machine learning algorithms that attempts to achieve high levels of abstraction (summarizing key content or functions in large amounts of data or complex data) through a combination of several nonlinear transformations. Deep learning can be seen as a field of machine learning that teaches computers how people think in a large framework.

When learning using such deep learning and machine learning, a large amount of learning data is required. However, since the surgical image is obtained during the actual surgical procedure, the amount of data is small and it is difficult to secure various types of surgical images.

On the other hand, in recent years, the surgery is performed using a variety of methods as well as the case where the medical staff directly perform the surgery. For example, a robotic surgery system using a surgical robot, a virtual surgery system that provides simulation of virtual surgery by providing the same virtual environment as in actual surgery, and a remote surgery system that can guide surgery remotely during surgery. Method can be used. Accordingly, there is a need for a method that allows the operation process to proceed by using various methods during actual surgery.

In addition, conventional scenarios for optimizing the surgical process were referred to previously taken medical images or consulted by highly skilled doctors, but medical images alone made it difficult to judge unnecessary processes. It was difficult to get the right advice. Therefore, medical imaging and the advice of an experienced doctor were often difficult to use as an aid for the optimization of the surgical process for the patient.

Thus, by optimizing the surgical process by minimizing unnecessary processes in performing surgery using a three-dimensional medical image (for example, a virtual image of the internal movement caused by the movement of the three-dimensional surgical instruments and tools) And, there is a need for the development of a method that can provide surgical assistance information based on this.

In particular, in the case of a three-dimensional medical image including blood vessels, by accurately providing the location and type of each blood vessel, it is possible to grasp the overall hierarchy of blood vessels and the flow of blood flow, etc. to provide optimized surgical information during surgery or simulation. How to do it.

The problem to be solved by the present invention is to provide a surgical information construction method and apparatus.

The problem to be solved by the present invention is to provide an artificial data generating method and apparatus.

An object of the present invention is to provide a method and apparatus for generating artificial data from an actual surgical image.

The problem to be solved by the present invention is to provide a method and apparatus for generating artificial data close to the actual surgical image through learning.

An object of the present invention is to provide a method and apparatus for generating learning data based on a surgical image.

An object of the present invention is to provide a method and apparatus for recognizing various surgical information included in a surgical image and expressing a relationship between the recognized surgical information.

The problem to be solved by the present invention is to provide a method and apparatus for generating training data based on each surgical information recognized from the surgical image, and to build a learning model using the generated training data.

An object of the present invention is to provide a method and apparatus for providing blood vessel information using a blood vessel model.

The problem to be solved by the present invention is to provide a method and apparatus for constructing the training data using the polygon information in the 3D blood vessel model and learning to classify blood vessels.

The problem to be solved by the present invention is to provide a method and apparatus for identifying the vessel type and vascular hierarchy for the vessel information of the new patient through the vessel classification learning model.

Problems to be solved by the present invention are not limited to the above-mentioned problems, and other problems not mentioned will be clearly understood by those skilled in the art from the following description.

The relay server for constructing surgery information according to an embodiment of the present invention, while relaying the communication between the client, receives the actual surgery information generated based on the actual surgery image from the first client, the received actual surgery information Transmits the virtual surgery information generated by performing a virtual surgery based on the actual surgery information from the second client, and transmits the virtual surgery information to the first client.

A method of generating artificial data performed by a computer according to an embodiment of the present invention includes obtaining a background image and an object image, and generating artificial data by combining the background image and the object image. The background image includes a specific area inside the body taken by the endoscope camera, and the object image includes a surgical tool or blood, and the artificial data includes a surgical tool or blood in the object image, and the It is characterized in that it is generated to approximate the actual surgical image based on the arrangement relationship between specific regions within the body in the background image.

According to an embodiment of the present invention, a method for generating training data based on a surgical image performed by a computer may include obtaining a surgical image including a plurality of image frames, and performing surgery recognition information from each of the plurality of image frames. Recognizing, and generating, for each of the plurality of image frames, relational representation information representing a relationship between surgical elements included in the surgical recognition information based on the surgical recognition information. Include.

In a method of providing blood vessel information using a vessel model performed by a computer according to an embodiment of the present invention, the method may include obtaining vessel vessel polygons constituting a vessel in a vessel model, the vessel vessel information and connection information of the vessel vessel. Comprising the learning data based on the step, and classifying the blood vessels based on the learning data.

According to an embodiment of the present invention, surgical information can be collected from various clients in real time at the time of an actual operation, so that surgical medical staff can effectively proceed with the surgical procedure. In addition, according to an embodiment of the present invention, since the operation can be performed through a variety of surgical information and surgical system as compared to the case of performing the operation only by the surgical medical staff, it is possible to perform a more accurate operation.

According to the embodiment of the present invention, rather than transmitting and receiving the surgical information for each client, it is possible to reduce the communication load because the surgical information can be transmitted and received to each other through the relay server. In addition, since each client only needs to perform communication through a relay server, even if the number of clients increases, communication between each other can be easily performed, and an additional load does not occur. Thus, increasing the number of clients does not affect performance.

According to the embodiment of the present invention, since all the surgical information can be constructed through the relay server, it can be used as learning data later.

According to the present invention, artificial data close to the actual surgical image can be generated through learning.

According to the present invention, learning data can be constructed using artificial data.

According to the present invention, a sufficient learning effect can be obtained by providing artificially generated surgical images together with actual surgical images, such as artificial intelligence, deep learning, and machine learning, which require a large amount of learning data.

According to the present invention, an artificially generated surgical image including various surgical information that cannot be obtained during actual surgery can be secured.

According to the present invention, it is possible to obtain more meaningful information through the image frame by defining various surgical elements that can be recognized from the image frame, and grasping the relationship information between the surgical elements as well as the information on each surgical element itself. .

According to the present invention, rather than generating training data for building one learning model, it provides the basis data for building various learning models. In addition, by providing such basic learning data, the learning data may be increased to obtain more improved learning results.

According to the present invention, it is possible to flexibly define various information necessary at the time of surgery through the relationship expression information according to the learning purpose, thereby performing effective learning.

According to the present invention, it is possible to accurately grasp geometric and hierarchical information of blood vessels by building a learning model for classifying blood vessels. In addition, when the simulation is performed using this, more accurate surgical simulation is possible because the vessel information about the arteries and veins located in the surgical target and the surroundings can be accurately provided.

According to the present invention, it is effective to classify blood vessels and stratify blood vessels by constructing training data based on connection information between one polygon and an adjacent polygon.

According to the present invention, by using the position information of the polygon derived based on the reference vessel point and the reference organ, it is possible to build a more accurate vessel classification learning model, through which a higher recognition rate can be obtained.

Effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the following description.

1 is a schematic diagram of a system capable of performing robot surgery according to an embodiment of the present invention.

2 is a view schematically showing the configuration of a surgical information building system according to an embodiment of the present invention.

3 is a flowchart illustrating a method of building surgery information using a relay server according to an embodiment of the present invention.

4 is a diagram schematically showing the configuration of an apparatus 300 for performing surgery information construction according to an embodiment of the present invention.

5 is a flowchart illustrating a method of generating artificial data according to an embodiment of the present invention.

6 is a view for explaining a method of generating artificial data based on learning according to an embodiment of the present invention.

7 to 10 are diagrams for explaining embodiments of a method for generating artificial data based on learning according to an embodiment of the present invention.

11 is a schematic diagram of a system capable of performing robot surgery according to an embodiment of the present invention.

12 is a diagram schematically showing the configuration of an apparatus 700 for performing a method of generating artificial data according to an embodiment of the present invention.

FIG. 13 is a schematic diagram of a system capable of performing robot surgery according to an embodiment of the present invention. FIG.

14 is a flowchart schematically illustrating a method of generating learning data based on a surgical image according to an embodiment of the present invention.

15 is a diagram illustrating a process of generating relationship expression information by recognizing surgical recognition information from an image frame according to an embodiment of the present invention.

16 is a diagram illustrating relationship expression information generated for a plurality of image frames in a surgical image according to an embodiment of the present invention.

17 is a diagram illustrating an example of a process of performing learning based on relationship expression information generated for a plurality of image frames in a surgical image according to an embodiment of the present invention.

18 is a diagram schematically illustrating a configuration of an apparatus 400 for performing a method for generating learning data based on a surgical image according to an embodiment of the present invention.

19 is a flowchart schematically illustrating a method of providing blood vessel information using a blood vessel model according to an embodiment of the present invention.

20 and 21 are diagrams for explaining a process of deriving connection information between a vascular polygon and an adjacent polygon according to an embodiment of the present invention.

22 is a diagram illustrating a process of calculating the midpoint of the OBB.

FIG. 23 is a diagram illustrating a learning process for classifying blood vessel types according to one embodiment of the present invention.

24 and 25 are diagrams showing an example of blood vessel classification derived as a learning result according to an embodiment of the present invention.

FIG. 26 is a diagram schematically illustrating a configuration of an apparatus 200 for performing a method of providing blood vessel information using a blood vessel model according to an embodiment of the present invention.

Advantages and features of the present invention and methods for achieving them will be apparent with reference to the embodiments described below in detail with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but can be embodied in various different forms, and the present embodiments only make the disclosure of the present invention complete, and those of ordinary skill in the art to which the present invention belongs. It is provided to fully inform the skilled worker of the scope of the invention, which is defined only by the scope of the claims.

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. In this specification, the singular also includes the plural unless specifically stated otherwise in the phrase. As used herein, "comprises" and / or "comprising" does not exclude the presence or addition of one or more other components in addition to the mentioned components. Like reference numerals refer to like elements throughout, and "and / or" includes each and all combinations of one or more of the mentioned components. Although "first", "second", etc. are used to describe various components, these components are of course not limited by these terms. These terms are only used to distinguish one component from another. Therefore, of course, the first component mentioned below may be a second component within the technical spirit of the present invention.

Unless otherwise defined, all terms used in the present specification (including technical and scientific terms) may be used in a sense that can be commonly understood by those skilled in the art. In addition, terms that are defined in a commonly used dictionary are not ideally or excessively interpreted unless they are specifically defined clearly.

As used herein, the term "part" or "module" refers to a hardware component such as software, FPGA, or ASIC, and the "part" or "module" plays certain roles. However, "part" or "module" is not meant to be limited to software or hardware. The “unit” or “module” may be configured to be in an addressable storage medium or may be configured to play one or more processors. Thus, as an example, a "part" or "module" may include components such as software components, object-oriented software components, class components, and task components, processes, functions, properties, Procedures, subroutines, segments of program code, drivers, firmware, microcode, circuitry, data, databases, data structures, tables, arrays, and variables. Functions provided within components and "parts" or "modules" may be combined into smaller numbers of components and "parts" or "modules" or into additional components and "parts" or "modules". Can be further separated.

Hereinafter, with reference to Figures 1 to 4, it will be described in detail with respect to the operation information building method performed by the relay server and the relay server for establishing the surgical information according to an embodiment of the present invention.

As used herein, the term "computer" includes all the various devices capable of performing arithmetic processing to provide a result to a user. For example, a computer can be a desktop PC, a notebook, as well as a smartphone, a tablet PC, a cellular phone, a PCS phone (Personal Communication Service phone), synchronous / asynchronous The mobile terminal of the International Mobile Telecommunication-2000 (IMT-2000), a Palm Personal Computer (PC), a Personal Digital Assistant (PDA), and the like may also be applicable. In addition, when a head mounted display (HMD) device includes a computing function, the HMD device may be a computer. Also, the computer may correspond to a server that receives a request from a client and performs information processing.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings.

According to FIG. 1, the robotic surgical system includes a medical imaging apparatus 10, a server 100, a control unit 30 provided in an operating room, a display 32, and a surgical robot 34. According to an embodiment, the medical imaging apparatus 10 may be omitted in the robot surgery system according to the disclosed embodiment.

In one embodiment, surgical robot 34 includes imaging device 36 and surgical instrument 38.

In one embodiment, the robot surgery is performed by the user controlling the surgical robot 34 using the control unit 30. In one embodiment, the robot surgery may be automatically performed by the controller 30 without the user's control.

The server 100 is a computing device including at least one processor and a communication unit.

The controller 30 includes a computing device including at least one processor and a communication unit. In one embodiment, the control unit 30 includes hardware and software interfaces for controlling the surgical robot 34.

The imaging device 36 includes at least one image sensor. That is, the imaging device 36 includes at least one camera device and is used to photograph an object, that is, a surgical site. In one embodiment, the imaging device 36 includes at least one camera coupled with a surgical arm of the surgical robot 34.

In an embodiment, the image photographed by the photographing apparatus 36 is displayed on the display 340.

In one embodiment, surgical robot 34 includes one or more surgical tools 38 that can perform cutting, clipping, fixing, grabbing operations, and the like, of the surgical site. Surgical tool 38 is used in conjunction with the surgical arm of the surgical robot 34.

The controller 30 receives information necessary for surgery from the server 100 or generates information necessary for surgery and provides the information to the user. For example, the controller 30 displays the information necessary for surgery, generated or received, on the display 32.

For example, the user performs the robot surgery by controlling the movement of the surgical robot 34 by manipulating the control unit 30 while looking at the display 32.

The server 100 generates information necessary for robotic surgery using medical image data of an object previously photographed from the medical image photographing apparatus 10, and provides the generated information to the controller 30.

The controller 30 displays the information received from the server 100 on the display 32 to provide the user, or controls the surgical robot 34 by using the information received from the server 100.

In one embodiment, the means that can be used in the medical imaging apparatus 10 is not limited, for example, other various medical image acquisition means such as CT, X-Ray, PET, MRI may be used.

In the disclosed embodiment, the surgical image obtained by the imaging device 36 is transmitted to the controller 30.

Hereinafter, a description will be given of the process of establishing the surgical information by using a relay server for transmitting and receiving information between the clients. The relay server may mean the server 100 of FIG. 1 or may be separately installed and operate in conjunction with the server 100 of FIG. 1.

In recent years, the medical staff may perform the surgery by various methods such as using a surgical robot or a simulator or guiding the operation remotely, as well as performing the surgery directly in the operating room. Accordingly, there is a need for a method of connecting various devices used during surgery and transmitting and receiving information therebetween to provide a more effective and optimized surgery to the patient during surgery. Therefore, the present invention is to provide a method for connecting each device using a relay server and through this to collect and utilize information.

2, the surgical information building system according to an embodiment of the present invention may include a relay server 200 and a plurality of clients. As an example, in FIG. 2, the first client 210, the second client 220, and the third client 230 are described as one example. However, the present invention is not limited thereto. . Of course, it can be configured to include fewer or more clients than shown in FIG.

The relay server 200 is a device that relays communication between a plurality of clients, and may perform a function such as, for example, a chat server. For example, the relay server 200 may relay a surgery by connecting a robot surgery system using a surgery robot, a virtual surgery system using a simulator, a remote system that guides the surgery remotely during surgery, and the like.

In addition, the relay server 200 may build learning data in response to receiving various surgical information from a plurality of clients. According to an embodiment, the relay server 200 may perform learning based on the learning data and calculate an optimized surgical process, or may provide the learning data to an external server. In this case, the external server may calculate the optimized surgical process by performing the learning based on the learning data provided from the relay server 200.

The first client 210 may be an apparatus for acquiring an actual surgery image and generating actual surgery information. In one embodiment, the first client 210 may be a device provided in the operating room (operating site). For example, it may correspond to the surgical robot 34 or the control unit 30 shown in FIG. 1, or may be provided in the operating room as a separate device and connected to the surgical robot 34 or the control unit 30.

The first client 210 may access the relay server 200 and transmit the actual surgery information generated at the time of operation to the relay server 200. In addition, the first client 210 may receive various information generated by another client from the relay server 200.

Here, the actual surgery image may mean data obtained as the actual medical staff performs the surgery. For example, it may be an image of an actual surgical scene actually performed by the surgical robot 34. In other words, the actual surgical image is data recorded on the surgical site and the operation during the actual surgical procedure.

The actual surgery information is information obtained from an actual surgery image, and may include, for example, surgical site information of a patient, surgical tool information, and information about a camera. In this case, the surgical site refers to a body part of the patient where the actual surgery is performed, and may be, for example, an organ or a blood vessel. The surgical tool may be a tool used during surgery, for example, a tool required for performing a robotic operation or a consumable used during surgery. The camera is a device for photographing an actual surgical scene. For example, the camera may be installed in an operating room to photograph a surgical scene, or may be provided in an endoscope that enters a patient's body when performing a robotic surgery to photograph an internal body.

For example, the first client 210 acquires a real surgery image from a camera or an external device included in a surgical robot, recognizes a surgical site, a surgical tool, and a camera from the acquired real surgery image, and based on the actual surgery information. Can be generated. That is, the first client 210 may generate the type and position information of the surgical site, the type, position, direction, and movement information of the surgical tool, the position and movement direction of the camera, and the like from the actual surgical image. Thereafter, the first client 210 may transmit the generated actual surgery information to the relay server 200, and provide the generated actual surgery information to other clients connected to the relay server 200.

The second client 220 may be an apparatus for generating virtual surgery information by performing virtual surgery. In one embodiment, the second client 220 may be a device provided in an operating room (operating site) or a remote location, for example, rehearsing or simulating a virtual body model modeled the same as the patient (ie, a patient). It may be a simulator capable of performing an action.

Here, the virtual surgery refers to a virtual surgery that performs rehearsal or simulation on a virtual body model that is implemented in the same manner as the physical state based on the medical image data (eg, CT, MRI, PET images, etc.) of the subject. . The virtual surgery information is information obtained based on the image data on which the rehearsal or simulation is performed on the virtual body model in the virtual space, and may include simulation data. For example, the simulation data may include surgical site information, surgical tool information, and information about a camera of a surgical subject. That is, the virtual surgery information may include all the information included in the data recorded for the surgical operation performed on the virtual body model.

In addition, the virtual body model may be 3D modeling data generated based on medical image data previously photographed inside the body of the subject. For example, the model may be modeled in accordance with the body of the surgical subject, and may be corrected to the same state as the actual surgical state.

The second client 220 may access the relay server 200 and transmit the virtual surgery information generated by performing the virtual surgery to the relay server 200. In addition, the second client 220 may receive various information generated by another client from the relay server 200.

For example, the second client 220 receives the actual surgery information generated by the first client 210 from the relay server 200 and performs virtual surgery based on the received actual surgery information to perform virtual surgery information. Can be generated.

In the case of virtual surgery, it is important to provide a virtual surgical condition that is identical to the actual surgical condition for the subject. Accordingly, in the present invention, the second client 220 receives the actual surgery information (eg, position, direction, and movement information of the surgical tool) obtained by performing the actual surgery on the first client 210 through the relay server 200. Since it can be transmitted in real time, the second client 220 can provide the surgical medical staff with more accurate virtual surgery information reflecting the actual surgery information in real time. Therefore, since the surgical medical staff can use both the information of the first client 210 and the second client 220 through the relay server 200, it is possible to effectively perform the actual surgery for the surgical subject.

The third client 230 may be a device for generating surgery guide information corresponding to the actual surgery image. In one embodiment, the third client 230 may be a device provided in the operating room (operation site) or a remote location, for example, when a specific event occurs during the operation may generate the surgical guide information in response thereto. Alternatively, another medical staff may access the third client 230 at the request of the medical staff performing the surgery in the operating room (operation site). In this case, by performing communication between the third client 230 through the relay server 200, other medical staff can provide information about the surgery to the medical staff performing the surgery in the operating room (operation site) in real time.

The third client 230 may access the relay server 200 to transmit the surgical guide information generated by the third client 230 to the relay server 200, and receive various information generated from other clients from the relay server 200. .

For example, the third client 230 may monitor the surgical situation according to the information (ie, actual surgery information and virtual surgery information) provided by the other clients 210 and 220 from the relay server 200. . In addition, the third client 230 may recognize whether a specific event occurs during the surgery based on the monitoring, and generate surgery guide information corresponding thereto. According to an embodiment, the third client 230 may automatically recognize whether a specific event occurs and generate surgery guide information determined according to the specific event. Alternatively, the medical staff may monitor the actual surgical situation through the third client 230 to recognize that a specific event occurs, and input surgery guide information corresponding thereto.

Relay server 200 according to an embodiment of the present invention, in addition to the above-described first, second, third client (210, 220, 230) can additionally connect the necessary client in accordance with the operation, and transmit and receive information between these clients It can play a role in controlling it.

On the other hand, when directly transmitting and receiving the surgical information between the client without passing through the relay server 200, since the surgical information is shared only between the client directly communicates with other clients can not obtain information about the operation process . Accordingly, there is a problem that other clients are difficult to intervene in the middle of the operation and cannot confirm the operation process in real time.

In the embodiment of the present invention to solve this problem, as described above, the relay server 200 serves to relay the communication between the client, the intermediate intervention through the relay server 200 during the actual operation to perform Can be. In addition, the relay server 200 may monitor the actual surgical process in real time. In addition, since the relay server 200 can obtain the surgical information from all the clients, it is possible to derive information about the entire surgical procedure. Accordingly, the relay server 200 may record the entire surgical procedure in order through such surgical information to form historical information about the corresponding surgery.

3 is a flowchart illustrating a method of building surgery information using a relay server according to an embodiment of the present invention. The method of FIG. 3 may be performed in the surgical information building system shown in FIG. 2.

According to an embodiment, first, the relay server 200 may generate at least one client required as a group in operation. For example, the relay server 200 may generate clients required for actual surgery as a group based on a specific patient, or may receive a group request signal directly from a client and create a group in response thereto. . Alternatively, when the relay server 200 receives an access request from the client, the relay server 200 may automatically determine whether the relay server belongs to the corresponding group and process the client to access the group.

Subsequently, the relay server 200 may transmit and receive information between clients in the group as the surgery proceeds. A detailed operation process thereof will be described below.

Referring to FIG. 3, the first client 210 may acquire an actual surgery image and generate real surgery information corresponding to the obtained actual surgery image (S100).

In one embodiment, the first client 210 obtains a real surgery image from a camera or an external device included in the surgical robot, and recognizes a surgical site, a surgical tool, and a camera from the acquired real surgery image, based on the actual surgery. Information can be generated. As described above, the actual surgical information may include surgical site information (e.g., type and location information of the surgical site), surgical tool information (e.g., type, position, direction, and movement information of the surgical tool), and camera information (e.g., camera). Location and motion information), and the like.

The first client 210 may transmit the actual surgery information to the relay server 200 (S105), and the relay server 200 may store the actual surgery information received from the first client 210 in the database ( S110).

In addition, the relay server 200 may transmit the actual surgery information to at least one client (eg, the second client 220 or / and the third client 230) in the same group (S115).

The second client 220 may perform virtual surgery to generate virtual surgery information (S120).

In an embodiment, the second client 220 receives the actual surgery information generated by the first client 210 through the relay server 200 and performs virtual surgery based on the received actual surgery information to perform virtual surgery. Information can be generated.

The second client 220 may transmit the virtual surgery information to the relay server 200 (S125), and the relay server 200 may store the virtual surgery information received from the second client 220 in a database ( S130).

In addition, the relay server 200 may transmit the virtual surgery information to at least one client (eg, the first client 210 or / and the third client 230) in the same group (S135).

The third client 230 may monitor the surgery situation based on the surgery information on the surgery subject received through the relay server 200, and may generate the surgery guide information based on the monitoring result (S140).

In an embodiment, the third client 230 generates a specific event during the surgery based on the actual surgery information of the first client 210 and the virtual surgery information of the second client 220 received from the relay server 200. Whether or not, and generate surgical guide information corresponding thereto. In another embodiment, the third client 230 may acquire the actual surgical image through the relay server 200 and provide the same to the medical staff to directly generate the surgical guide information corresponding to the actual surgical image. In this case, the third client 230 may receive the surgical guide information from the medical staff, and transmit it to the relay server 200.

The third client 230 may transmit the surgical guide information to the relay server 200 (S145), the relay server 200 may store the surgical guide information received from the third client 230 in the database ( S150).

In addition, the relay server 200 may transmit the surgical guide information to at least one client (eg, the first client 210 or / and the second client 220) in the same group (S155).

As described in the above steps S100 ~ S155, the relay server 200 acts as a relay to transmit and receive information between the clients belonging to the group as the operation performed on the operation target object, and stores the transmitted and received information in the database Can be. In addition, the relay server 200 may perform learning to build learning data and provide an optimized surgical process by using various surgical information on the operation targets stored in the database.

The relay server 200 may generate cue sheet data about a surgical procedure of a surgery target subject based on various surgery information received from a client (S160).

In an embodiment, the relay server 200 may include at least one of actual surgery information of the first client 210, virtual surgery information of the second client 220, and surgery guide information of the third client 230. Based on the cue sheet data for the surgical procedure of the surgical subject can be generated.

Here, the cue sheet data may refer to data recorded in order by dividing a specific surgical procedure into detailed surgical operations. The detailed surgical operation may be a minimum operation unit constituting the surgical process, and may be divided by various criteria. For example, the detailed operation may include the type of surgery (e.g., laparoscopic surgery, robotic surgery, etc.), the anatomical body area on which the surgery is performed, the surgical tools used, the number of surgical tools, the direction or location of the surgical tools on the screen. This may be based on the movement of the surgical tool (e.g., forward / retract, etc.). Since the relay server 200 can grasp the information on the criteria for dividing the detailed operation through the actual surgery information, the virtual surgery information, the operation guide information, and the like stored in the database, the cue sheet data corresponding to the actual surgery of the patient Can be built accurately. According to an embodiment of the present invention, the relay server 200 may form a history of the entire surgical procedure by simply generating the cue sheet data. In addition, according to an embodiment of the present invention, the relay server 200 may perform step S165.

The relay server 200 may calculate the optimized cue sheet data corresponding to the surgical process of the surgical subject by performing the learning based on the cue sheet data (S165).

In one embodiment, the relay server 200 may configure the actual surgery information, virtual surgery information, surgery guide information, and the like stored in the database as the training data, or may configure the cue sheet data itself as the training data. Thereafter, the relay server 200 may perform learning based on the learning data and calculate an optimized surgical process for the surgical target. Here, the learning method may be a machine learning method such as supervised learning, unsupervised learning, reinforcement learning, for example, deep learning learning method.

The relay server 200 may store the optimized cuesheet data in a database and provide the optimized cuesheet data to at least one client in the group. Accordingly, the at least one client may use the surgical subject's surgery based on the received optimized cuesheet data.

In this embodiment, the relay server 200 calculates the optimized cue sheet data by performing the learning based on the training data, but according to the embodiment, the relay server 200 stores the actual surgical information and the virtual surgery stored in the database. Information, surgery guide information, etc. (ie, learning data) may be provided to an external server, and learning may be performed based on the learning data in the external server.

In addition, the embodiment of the present invention may implement a virtual body model of the surgical subject based on the cue sheet data for the surgical subject generated in step S160 or / and step S165, or conversely the operation of the surgery performed on the surgical subject You can also implement an image.

Referring to FIG. 4, the processor 310 may include a connection passage (for example, a bus or the like) that transmits and receives signals with one or more cores (not shown) and a graphic processor (not shown) and / or other components. ) May be included.

The processor 310 according to an embodiment executes one or more instructions stored in the memory 320 to perform the surgery information construction method described with reference to FIGS. 2 and 3.

For example, the processor 310 receives the actual surgery information on the actual surgery image from the first client, transmits the actual surgery information to the second client by executing one or more instructions stored in the memory 320, and the second client. The virtual surgery information generated based on the actual surgery information may be received from the client, and the actual surgery information and the virtual surgery information may be stored.

Meanwhile, the processor 310 may read random access memory (RAM) and read-only memory (ROM) for temporarily and / or permanently storing a signal (or data) processed in the processor 310. , Not shown) may be further included. In addition, the processor 310 may be implemented in the form of a system on chip (SoC) including at least one of a graphic processor, a RAM, and a ROM.

The memory 320 may store programs (one or more instructions) for processing and controlling the processor 310. Programs stored in the memory 320 may be divided into a plurality of modules according to their functions.

The surgical information building method according to an embodiment of the present invention described above may be implemented as a program (or an application) and stored in a medium to be executed in combination with a computer which is hardware.

Hereinafter, a method and apparatus for generating artificial data according to an embodiment of the present invention will be described in detail with reference to FIGS. 5 to 12.

Although the method of FIG. 5 is described as being performed by a computer for the convenience of description, the subject of each step is not limited to a specific device but may be used to encompass a device capable of performing computing processing. That is, in the present embodiment, the computer may mean an apparatus capable of performing the artificial data generating method according to the embodiment of the present invention.

Referring to FIG. 5, in the method of generating artificial data according to an embodiment of the present invention, obtaining a background image and an object image (S100), and combining the background image and an object image to generate artificial data (S110). ) May be included. Hereinafter, a detailed description of each step will be described.

The computer may acquire a background image and an object image (S100).

The background image and the object image may be part or the entire image area included in the actual surgery image. The actual surgery image may be data obtained by the actual medical staff performing the surgery. For example, medical staff may perform surgery directly on a patient, or perform minimally invasive surgery using a surgical robot, a laparoscope, an endoscope, or the like. At this time, it is possible to obtain a variety of information about the surgical operation performed in such a surgical process or surgical instruments, surgical sites related to the surgical operation. For example, an actual surgical image obtained by photographing organs, surgical instruments, and the like in the body may be obtained in a surgical procedure, or data recorded on a surgical operation performed in the surgical procedure may be acquired.

In one embodiment, the background image may include a specific area inside the body taken by the endoscope camera. For example, the image data may be image data obtained by photographing a specific region including organs (eg, liver, heart, uterus, brain, breast, abdomen, etc.), blood vessels, and tissues in the body.

In one embodiment, the object image may be image data including surgical instruments or blood (including bleeding out of the blood vessels due to blood vessel damage).

According to an embodiment, the computer may acquire an object image or a background image from an actual surgical image including an object and a background (ie, a background including a specific area inside the body). Alternatively, the computer may acquire the object image from the actual surgical image including only the object or the background image from the actual surgical image including only the background. Thereafter, the computer may generate various artificial data using the object image or the background image obtained from the actual surgical image. Specific embodiments thereof will be described with reference to FIGS. 7 to 10.

In addition, according to an embodiment, the computer may acquire the actual surgical image in units of frames or in units of sequences, which are sets of consecutive frames. For example, when generating artificial data about an object including a surgical tool, the computer may generate an artificial data by acquiring an actual surgical image by one frame and extracting an object image including the surgical tool therefrom. have. Alternatively, when generating artificial data about an object including blood or bleeding, the computer acquires an actual surgical image in a sequence unit consisting of a plurality of frames and extracts an object image including blood or bleeding from the artificial data. Can be generated. In this case, the computer inputs the actual surgical image including the object image in sequence units (ie, continuous frames) to the generator to be described later, and accordingly, the generator inputs artificial data in sequence units (ie, continuous frames). Can be generated and printed. In the case of an object such as blood or bleeding, it is difficult to see the meaning of the object by looking at only one frame image, and thus it may be meaningful to be generated as a set of consecutive frames instead of one frame. Accordingly, in the present invention, artificial data may be generated in units of frames or sequences according to the type of object. In addition, according to the present invention, even when generating artificial data corresponding to a specific surgical operation (for example, cutting, grabbing, etc.), object images are acquired in sequence units as in the case of generating artificial data including blood or bleeding. Artificial data can be generated.

In addition, according to the present invention, artificial data may be generated by acquiring an actual surgical image in a frame unit or a sequence unit as a change of an object occurs. In one embodiment, when the computer acquires a real surgery image that changes in a specific object, for example, to acquire a real surgery image performing a specific surgery operation according to the movement of the surgical tool or the surgical tool in a continuous frame In this case, the computer may continuously extract an object image including a specific object (eg, a surgical tool) from consecutive frames to generate artificial data reflecting a change in movement of the specific object (eg, a surgical tool). In this case, the artificial data may be generated in a continuous frame according to a change of a specific object. In addition, the computer may extract a background image from the actual surgical image in response to the movement change of a specific object (eg, a surgical tool). For example, the computer continuously extracts the object image and the background image according to the change of the movement of a specific object (for example, a surgical tool), and generates continuous artificial data by sequentially combining the continuous object image and the background image. You may. Therefore, by extracting and reflecting the background image corresponding to the change of the object, it is possible to generate more artificial data.

The computer may generate artificial data by combining the background image and the object image (S110).

In one embodiment, the computer is based on the placement relationship between objects in the object image (e.g. surgical instruments, blood, bleeding, etc.) and specific areas within the body (e.g. organs, blood vessels, tissues, etc.) within the background image. As such, artificial data may be generated by combining the object image and the background image to approximate the actual surgical image.

The arrangement relationship may be information derived based on information such as a shape, a location, a direction or a camera photographing direction, a type and a number of backgrounds and objects in each image.

In generating artificial data, the computer may generate artificial data through learning by using a generator and a discriminator. This will be described in detail with reference to FIG. 6.

In an embodiment, in generating artificial data by combining the object and the background based on the arrangement relationship between the object in the object image and the background in the background image, the computer may generate the artificial data closer to the actual surgical image. Learning can be done. In this case, the computer may perform the learning by using the generator 200 and the discriminator 210. For example, the generator 200 and the discriminator 210 may operate by competing with each other using a generative adversarial network (GAN) based learning scheme.

Referring to FIG. 6, the generator 200 trains a generation model using a real surgery image, and based on the trained generation model, artificial data close to the actual input data (actual surgery image) 220. And 230.

In an embodiment, the generator 200 may generate artificial data by combining a background image and an object image obtained from each actual surgical image using a generation model.

The discriminator 210 trains a discrimination model to discriminate whether the artificial data 230 generated by the generator 200 is real data or artificial data. At this time, the discriminator 210 may train the discrimination model to discriminate the authenticity of the artificial data 230 based on the actual surgical image 240 corresponding to the artificial data. The discriminator 210 may retrain the generated model based on the learned discrimination model.

In one embodiment, the discriminator 210 may determine whether the artificial data 230 generated by the generator 200 is real data or artificial data using the discrimination model. If the discriminator 210 determines that the artificial data is not real, that is, the generator 200 does not deceive the discriminator 210, the generator 200 may not deceive the discriminator 210. You can retrain the generation model in a way that reduces. The generator 200 may generate improved artificial data through relearning the generation model. On the contrary, in the case where it is determined that the discriminator 210 is the actual data, that is, when the discriminator 210 is deceived by the generator 200, the discriminator 210 reduces the discrimination model in the direction of reducing the error rate (error). You can relearn. This process is repeated in the generator 200 and the discriminator 210, thereby making artificial data close to reality.

Therefore, when the artificial data approximating the actual surgical image is generated through the mutual learning between the generator 200 and the discriminator 210, the computer may construct a training data set for learning a learning model for recognizing a surgical image. .

Surgical images can only be obtained through actual surgical procedures, and often contain only limited scenes. In actual operation, surgery is performed on a specific surgical site, and since surgery is performed using a surgical tool determined according to the surgical site or the surgical purpose, it is difficult to secure a surgical image including various surgical tools or surgical sites. . Therefore, when learning is performed using such a surgical image, it is difficult to construct sufficient learning data using only the actual surgical image obtained during the actual surgery. Thus, in the present invention, by generating artificial data almost no difference from the actual surgical image, it can be provided as the learning data of the learning model to perform the learning using the surgical image.

7 to 10, various embodiments in which a computer acquires a background image and an object image to generate artificial data will be described.

FIG. 7 illustrates a first embodiment of generating artificial data using first and second actual surgical images including an object and a background.

Referring to FIG. 7, the computer may acquire a first real surgery image 300 including an object and a background, and a second real surgery image 310 including an object and a background. In this case, the computer may extract the object image or the background image included in each of the first real surgery image 300 and the second real surgery image 310, and combine them to generate artificial data.

In an embodiment, the computer may extract the background image in the first real surgery image 300 and the object image in the second real surgery image 310. The computer may generate artificial data 330 close to the actual surgical image by using the generated model (ie, the generator) 320 to extract the extracted background image and the object image. For example, based on the learning in the generation model 320, the computer may use the background in the background image (ie, organs, blood vessels, tissues, etc., present in a specific area of the body) and the objects in the object image (ie, surgical instruments). Background image and object image can be combined based on the arrangement relationship between the blood, blood, and bleeding. In this case, the computer may provide an object segmentation 311 in the second real surgery image 310 to the generation model 320 together with the second real surgery image 310 as a correct answer. Since the generation model 320 is trained through the object segmentation 311 which is the correct answer, it is possible to generate artificial data closer to the actual surgical image.

The computer may provide the artificial data 330 generated by the generation model 320 to the discrimination model (ie, the discriminator) 350. The differential model 350 may determine the authenticity of the artificial data 330 based on the actual surgical image 340 corresponding to the artificial data 330. In this case, the computer may additionally provide the object segmentation 331 in the artificial data 330 and the object segmentation 341 in the actual surgical image 340 to learn the differential model 350.

The computer may relearn the generation model 320 or the differential model 350 according to the determination result of the differential model 350. For example, when it is determined that the artificial data 330 is artificial data rather than actual data, the computer may relearn the generated model 320. On the contrary, when it is determined that the artificial data 330 is real data, the computer may relearn the differential model 350.

In another embodiment, the computer may extract the object image in the first real surgery image 300, extract the background image in the second real surgery image 310, and generate artificial data. Since a detailed process thereof is similar to that described with reference to FIG. 7, the description thereof will be omitted.

According to the above-described embodiment, artificial data is generated by using an actual surgical image including different objects, which is performed on rare objects (eg, surgical instruments, blood or bleeding) that do not appear well at the time of surgery. You can accumulate images. In addition, manual labeling can replace expensive segmentation data with artificial data, thereby saving costs.

FIG. 8 is a second embodiment of generating artificial data using a first real surgery image including only an object and a second real surgery image including both an object and a background.

Referring to FIG. 8, the computer may acquire a first real surgery image 400 including only an object and a second real surgery image 410 including both an object and a background. In this case, the computer may extract the object image from the first real surgery image 400 and the background image from the second real surgery image 410. The computer may generate artificial data 430 close to the actual surgical image by using the generation model (ie, the generator) 420 of the extracted background image and the object image.

For example, based on the training in the generation model 420, the computer may use the background in the background image (ie, organs, blood vessels, tissues, etc., present in a specific area of the body) and the objects in the object image (ie, surgical instruments). Background image and object image can be combined based on the arrangement relationship between the blood, blood, and bleeding.

The computer may provide the artificial data 430 generated by the generation model 420 to the discrimination model (ie, the discriminator) 450. The differential model 450 may determine the authenticity of the artificial data 430 based on the actual surgical image 440 corresponding to the artificial data 430. In this case, the computer may additionally provide the object segmentation 431 in the artificial data 430 and the object segmentation 441 in the actual surgical image 440 to learn the differential model 450.

The computer may relearn the generation model 420 or the differential model 450 according to the determination result of the differential model 450. For example, when it is determined that the artificial data 430 is artificial data rather than real data, the computer may relearn the generated model 320. On the contrary, when it is determined that the artificial data 430 is real data, the computer may relearn the differential model 450.

According to the embodiment described above, flexibility for generating artificial data can be increased, and surgical images for various objects can be accumulated. For example, various types of surgical tool images, surgical images with different positions, directions, and arrangements of surgical tools can be obtained.

9 is a third embodiment of generating artificial data using a first real surgery image including only a background and a second real surgery image including both an object and a background.

Referring to FIG. 9, the computer may acquire a first real surgery image 500 including only a background and a second real surgery image 510 including both an object and a background. In this case, the computer may extract the background image from the first real surgery image 500 and the object image from the second real surgery image 510. The computer may generate artificial data 530 close to the actual surgical image by using the generated model (ie, generator) 520 of the extracted background image and the object image.

For example, based on the learning in the generation model 520, the computer may use the background in the background image (ie, organs, blood vessels, tissues, etc., present in a specific area of the body) and the objects in the object image (ie, surgical instruments). Background image and object image can be combined based on the arrangement relationship between the blood, blood, and bleeding.

In this case, the computer may provide the object segmentation 511 in the second actual surgery image 510 with the second actual surgery image 510 as a correct answer to the generation model 520. Since the generation model 520 is trained through the object segmentation 511 which is the correct answer, the generation model 520 may generate artificial data closer to the actual surgical image.

The computer may provide the artificial data 530 generated by the generation model 520 to the discrimination model (ie, discriminator) 550. The differential model 550 may determine the authenticity of the artificial data 530 based on the actual surgical image 540 corresponding to the artificial data 530. In this case, the computer may additionally provide an object segmentation 531 in the artificial data 530 and an object segmentation 541 in the actual surgical image 540 to learn the differential model 550.

The computer may relearn the generation model 520 or the differential model 550 according to the determination result of the differential model 550. For example, when it is determined that the artificial data 530 is artificial data rather than actual data, the computer may relearn the generated model 520. On the contrary, if it is determined that the artificial data 530 is real data, the computer may relearn the differential model 550.

According to the above-described embodiments, it is possible to increase flexibility for generating artificial data and to obtain an image of operating various types of internal organs, blood vessels, tissues, etc. on a specific object (eg, a surgical tool). For example, it is possible to obtain a surgical image for operating different body parts for one surgical tool.

FIG. 10 is a fourth embodiment of generating artificial data using a first real surgery image including only a background and a second real surgery image including only an object.

Referring to FIG. 10, the computer may acquire a first real surgery image 600 including only a background and a second real surgery image 610 including only an object. In this case, the computer may extract the background image from the first real surgery image 600 and the object image from the second real surgery image 610. The computer may generate artificial data 630 close to the actual surgical image by using the generated model (ie, generator) 620 of the extracted background image and the object image.

For example, based on the learning in the generation model 620, the computer may use the background in the background image (ie, organs, blood vessels, tissues, etc., present in a specific area of the body) and the objects in the object image (ie, surgical instruments). Background image and object image can be combined based on the arrangement relationship between the blood, blood, and bleeding.

The computer may provide the artificial data 630 generated by the generation model 620 to the discrimination model (ie, discriminator) 650. The differential model 650 may determine the authenticity of the artificial data 630 based on the actual surgical image 640 corresponding to the artificial data 630. In this case, the computer may additionally provide the object segmentation 631 in the artificial data 630 and the object segmentation 641 in the actual surgical image 640 to learn the differential model 650.

The computer may relearn the generation model 620 or the differential model 650 according to the determination result of the differential model 650. For example, when it is determined that the artificial data 630 is artificial data rather than real data, the computer may relearn the generated model 620. On the contrary, when it is determined that the artificial data 630 is real data, the computer may relearn the differential model 650.

According to the above-described embodiment, a surgical image including various objects and various internal body spaces may be acquired. For example, various types of surgical images may be obtained by freely arranging information on necessary surgical tools or organs, blood vessels, tissues, and the like.

9 and 10, the artificial data is generated by acquiring an actual surgical image (background image) including only a background. In this case, the actual surgical image including only the background does not include an object such as a surgical tool and includes only a background image. In this case, the computer may acquire image frames photographed at a specific position inside the body by the endoscope camera, and remove the object image such as a surgical tool from the image frames to generate a background image including only the background area. Alternatively, the computer may use an image frame obtained by the endoscope entering the body during the operation using the endoscope. In this case, an image frame having only a background image not including an object such as a surgical tool may be obtained. Alternatively, the computer may extract an image frame including only a background from an image frame obtained by a camera entering the body during minimally invasive surgery such as laparoscopic or robotic surgery. In the case of minimally invasive surgery, the operation of the first area is completed and the camera and the surgical tool are also moved from the first area to the second area when the operation is performed by moving to the second area. At this time, there is a constraint that the surgical tool does not move when the camera moves due to the nature of the surgery. Accordingly, when moving from the first area to the second area, the camera moves first and then the surgical tool moves, so that the camera moved first to the second area may acquire an image frame without the surgical tool.

The computer may construct the training data using artificial data generated from the above embodiments. For example, artificial data may be used as training data for training a training model for image recognition. Therefore, according to an embodiment of the present invention, by providing artificial surgical images generated with artificial surgery, deep learning, machine learning, etc. which requires a large amount of learning data together with the actual surgical images, sufficient learning effects can be obtained.

According to FIG. 11, the robotic surgical system includes a medical imaging apparatus 10, a server 100, a control unit 30 provided in an operating room, a display 32, and a surgical robot 34. According to an embodiment, the medical imaging apparatus 10 may be omitted in the robot surgery system according to the disclosed embodiment.

As described above, when performing a robotic operation, data including various surgical information may be acquired in a surgical image photographed in a surgical process or a control process of a surgical robot. In one embodiment, in the present invention, artificial data may be generated by acquiring the object image or the background image as described above based on the surgical information (ie, the surgical image) obtained in the robot surgery. Of course, in generating artificial data according to an embodiment of the present invention, utilizing the surgical image obtained in the robot surgery process is just one example, the present invention is not limited thereto. In another embodiment, minimally invasive surgery using a laparoscope, endoscope, or the like, and a surgical image obtained when a medical staff directly operates on a patient may also be used to generate artificial data.

Referring to FIG. 12, the processor 710 may include a connection passage (eg, a bus or the like) that transmits and receives a signal with one or more cores (not shown) and a graphic processor (not shown) and / or other components. ) May be included.

The processor 710 according to an embodiment executes one or more instructions stored in the memory 720 to perform the artificial data generation method described with reference to FIGS. 5 to 10.

For example, the processor 710 may acquire a background image and an object image by executing one or more instructions stored in the memory 720, and generate artificial data by combining the background image and the object image.

Meanwhile, the processor 710 may include random access memory (RAM) and read-only memory (ROM) for temporarily and / or permanently storing a signal (or data) processed in the processor 710. , Not shown) may be further included. In addition, the processor 710 may be implemented in the form of a system on chip (SoC) including at least one of a graphic processor, a RAM, and a ROM.

The memory 720 may store programs (one or more instructions) for processing and controlling the processor 710. Programs stored in the memory 720 may be divided into a plurality of modules according to their functions.

The artificial data generation method according to an embodiment of the present invention described above may be implemented as a program (or an application) and stored in a medium to be executed in combination with a computer which is hardware.

Hereinafter, a method and apparatus for generating learning data based on a surgical image according to an embodiment of the present invention will be described in detail with reference to FIGS. 13 to 18.

According to FIG. 13, the robotic surgical system includes a medical imaging apparatus 10, a server 100, a control unit 30 provided in an operating room, a display 32, and a surgical robot 34. According to an embodiment, the medical imaging apparatus 10 may be omitted in the robot surgery system according to the disclosed embodiment.

As described above, when performing a robot operation, it is possible to obtain data including various surgical information in the surgical image taken during the surgical procedure or the control of the surgical robot. Thus, in the present invention, it is possible to use a surgical image or surgical data that can be obtained in the surgical process as a learning material, and by learning based on the learning data, a learning model that can infer a variety of information related to surgery. We want to provide a way to build it.

Hereinafter, for convenience of description, it will be described that a "computer" performs a method of generating learning data based on a surgical image according to an embodiment disclosed herein. "Computer" may mean the server 100 or the controller 30 of FIG. 13, but is not limited thereto. The term "computer" may be used to mean a device capable of performing computing processing. For example, the computer may be a computing device provided separately from the device shown in FIG. 13.

In addition, the embodiments disclosed below may not be applicable only in connection with the robotic surgery system illustrated in FIG. 13, but may be applied to all kinds of embodiments that may acquire and utilize a surgical image in a surgical procedure. For example, in addition to robotic surgery, it can be applied in connection with minimally invasive surgery such as laparoscopic surgery or endoscopic surgery.

Referring to FIG. 14, in the method of generating training data based on a surgical image according to an exemplary embodiment of the present disclosure, acquiring a surgical image including a plurality of image frames (S100), each of the plurality of image frames Recognizing surgical recognition information from the step (S200), and for each of the plurality of image frames, a relational expression representing a relationship between the surgical element (surgical element) included in the surgical recognition information based on the surgical recognition information (Relational) Representation) may include the step of generating (S300). Hereinafter, a detailed description of each step will be described.

The computer may acquire a surgical image including a plurality of image frames (S100).

When performing certain operations (eg, gastric cancer surgery, colorectal cancer surgery, etc.) on the patient, the medical staff may perform the actual surgery on the patient directly, as well as laparoscopes or endoscopes, including the surgical robot as described in FIG. Minimally invasive surgery may be performed.

In this case, the computer may acquire a surgical image photographing a scene including a surgical operation performed in the surgical procedure, a surgical tool related thereto, a surgical site, and the like. In an embodiment, the computer may acquire a surgical image photographing a scene including a surgical site and a surgical tool that is currently undergoing surgery from a camera entering the patient's body.

The surgical image may include one or more image frames. Each image frame may represent a scene in which a surgical operation is performed, including a surgical site, a surgical tool, and the like of a surgical subject. For example, the surgical image may be composed of image frames in which a surgical operation is recorded for each scene (scene) according to time during the surgical procedure. Alternatively, the surgical image may be composed of image frames that record each surgical scene according to the spatial movement such as the surgical site or the position of the camera during the surgery.

In addition, the surgical image may be composed of all image frames including the entire surgical procedure, or may be in the form of at least one video clip divided according to a specific classification criterion.

According to an embodiment, the computer may acquire a surgical image in the form of a video clip, or may obtain a surgical image including all or part of a surgical procedure and divide the surgical image into at least one video clip.

In an embodiment, the computer may acquire a surgical image including all or part of a surgical procedure and divide the surgical image into at least one video clip according to a specific classification criterion. For example, the computer may divide the surgical image according to the time course of the surgery, or may divide the surgical image according to the position or state change of the surgical site based on the surgical site during the surgery. Alternatively, the computer may segment the surgical image based on the position of the camera or the moving range of the camera during the operation, or divide the surgical image based on the change (eg replacement) of the surgical tool during the operation. You may. Alternatively, the computer may segment the surgical image based on image frames formed of a series of surgical operations in relation to the surgical operation. In addition, each operation may be predetermined surgery stages classified according to a specific classification criteria. In this case, the computer may segment the surgical image based on the surgical stage.

The computer may recognize surgical recognition information from each of the plurality of image frames in the acquired surgical image (S200).

As described above, each image frame in the surgical image includes surgery related information. For example, it includes various surgical-related information such as surgical instruments, surgical operations, surgical sites (ie, body parts such as organs, blood vessels, tissues), and bleeding. Therefore, in the present invention, each element that can be recognized as surgery related information from an image frame is defined as a surgical element. The surgical element may be a concept corresponding to the object, but in the present invention, a wide concept encompassing not only an object but also information such as an operation, a function, and a state related to the object.

In an embodiment, the computer may extract surgical recognition information determined as a surgical element from each of the plurality of image frames. Here, the surgical recognition information refers to the surgical element information recognized from the image frame, for example, surgical instruments, surgical operations, body parts, bleeding, surgical stages, surgery time (eg, remaining surgery time, surgery time, etc.), and camera It may include at least one surgical element of information (eg, camera position, angle, direction, movement, etc.).

In addition, the computer may derive the position information of the surgical element in the surgical recognition information extracted from each of the plurality of image frames. The position information of the surgical element may be region information where the surgical element is located on the image frame. For example, the position information of the surgical element may be calculated based on the coordinate information on the 2D space or the coordinate information on the 3D space in the image frame.

The computer may generate relational representation information indicating a relationship between surgical elements included in each surgical recognition information based on the surgical recognition information recognized for each of the plurality of image frames (S300).

In an embodiment, the computer may include the first image based on the plurality of surgical elements extracted from the first image frame among the plurality of image frames (eg, the first to nth image frames) and the position information on the plurality of surgical elements. It may be determined whether a correlation exists between a plurality of surgical elements in the frame. Next, the computer may generate relationship expression information for the first image frame based on the correlation between the plurality of surgical elements. The computer generates relationship expression information for not only the first image frame but also the second to nth image frames in the surgical image in the same manner.

In determining the correlation between the plurality of surgical elements, the computer may determine whether or not there is a correlation between the plurality of surgical elements based on the relationship information on the predefined surgical elements. Here, the relationship information on the predefined surgical element may be information set based on at least one of the type of surgical element, location information of the surgical element, state information of the surgical element, and operation information of the surgical element.

As described above, surgical tools, surgical operations, body parts, bleeding status, surgical steps, surgery time, camera information, etc. can be defined as a surgical element. Each defined surgical element may further include additional information such as location information, state information, and operation information according to the type of surgical element. For example, the type of surgical tool can be determined according to the surgical site, and the operation can be determined according to each surgical tool operating state information such as open (close), energy (close), the presence of energy, contact, etc. have. Whether or not bleeding may have color information based on the bleeding point. In other words, the computer may preset, as relational information, each surgical element and additional information determined according to the type of each surgical element. Accordingly, the computer may recognize each surgical element from the first image frame and grasp relationship information about each recognized surgical element.

For example, when the computer extracts the surgical tool and the surgical site from the first image frame, the computer identifies correlation information between the specific surgical tool and the specific surgical site based on the predefined relationship information, and the correlation between the two surgical elements. Can be determined to exist.

Or, when the computer extracts the surgical tool from the first image frame, the correlation between the surgical tool and the surgical operation exists by grasping the relationship information on the specific surgical tool and its operation based on the predefined relationship information. You can judge.

Also, when the plurality of surgical elements having the same or similar location information are extracted from the first image frame, the computer may determine that there is a correlation between the plurality of surgical elements. For example, if the computer extracts the body organs and surgical instruments from the first image frame, and recognizes that the extracted body organs and surgical instruments are present at the same location, a positional correlation exists between the two surgical elements. You can judge. The computer may determine that the positional correlation exists, for example, when the surgical tool is exerting a specific motion on the corresponding organ or when the surgical tool is in contact with the organ.

The computer may generate relationship expression information for the first image frame based on whether there is a correlation between a plurality of surgical elements recognized from the first image frame.

In an embodiment, the computer may generate relational expression information by mapping each surgical element and information about each surgical element. For example, the relational expression information may be generated in the form of a matrix. Each surgical element may be arranged corresponding to a row and a column, and information derived based on a correlation between each surgical element may be represented as an element of a matrix.

The computer may generate relational expression information for the second to nth image frames in the surgical image in the same manner as the first image frame as described above.

15 is a diagram illustrating a process of generating relationship expression information by recognizing surgical recognition information from an image frame according to an embodiment of the present invention. For example, FIG. 15 illustrates a process of recognizing surgical recognition information from the first image frame 200 in the surgical image and generating it as the relational expression information 300.

In an embodiment, the computer may recognize each surgical element by applying an image recognition technology using deep learning from the first image frame 200. In recognizing each surgical element, the computer may apply various individual recognition techniques according to the characteristics of each surgical element. For example, the computer extracts feature information (eg, a feature map) from the first image frame 200, and uses a body feature (eg, texture feature information that expresses color or texture). Body organs such as liver, stomach, and blood vessels) and bleeding. Alternatively, the computer may recognize a surgical instrument, a surgical operation, or the like using the feature information about the shape. Alternatively, the computer may recognize the operation by using the location feature information between the surgical elements. Alternatively, the computer may recognize camera information (eg, motion information such as camera angle and movement), a surgery step, a surgery operation, a surgery time, and the like by using feature information between the plurality of image frames.

As described above, as a result of the computer recognizing the surgical recognition information for the first image frame 200 based on the characteristic information of each surgical element, as shown in FIG. Operation status, gauze, surgical site, etc. were recognized as surgical elements.

In addition, the computer may derive position information on each surgical element recognized from the first image frame 200. In an embodiment, the computer may divide the first image frame 200 into at least one region, and calculate position information of each surgical element based on a region where each surgical element exists among the divided regions. In this case, the computer may divide the space in the first image frame 200 into at least one region based on two-dimensional or three-dimensional coordinate information, and for each divided region, a specific coordinate value (for example, a center point of each region) ) Or index value to use as location information of each area.

For example, as illustrated in FIG. 15, the computer may divide the first image frame 200 into nine regions and assign index values of 0 to 8 to each divided region to use the position information. . The computer determines that the surgical element 'gauze' recognized from the first image frame 200 is located in the

index

0, 1 region among the divided regions, and then the position information of the surgical element 'gauze' based on the

index

0, 1 Can be calculated. Position information of each surgical element may be calculated in the same manner with respect to other surgical elements (for example, two surgical instruments and surgical regions) recognized from the first image frame 200.

The computer may generate the relationship expression information 300 based on each surgical element recognized from the first image frame 200 and position information of each surgical element. In an embodiment, the computer may configure the relational expression information 300 in a matrix form. Each surgical element may be arranged in each row and each column, and relationship information between corresponding surgical elements may be represented in each component value of the matrix.

For example, as shown in FIG. 15, the relationship expression information 300 of the first image frame 200 may be represented by a matrix. In one example, each row and each column may be arranged from surgical element a to surgical element s. In this case, the operation element a is the operation stage, the surgical elements b to h surgical instruments, the surgical elements i to l body parts, the surgical elements m to q surgical operation, the surgical element r is bleeding, the surgical element s is camera information, etc. It can be defined as. In addition, the i th column of the i th row of the matrix may represent position information of a corresponding surgical element (eg, a surgical element, b surgical element, etc.). The j th column of the i th row of the matrix may represent relationship information between the surgical elements arranged in the i th row and the surgical elements arranged in the j th column. For example, in the row corresponding to the surgical element b of the matrix and the column corresponding to the surgical element m, the specific surgical tool (the surgical tool defined as the surgical element b, for example, 'Harmonics') and the specific surgical operation (the surgical element m are defined). Information about a surgical operation, for example, open / grap). Depending on the surgical element, other specific values may be assigned to the components of the matrix to represent the relationship information. For example, when an event related to a surgical element occurs or implies such an event, such as camera information or an operation step, the matrix component for the corresponding surgical element is used instead of the position information or relationship information of the corresponding surgical element. Information related to the event may be given.

The computer may repeatedly perform the process as described with reference to FIG. 15 for each image frame in the surgical image. That is, the computer may recognize a surgical element for each image frame in the surgical image and generate relational expression information based on the recognized surgical element.

Referring to FIG. 16, the computer may generate relationship expression information for each of a plurality of image frames in a surgical image. In this case, since each relationship expression information expresses the relationship information between a plurality of surgical elements at the same time, it may have tensor information. In an embodiment, the computer may generate the relation expression information generated from the plurality of image frames in the form of a tensor. In addition, the computer may express a relationship between the plurality of image frames by grouping the plurality of image frames based on a temporal size of the tensor. For example, when it is necessary to generate relational expression information based on a plurality of image frames, such as camera information or a surgical step, the computer generates a group of a specific number of image frames and generates each relational expression information in the form of a tensor, and the camera information. Alternatively, information related to the surgical stage may be related to image frames and may be represented by a tensor.

According to an embodiment of the present invention, as described above, the relation expression information may be generated for each image frame in the surgical image, and the learning data may be constructed based on the generated relation expression information. In addition, learning may be performed to derive a specific learning result based on the learning data constructed as described above.

Referring to FIG. 17, the computer may generate relational expression information for each image frame in the surgical image, and construct learning data based on the generated relational expression information (S400).

In one embodiment, in constructing the learning data by generating the relationship expression information for each image frame in the surgical image, the labeling learning may be performed by a medical staff, or the labeling learning may be automatically performed by a computer. In addition, when learning is automatically performed by a computer, semi-supervised learning may be performed. For example, the computer may label each image frame by applying various individual recognition techniques (eg, body organ recognition, surgical tool recognition, bleeding recognition, surgical motion recognition, camera movement recognition, etc.) according to the characteristics of each surgical element as described above. Extracted information and relationship expression information can be generated based on the extracted information.

The computer may derive a specific learning result by performing learning based on the learning data (S500).

Here, the specific learning result is a specific learning model by constructing a specific learning model by performing learning using the learning data generated according to an embodiment of the present invention as an input value, and obtaining a result of the learning that can be obtained as an output value of the specific learning model. Say.

For example, the computer may determine that the learning result is to derive the relationship expression information according to the embodiment of the present invention. That is, the computer may learn about the newly input image frame based on the previously constructed learning data (relational expression information), and acquire the relationship expression information on the newly input image frame as a result of the learning.

Alternatively, the computer may perform learning by using the learning data generated according to an embodiment of the present invention as an input value to acquire a variety of learning results such as recognizing a surgical procedure through a surgical image or recognizing the meaning of a surgical operation. Can be.

According to an embodiment, the computer may use image frames as an input value for deriving a specific learning result together with the learning data generated according to the embodiment of the present invention. In an embodiment, the computer may learn a plurality of image frames in the surgical image by using a convolutional neural network (CNN), and may acquire feature information of each image frame (S410). In addition, the computer may generate the relationship expression information for the plurality of image frames in the surgical image to obtain the training data (S400). The computer may finally perform the learning based on the feature information and the relationship expression information of each image frame, and may obtain a specific output value as a result of the learning (S500).

In addition, according to the present invention, rather than generating learning data for building a learning model, it provides the basis data for building a variety of learning models. In addition, by providing such basic learning data, the learning data may be increased to obtain more improved learning results.

In addition, according to the present invention, it is possible to flexibly define a variety of information necessary for the operation through the relationship expression information according to the learning purpose, thereby performing effective learning.

Referring to FIG. 18, the processor 410 may include a connection passage (for example, a bus or the like) that transmits and receives signals with one or more cores (not shown) and a graphics processor (not shown) and / or other components. ) May be included.

The processor 410 according to an exemplary embodiment executes one or more instructions stored in the memory 420 to perform a method of generating learning data based on the surgical image described with reference to FIGS. 14 to 17.

For example, the processor 410 may acquire a surgical image including a plurality of image frames by executing one or more instructions stored in the memory 420, recognizing surgical recognition information from each of the plurality of image frames, and For each of the plurality of image frames, generating relational representation information representing a relationship between surgical elements included in the surgical recognition information based on the surgical recognition information.

Meanwhile, the processor 410 may include random access memory (RAM) and read-only memory (ROM) for temporarily and / or permanently storing a signal (or data) processed in the processor 410. , Not shown) may be further included. In addition, the processor 210 may be implemented in the form of a system on chip (SoC) including at least one of a graphic processor, a RAM, and a ROM.

The memory 420 may store programs (one or more instructions) for processing and controlling the processor 410. Programs stored in the memory 420 may be divided into a plurality of modules according to their functions.

The method for generating learning data based on the surgical image according to the exemplary embodiment of the present invention described above may be implemented as a program (or an application) and stored in a medium to be executed in combination with a computer which is hardware.

Hereinafter, a method and apparatus for providing blood vessel information using a blood vessel model according to an embodiment of the present invention will be described in detail with reference to FIGS. 19 to 26.

Although the method of FIG. 19 is described as being performed by a computer for the convenience of description, the subject of each step is not limited to a specific device but may be used to encompass a device capable of performing computing processing. That is, in the present embodiment, the computer may mean an apparatus capable of performing the method of providing blood vessel information using the blood vessel model according to the embodiment of the present invention.

Referring to FIG. 19, a method of providing blood vessel information using a blood vessel model according to an embodiment of the present disclosure may include obtaining a blood vessel polygon constituting a blood vessel in a blood vessel model (S100), the blood vessel polygon information, and the Comprising the step of configuring the training data based on the connection information of the vascular polygon (S200), and classifying the blood vessels based on the learning data (S300). Hereinafter, a detailed description of each step will be described.

The computer may acquire a blood vessel polygon constituting a blood vessel in the blood vessel model (S100).

In one embodiment, the computer may generate a 3D blood vessel model based on medical image data of the inside of the body of the object (eg, the patient). Here, the medical image data is a medical image photographed by a medical image photographing apparatus and includes all medical images that can be implemented as a three-dimensional model of the body of the object. For example, the medical image data may include a computed tomography (CT) image, a magnetic resonance imaging (MRI), a positron emission tomography (PET) image, and the like. The computer may extract the blood vessel of the patient from the medical image data and 3D model the extracted blood vessel. For example, the computer may sequentially extract arteries and veins from the medical image data, and 3D model the vessel models including the arteries and the vessel models including the veins, respectively, and then match them. In this case, the blood vessel model may be a polygon model of 3D modeling by constructing at least one polygon of blood vessels extracted from the medical image data. Thus, the computer can obtain at least one polygon constituting the vessel from the 3D vessel model. Polygon refers to a polygon, which is the smallest unit used to express a three-dimensional shape of an object in 3D computer graphics, and polygons may be gathered to represent a 3D object (ie, a blood vessel).

In another embodiment, the computer may obtain a 3D blood vessel model pre-built for the subject and obtain at least one polygon constituting the blood vessel therefrom. In this case, the computer may construct and store the 3D blood vessel model in advance, or may acquire and use a 3D blood vessel model generated by another device.

That is, the computer may obtain at least one vascular polygon constituting the blood vessel from the 3D modeled vascular model using the polygon.

The computer may configure learning data for each of the acquired at least one vascular polygon based on the vascular polygon information and the connection information of the vascular polygon (S200). In other words, the computer may construct training data based on its own information about each vascular polygon in the 3D vascular model and connection information with adjacent polygons connected thereto.

Here, the vascular polygon information may include a position vector of the vascular polygon and blood vessel information of the vascular polygon. The connection information of the vascular polygon may include adjacent polygons adjacent to the vascular polygon, and position vector difference information between the adjacent polygon and the vascular polygon.

First, the computer can obtain the vascular polygon itself information from the vascular polygon. In one embodiment, the computer may derive the position vector and vessel information of the vessel polygon.

The computer may calculate a position vector of the vascular polygon based on a predetermined reference vessel point in the 3D vessel model. For example, the computer may set a representative branching point or an easily distinguishable vessel of a vessel, such as a celiac trunk or a celiac axis, in the 3D vessel model as a reference vessel point. The computer may set the set reference vessel point as a reference point (eg, an origin point) in the coordinate space, and calculate a position vector of the vessel polygon relative to the reference vessel point. At this time, the computer may calculate a position vector with respect to the midpoint in the vascular polygon. The midpoint of the vascular polygon can be calculated using the inner core, outer core, center of gravity, and the like.

In addition, the computer may derive the vessel information of the vessel polygon based on the predetermined artery and vein classification information. For example, the computer may use the vessel information classified in detail for the predefined arteries and veins as shown in Table 1 below to determine what the vessel types of the vessel polygon are. For example, the computer can obtain vascular information such as artery 'Aorta' or vein 'PV' for vascular polygons.

TABLE 1

As another example, the computer may determine what the types of blood vessels of the blood vessel polygon are by using the color code information of the blood vessels to which the color information is assigned to the predefined arteries and veins as shown in Table 2 below. In this case, the blood vessel classification result derived as a learning result about the blood vessel polygon may be confirmed by color information.

TABLE 2

Next, the computer can obtain the connection information from the vascular polygon to the adjacent polygon connected thereto. In an embodiment, the computer may acquire adjacent polygons adjacent to the vascular polygon and derive the position vector difference information with the adjacent polygons as connection information.

When the computer acquires at least one vascular polygon as shown in FIG. 20, the computer may derive connection information between each vascular polygon and an adjacent polygon adjacent thereto as shown in Table 3 below. For example, each vascular polygon may be a triangular polygon, and each triangular polygon may be considered a node.

TABLE 3

According to Table 3, since node 1 is adjacent to node 2, in the case of the vascular polygon of node 1, the vascular polygon of node 2 may be acquired as an adjacent polygon. In the case of the vascular polygon of the second node, the vascular polygons of the first and third nodes may be acquired as adjacent polygons.

In addition, the computer may derive the position vector difference from the vascular polygon as connection information by using the acquired position vector of the adjacent polygon. In this case, the position vector of the adjacent polygon may be calculated with respect to the midpoint of the adjacent polygon based on the reference blood vessel point as described above.

Referring to FIG. 21 as an example, when deriving a position vector difference for a polygon centered on Y, the computer acquires adjacent polygons adjacent thereto and calculates a position vector (X, Z, Q position vector) of each adjacent polygon. can do. The computer may then calculate a difference vector between the position vectors (X, Z, Q position vectors) of the respective polygons and the position vectors of the polygons centered on Y. This may be calculated as in Equation 1 below.

[Equation 1]

According to an embodiment, the computer may derive connection information with major organs and use it as learning data. In one embodiment, the computer may acquire at least one reference organ, and derive the distance information between the obtained reference organ and the vascular polygon to construct the learning data.

The computer can acquire fixed-position organs, such as liver, heart, spleen, lung, kidney, etc., as reference organs. In this case, the reference organ may be composed of 3D modeled polygons. The computer may then place reference organs based on a predetermined reference vessel point in the 3D vessel model as a reference (eg, origin). Thus, the computer may calculate a position vector for the reference organ based on the reference vessel point. The computer may derive the distance information between the reference organ and the vascular polygon based on the calculated position vector of the reference organ and the position vector of the vascular polygon. For example, if the position vector of the reference organ is (x ₁ , y ₁ , z ₁ ) and the position vector of the vascular polygon is (x ₂ , y ₂ , z ₂ ), the computer generates distance information as shown in Equation 2 below. (d) can be calculated.

[Equation 2]

Here, the position vector (x ₂ , y ₂ , z ₂ ) of the vascular polygon may be the midpoint of the polygon. The position vector (x ₁ , y ₁ , z ₁ ) of a reference organ may be the center of the OBB when a cubic Oriented Bounding Box (OBB) is placed on the reference organ. 22 is a diagram illustrating a process of calculating the midpoint of the OBB. Referring to FIG. 22, a computer may cover a hexahedron OBB corresponding to the organ. In this case, since the midpoint of the cube OBB can be calculated using a diagonal line, the computer uses the two positions of the cube V1 (x ₁ , y ₁ , z ₁ ) and V2 (x ₂ , y ₂ , z ₂ ) As shown in Equation 3, the midpoint M (x ₃ , y ₃ , z ₃ ) of the OBB can be calculated.

[Equation 3]

As described above, the computer may include the position vector of the vascular polygon, the vessel information of the vascular polygon (ie, the vessel type), the position vector of the adjacent polygon, the difference vector information between the adjacent polygon and the vascular polygon, and the distance information between the reference organ and the vascular polygon. At least one may be derived, and learning data may be constructed based on the at least one.

The computer may classify blood vessels based on the learning data (S300).

In an embodiment, the computer may generate a vessel classification learning model for classifying vessels by performing learning based on the training data. For example, a computer can learn learning data through machine learning. For example, a map such as a decision tree, a K-nearest neighbor, a neural network, and a support vector machine (SVM) can be used. You can use learning. That is, the computer may generate a blood vessel classification learning model by performing supervised learning for classifying blood vessels based on the training data generated based on the blood vessel polygon.

For example, the computer may perform learning to classify blood vessel types based on the learning data as described above. The computer may derive the structure of the blood vessel as shown in FIG. 24 to be described later as a learning result, and may specify the type of blood vessel to which the blood vessel polygon belongs on the structure of the blood vessel.

In addition, the computer may perform learning to classify blood vessel types based on the learning data. In this case, the learning data may use blood vessel types as input information as blood vessel information. FIG. 23 is a diagram illustrating a learning process for classifying blood vessel types according to one embodiment of the present invention. For example, FIG. 23 is a diagram illustrating types of hepatic artery (type 10, for example) according to shape. The computer may use, as learning data, information classified by type of hepatic artery as shown in FIG. 23. In this case, the computer can perform the learning based on the shape of these vessels. The computer may derive a hierarchical structure of blood vessels as shown in FIG. 25 to be described later as a learning result, and specify a hierarchy of blood vessels to which a vascular polygon belongs on the blood vessel hierarchy.

In classifying blood vessels based on the training data, the computer may generate a blood vessel classification learning model by performing a learning process for classifying blood vessel types and a learning process for classifying blood vessel types, respectively. Alternatively, the computer may generate a blood vessel classification learning model by simultaneously performing a learning process for classifying blood vessel types and a learning process for classifying blood vessel types.

In addition, according to an embodiment, in order to improve the recognition rate of the blood vessel classification learning model, the computer may perform the learning using the additional learning data. In an embodiment, the computer may acquire a 2D projection image of projecting a blood vessel corresponding to a blood vessel polygon in the 3D blood vessel model in a 2D space, and use the additional learning data.

For example, the computer may extract 3D vessels corresponding to the vascular polygons in the 3D vessel model, and project the extracted 3D vessels in a two-dimensional space to a specific view to obtain a two-dimensional projection image. The computer may obtain at least one two-dimensional projection image by projecting the extracted 3D blood vessel onto a two-dimensional space for at least one viewpoint corresponding to a predetermined range based on a specific viewpoint. The computer may perform learning (eg, CNN) using the acquired two-dimensional projection image set as learning data. For example, the computer can learn what the vessel types are by using a two-dimensional projection image set.

In an embodiment of the present invention, the computer may obtain a new vascular polygon model and classify the blood vessels in the new vascular polygon model through the vascular classification learning model.

In one embodiment, the computer may extract at least one vascular polygon from the new vascular polygon model. The computer may obtain at least one of the extracted vector of the vascular polygon, the position vector of the vascular polygon, the adjacent polygon adjacent to the vascular polygon, the position vector of the adjacent polygon, and the distance information from the reference organ. The computer may input information about the vascular polygon obtained in the vessel classification learning model, and may derive at least one of blood vessel type and vascular hierarchy information corresponding to the vascular polygon as a result of the output.

Figure 24 (a) shows the structure of the main vein vessels, Figure 24 (b) shows the structure of the major arterial vessels. 25 shows the hierarchical structure of the aorta and its detailed arterial vessels.

As described above, when a new vascular polygon model is input, the artery or vein type to which the vascular polygon belongs may be specified as a learning result through the vascular classification learning model. For example, it is possible to grasp the type of blood vessel to which the corresponding blood vessel polygon belongs in the blood vessel structure as shown in FIG. 24 (a) or FIG. 24 (b). Furthermore, as a result of learning through the blood vessel classification learning model, the type of blood vessel (ie, the type of artery or vein) to which the corresponding blood vessel polygon belongs may be specified. For example, the hierarchical structure of blood vessels as shown in FIG. 25 may specify a hierarchy of blood vessels to which the corresponding blood vessel polygon belongs.

Referring to FIG. 26, the processor 210 may include a connection passage (eg, a bus or the like) that transmits and receives a signal with one or more cores (not shown) and a graphic processor (not shown) and / or other components. ) May be included.

The processor 210 according to an embodiment executes one or more instructions stored in the memory 220 to perform a method of providing blood vessel information using the blood vessel model described with reference to FIGS. 19 to 25.

In one example, the processor 210 acquires a blood vessel polygon constituting a blood vessel in a blood vessel model by executing one or more instructions stored in the memory 220, based on the blood vessel polygon information and connection information of the blood vessel polygon. And organizing the learning data, and classifying blood vessels by performing learning based on the learning data.

On the other hand, the processor 210 is a random access memory (RAM) and a ROM (Read-Only Memory) for temporarily and / or permanently storing signals (or data) processed in the processor 210. , Not shown) may be further included. In addition, the processor 210 may be implemented in the form of a system on chip (SoC) including at least one of a graphic processor, a RAM, and a ROM.

The memory 220 may store programs (one or more instructions) for processing and controlling the processor 210. Programs stored in the memory 220 may be divided into a plurality of modules according to their functions.

The blood vessel information providing method using the blood vessel model according to the above-described embodiment of the present invention may be implemented as a program (or an application) to be executed by being combined with a computer which is hardware and stored in a medium.

Meanwhile, in the embodiments of the present invention described above, the program may be read by the computer's processor (CPU) through the device interface of the computer in order for the computer to read the program and execute the methods implemented as the program. Code that is coded in a computer language such as C, C ++, JAVA, or machine language. Such code may include functional code associated with a function or the like that defines the necessary functions for executing the methods, and includes control procedures related to execution procedures necessary for the computer's processor to execute the functions according to a predetermined procedure. can do. In addition, the code may further include memory reference code for additional information or media required for the computer's processor to execute the functions at which location (address address) of the computer's internal or external memory should be referenced. have. Also, if the processor of the computer needs to communicate with any other computer or server remotely in order to execute the functions, the code may be used to communicate with any other computer or server remotely using the communication module of the computer. It may further include a communication related code for whether to communicate, what information or media should be transmitted and received during communication.

The stored medium is not a medium for storing data for a short time such as a register, a cache, a memory, but semi-permanently, and means a medium that can be read by the device. Specifically, examples of the storage medium include, but are not limited to, a ROM, a RAM, a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, and the like. That is, the program may be stored in various recording media on various servers to which the computer can access or various recording media on the computer of the user. The media may also be distributed over network coupled computer systems so that the computer readable code is stored in a distributed fashion.

The steps of a method or algorithm described in connection with an embodiment of the present invention may be implemented directly in hardware, in a software module executed by hardware, or by a combination thereof. Software modules may include random access memory (RAM), read only memory (ROM), erasable programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory, hard disk, removable disk, CD-ROM, or It may reside in any form of computer readable recording medium well known in the art.

In the above, embodiments of the present invention have been described with reference to the accompanying drawings, but those skilled in the art to which the present invention pertains may realize the present invention in other specific forms without changing the technical spirit or essential features thereof. I can understand that. Therefore, it should be understood that the embodiments described above are exemplary in all respects and not restrictive.

Claims

In the relay server for building the surgical information,

Relay communications between clients,

Receiving actual surgery information generated based on the actual surgery image from the first client, transmitting the received actual surgery information to the second client,

And receiving virtual surgery information generated by performing virtual surgery based on the actual surgery information from the second client, and transmitting the virtual surgery information to the first client.
The method of claim 1,

The relay server,

And storing the actual surgery information received from the first client and the virtual surgery information received from the second client.
The method of claim 1,

The relay server,

Relay server, characterized in that for generating a cue sheet data on the surgical procedure based on the actual surgery information and the virtual surgery information.
The method of claim 3,

The cue sheet data,

And a plurality of detailed surgery operations generated by dividing a surgery procedure corresponding to the actual surgery image.
The method of claim 3,

The relay server,

And performing the learning based on the cue sheet data to calculate optimized cue sheet data corresponding to the actual surgical image.
The method of claim 1,

The relay server,

Receiving surgery guide information from a third client, transmitting the received surgery guide information to the first client or the second client,

The third client,

And receiving the actual surgery information and the virtual surgery information from the relay server and generating surgery guide information corresponding to the actual surgery image.
The method of claim 1,

The actual surgery information,

At least one of the camera information, surgical site information, and surgical instrument information obtained from the actual surgical image,

The virtual surgery information,

Relay server, characterized in that it comprises a simulation data generated by performing the virtual surgery.
In the surgery information construction method performed by the relay server,

Receiving actual surgery information on the actual surgery image from the first client;

Transmitting the actual surgical information to a second client;

Receiving virtual surgery information generated based on the actual surgery information from the second client; And

Surgical information construction method comprising the step of storing the actual surgery information and the virtual surgery information.
A computer program, coupled to a computer, which is hardware, stored on a recording medium readable by a computer to perform the method of claim 8.
In the method for generating artificial data performed by a computer,

Obtaining a background image and an object image; And

Generating artificial data by combining the background image and the object image;

The background image includes a specific area inside the body taken by the endoscope camera,

The object image includes a surgical instrument or blood,

The artificial data is artificial data generation method, characterized in that generated on the basis of the relationship between the surgical tool or blood in the object image, and the specific region of the body in the background image approximating the actual surgical image.
The method of claim 10,

Generating the artificial data,

Artificial data generation method characterized in that to generate the artificial data using a generation model (Generative Model) learned based on the actual surgical image.
The method of claim 11,

Generating the artificial data,

Determining whether the artificial data is real data or artificial data by using a discrimination model trained to discriminate whether the actual data is artificial data or artificial data; And

And re-learning the generation model according to the determination result.
The method of claim 12,

The differentiation model,

The artificial data generation method characterized in that the learning using the artificial data and the object segmentation (segmentation) in the actual surgical image corresponding to the artificial data.
The method of claim 12,

And constructing a training data set for surgical image recognition based on the new artificial data generated from the retrained generation model.
The method of claim 10,

Acquiring the background image and the object image,

And obtaining the object image from the first real surgical image including an object and a background, and obtaining the background image from a second real surgical image including an object and a background.
The method of claim 10,

Acquiring the background image and the object image,

And obtaining the object image from the first real surgical image including an object, and obtaining the background image from a second real surgical image including an object and a background.
The method of claim 10,

Acquiring the background image and the object image,

And obtaining the object image from the first real surgery image including an object and a background, and obtaining the background image from a second real surgery image including a background.
The method of claim 10,

Acquiring the background image and the object image,

And obtaining the object image from the first real surgical image including an object, and obtaining the background image from the second real surgical image including a background.
Memory for storing one or more instructions; And

A processor for executing the one or more instructions stored in the memory,

The processor executes the one or more instructions,

Obtaining a background image and an object image; And

Performing artificial data by combining the background image and the object image;

The background image includes a specific area inside the body taken by the endoscope camera,

The object image includes a surgical instrument or blood,

The artificial data may be generated based on an arrangement relationship between a surgical tool or blood in the object image and a specific region within the body in the background image.
A computer program, coupled to a computer, which is hardware, stored on a recording medium readable by a computer so as to perform the method of claim 10.
In the method for generating training data based on a surgical image performed by a computer,

Acquiring a surgical image comprising a plurality of image frames;

Recognizing surgical recognition information from each of the plurality of image frames; And

For each of the plurality of image frames, generating relational representation information representing a relationship between surgical elements included in the surgical recognition information based on the surgical recognition information. How to generate learning data based on the surgical image.
The method of claim 21,

The surgery recognition information,

Method for generating learning data based on the surgical image, characterized in that it comprises at least one of the surgical elements, surgical operation, body parts, bleeding status, operation stage, operation time, and camera information.
The method of claim 21,

Recognizing the surgical recognition information,

Extracting the surgical recognition information determined as a surgical element from each of the plurality of image frames; And

And deriving position information of a surgical element in the surgical recognition information extracted from each of the plurality of image frames.
The method of claim 23, wherein

Deriving the position information of the surgical element,

Dividing each of the plurality of image frames into at least one area; And

And deriving position information of the surgical element based on an area in which the surgical element exists among the divided regions.
The method of claim 23, wherein

Generating the relationship expression information,

It is determined whether there is a correlation between the plurality of surgical elements in the first image frame based on the plurality of surgical elements extracted from the first image frame among the plurality of image frames and the position information of the plurality of surgical elements. Making; And

And generating relationship expression information for the first image frame based on the correlation between the plurality of surgical elements.
The method of claim 25,

Determining whether there is a correlation between the plurality of surgical elements,

It is determined whether a correlation exists between the plurality of surgical elements based on the relationship information on a predefined surgical element,

The relationship information for the predefined surgical element,

Method for generating learning data based on the surgical image, characterized in that the information set on the basis of at least one of the type of operation element, the position information of the operation element, the state information of the operation element, the operation information of the operation element.
The method of claim 21,

And constructing learning data based on the relationship expression information generated for each of the plurality of image frames.
The method of claim 27,

And learning based on the surgical data to derive a specific learning result.
Memory for storing one or more instructions; And

A processor for executing the one or more instructions stored in the memory,

The processor executes the one or more instructions,

Acquiring a surgical image comprising a plurality of image frames;

Recognizing surgical recognition information from each of the plurality of image frames; And

For each of the plurality of image frames, generating relational representation information representing a relationship between surgical elements included in the surgical recognition information based on the surgical recognition information. Device.
A computer program, coupled to a computer, which is hardware, stored on a recording medium readable by a computer to perform the method of claim 21.
In the blood vessel information providing method using a vessel model performed by a computer,

Obtaining a vascular polygon constituting a blood vessel in the vascular model;

Constructing training data based on the vascular polygon information and the connection information of the vascular polygon; And

Vessel information providing method using a vessel model, characterized in that it comprises the step of classifying the vessel on the basis of the learning data.
The method of claim 31, wherein

The vascular polygon information,

Location vector of the vascular polygon, and vessel information of the vascular polygon;

The connection information of the vascular polygon is

And a vessel vector adjacent to the vessel polygon, and position vector difference information between the neighbor polygon and the vessel polygon.
33. The method of claim 32,

The position vector of the vascular polygon is

A position vector of a midpoint of the vascular polygon with respect to a predetermined reference vessel point in the vessel model,

Blood vessel information of the vascular polygon,

Method of providing blood vessel information using a blood vessel model, characterized in that the type of blood vessels determined based on the predetermined artery and vein classification information.
The method of claim 31, wherein

Obtaining at least one reference organ,

Comprising the learning data,

Method for providing blood vessel information using a blood vessel model, characterized in that for constructing the learning data by deriving the distance information between the reference organ and the blood vessel polygon.
The method of claim 34, wherein

Acquiring the reference organs,

Placing the reference organ based on a predetermined reference vessel point in the vessel model; And

Deriving a position vector for the reference organ based on the reference vessel point,

Distance information between the reference organ and the vascular polygon,

The vessel information providing method using the vessel model, which is derived based on the position vector of the reference organ and the position vector of the vascular polygon.
The method of claim 31, wherein

The step of classifying the blood vessels,

A vessel information providing method using a vessel model, characterized in that to generate a vessel classification learning model for classifying vessels by performing learning based on the learning data.
The method of claim 31, wherein

Obtaining a two-dimensional projection image of projecting a blood vessel corresponding to the blood vessel polygon into a two-dimensional space; And

And performing learning using the two-dimensional projection image as additional learning data.
The method of claim 31, wherein

Obtaining a new vascular polygon model,

The step of classifying the blood vessels,

Vessel information providing method using a vessel model, characterized in that to classify the vessel in the new vessel polygon model based on the learning data.
The method of claim 38,

Acquiring the new blood vessel polygon model,

Extracting at least one vascular polygon in the new vascular polygon model; And

Obtaining at least one of a position vector of the at least one vascular polygon, an adjacent polygon adjacent to the at least one vascular polygon, a position vector of the adjacent polygon, and distance information with a reference organ,

The step of classifying the blood vessels,

And at least one of a vessel type and vessel hierarchy information corresponding to the at least one vessel polygon based on the learning data.
Memory for storing one or more instructions; And

A processor for executing the one or more instructions stored in the memory,

The processor executes the one or more instructions,

Obtaining a vascular polygon constituting a blood vessel in the vascular model;

Constructing training data based on the vascular polygon information and the connection information of the vascular polygon; And

And classifying blood vessels based on the learning data.
A computer program, coupled with a computer, which is hardware, stored on a recording medium readable by a computer to perform the method of claim 31.