WO2019164278A1

WO2019164278A1 - Method and device for providing surgical information using surgical image

Info

Publication number: WO2019164278A1
Application number: PCT/KR2019/002096
Authority: WO
Inventors: 이종혁; 형우진; 양훈모; 김호승; 어수행
Original assignee: (주)휴톰
Priority date: 2018-02-20
Filing date: 2019-02-20
Publication date: 2019-08-29

Abstract

A method for providing surgical information using surgical image is provided. The method comprises the steps of: acquiring a surgical image for a specific surgery; recognizing a surgical step in the specific surgery corresponding to the surgical image; estimating a candidate organ group comprising at least one organ which can be extracted in the surgical step; and specifying an organ area in the surgical image on the basis of the positional relationship between organs in the candidate organ group.

Description

Surgical information provision method and apparatus using surgical images

The present invention relates to a method and apparatus for providing surgical information using a surgical image.

Medical surgery can be classified into open surgery, laparoscopic surgery, minimally invasive surgery (MIS) including radio surgery, radio surgery, and the like. Laparoscopic surgery refers to surgery performed by medical staff to see and touch the part to be treated. Minimally invasive surgery is also known as keyhole surgery, and laparoscopic surgery and robotic surgery are typical. In laparoscopic surgery, a small hole is made in a necessary part without opening, and a laparoscopic with a special camera is attached and a surgical tool is inserted into the body and observed through a video monitor. Microsurgery is performed using a laser or a special instrument. In addition, robot surgery is to perform minimally invasive surgery using a surgical robot. Furthermore, radiation surgery refers to surgical treatment with radiation or laser light outside the body.

In the case of such medical surgery, especially in the case of minimally invasive surgery, the surgical image is obtained during the actual surgery and the surgery is performed based on this. Therefore, it is important to provide the surgical site and various information related thereto through the surgical image acquired during the actual surgery.

The problem to be solved by the present invention is to provide a method and apparatus for providing surgical information using a surgical image.

The problem to be solved by the present invention is to provide a method and apparatus for estimating the observable organs in the surgical image by identifying the current surgical stage through the surgical image, and providing information on the estimated organs.

The problem to be solved by the present invention is to provide a method and apparatus for providing the same simulation environment as the actual progress of the operation through the surgical image.

The problem to be solved by the present invention is to provide a method and apparatus for providing information of organs in a more significant surgical image by analyzing the surgical image using deep learning.

Problems to be solved by the present invention are not limited to the above-mentioned problems, and other problems not mentioned will be clearly understood by those skilled in the art from the following description.

Surgical information providing method using a surgical image performed by a computer according to an embodiment of the present invention, obtaining a surgical image for a specific operation, recognizing the operation step in the specific surgery corresponding to the surgical image Estimating an organ candidate group including at least one organ that can be extracted in the operation step, and specifying an organ region in the surgical image based on a positional relationship between organs in the organ candidate group. do.

In one embodiment of the present invention, the step of recognizing the operation step, the step of performing the learning based on at least one surgical image of the previous time and the surgical image of the current time point, for the surgical image through the learning Deriving context information, and recognizing a surgery step corresponding to the surgery image among the surgery steps determined for the specific surgery based on the context information on the surgery image.

In one embodiment of the present invention, the step of recognizing the operation step, when the specific surgery comprises a hierarchical structure including the lowest hierarchy to the highest hierarchy, any one operation operation belonging to a specific hierarchy on the hierarchy It may be recognized as a surgical step corresponding to the surgical image.

In one embodiment of the present invention, specifying the organ region in the surgical image, learning the positional relationship between the organs in the organ candidate group, calculating the position information of the organs included in the surgical image, And specifying an organ region in which the organ exists in the surgical image based on the location information of the organ.

In one embodiment of the present invention, the step of specifying the organ region in the surgical image, by learning the texture (texture) information on the organs in the organ candidate population, the texture information of the organs included in the surgical image The method may further include detecting an organ region corresponding to the texture information of the organ in the surgical image based on the calculation and the location information of the organ.

In an embodiment of the present invention, the method may further include displaying the specific organ region on the surgical image and providing the same to the user.

In one embodiment of the present invention, further comprising the step of matching the specified organ region on the virtual body model, and simulating based on the matched organ region on the virtual body model, the virtual body model is The 3D modeling data may be generated based on medical image data photographing the inside of the body of the specific surgery subject.

In one embodiment of the present invention, the step of simulating, by applying the surgical instruments and surgical operations used during the specific surgery based on the matched organ region on the virtual body model, but through the simulation The method may further include generating cue sheet data for the specific surgery by obtaining simulation data.

An apparatus according to an embodiment of the present invention includes a memory for storing one or more instructions, and a processor for executing the one or more instructions stored in the memory, wherein the processor executes the one or more instructions to perform a particular operation. Obtaining a surgical image for the patient, recognizing a surgical step in the specific surgery corresponding to the surgical image, estimating an organ candidate group including at least one organ that can be extracted in the surgical step, and Specifying an organ region in the surgical image based on the positional relationship between organs in the organ candidate population.

A computer program according to an embodiment of the present invention is combined with a computer, which is hardware, and stored in a computer-readable recording medium to perform a method for providing surgical information using the surgical image.

According to the present invention, it is possible to accurately determine the operation stage in the surgery currently being performed through the surgical image. In addition, it is effective in estimating the observable organs in the surgical image by identifying the current stage of surgery, thereby improving the long-term recognition rate.

According to the present invention, it is possible to accurately determine the position and shape of the actual organs through the surgical image. In addition, the location and shape of these organs can be used to provide more meaningful information to the medical staff performing the surgery.

According to the present invention, by accurately specifying the position and shape of the actual organs through the surgical image it is possible to implement the same simulation environment as the actual operation is in progress. In addition, by implementing the same simulation environment as the actual operation process, medical staff can perform rehearsal or virtual surgery more accurately and effectively, thereby improving the learning effect of medical staff.

Effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the following description.

1 is a schematic diagram of a system capable of performing robot surgery according to an embodiment of the present invention.

2 and 3 are flowcharts illustrating a method for providing surgical information using a surgical image according to an embodiment of the present invention.

4 is an example showing a process of recognizing a surgical step based on a surgical image according to an embodiment of the present invention.

5 is an example illustrating a process of specifying an organ region in a surgical image based on information about an organ in an organ candidate group according to an embodiment of the present invention.

6 is a view schematically showing the configuration of an apparatus 200 for performing a method for providing surgical information using a surgical image according to an embodiment of the present invention.

Advantages and features of the present invention and methods for achieving them will be apparent with reference to the embodiments described below in detail with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but can be embodied in various different forms, and the present embodiments only make the disclosure of the present invention complete, and those of ordinary skill in the art to which the present invention belongs. It is provided to fully inform the skilled worker of the scope of the invention, which is defined only by the scope of the claims.

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. In this specification, the singular also includes the plural unless specifically stated otherwise in the phrase. As used herein, "comprises" and / or "comprising" does not exclude the presence or addition of one or more other components in addition to the mentioned components. Like reference numerals refer to like elements throughout, and "and / or" includes each and all combinations of one or more of the mentioned components. Although "first", "second", etc. are used to describe various components, these components are of course not limited by these terms. These terms are only used to distinguish one component from another. Therefore, of course, the first component mentioned below may be a second component within the technical spirit of the present invention.

Unless otherwise defined, all terms used in the present specification (including technical and scientific terms) may be used in a sense that can be commonly understood by those skilled in the art. In addition, terms that are defined in a commonly used dictionary are not ideally or excessively interpreted unless they are specifically defined clearly.

As used herein, the term "part" or "module" refers to a hardware component such as software, FPGA, or ASIC, and the "part" or "module" plays certain roles. However, "part" or "module" is not meant to be limited to software or hardware. The “unit” or “module” may be configured to be in an addressable storage medium or may be configured to play one or more processors. Thus, as an example, a "part" or "module" may include components such as software components, object-oriented software components, class components, and task components, processes, functions, properties, Procedures, subroutines, segments of program code, drivers, firmware, microcode, circuitry, data, databases, data structures, tables, arrays, and variables. Functions provided within components and "parts" or "modules" may be combined into smaller numbers of components and "parts" or "modules" or into additional components and "parts" or "modules". Can be further separated.

As used herein, the term "computer" includes all the various devices capable of performing arithmetic processing to provide a result to a user. For example, a computer can be a desktop PC, a notebook, as well as a smartphone, a tablet PC, a cellular phone, a PCS phone (Personal Communication Service phone), synchronous / asynchronous The mobile terminal of the International Mobile Telecommunication-2000 (IMT-2000), a Palm Personal Computer (PC), a Personal Digital Assistant (PDA), and the like may also be applicable. In addition, when a head mounted display (HMD) device includes a computing function, the HMD device may be a computer. Also, the computer may correspond to a server that receives a request from a client and performs information processing.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings.

According to FIG. 1, the robotic surgical system includes a medical imaging apparatus 10, a server 100, a control unit 30 provided in an operating room, a display 32, and a surgical robot 34. According to an embodiment, the medical imaging apparatus 10 may be omitted in the robot surgery system according to the disclosed embodiment.

In one embodiment, surgical robot 34 includes imaging device 36 and surgical instrument 38.

In one embodiment, the robot surgery is performed by the user controlling the surgical robot 34 using the control unit 30. In one embodiment, the robot surgery may be automatically performed by the controller 30 without the user's control.

The server 100 is a computing device including at least one processor and a communication unit.

The controller 30 includes a computing device including at least one processor and a communication unit. In one embodiment, the control unit 30 includes hardware and software interfaces for controlling the surgical robot 34.

The imaging device 36 includes at least one image sensor. That is, the imaging device 36 includes at least one camera device and is used to photograph an object, that is, a surgical site. In one embodiment, the imaging device 36 includes at least one camera coupled with a surgical arm of the surgical robot 34.

In an embodiment, the image photographed by the photographing apparatus 36 is displayed on the display 340.

In one embodiment, surgical robot 34 includes one or more surgical tools 38 that can perform cutting, clipping, fixing, grabbing operations, and the like, of the surgical site. Surgical tool 38 is used in conjunction with the surgical arm of the surgical robot 34.

The controller 30 receives information necessary for surgery from the server 100 or generates information necessary for surgery and provides the information to the user. For example, the controller 30 displays the information necessary for surgery, generated or received, on the display 32.

For example, the user performs the robot surgery by controlling the movement of the surgical robot 34 by manipulating the control unit 30 while looking at the display 32.

The server 100 generates information necessary for robotic surgery using medical image data of an object previously photographed from the medical image photographing apparatus 10, and provides the generated information to the controller 30.

The controller 30 displays the information received from the server 100 on the display 32 to provide the user, or controls the surgical robot 34 by using the information received from the server 100.

In one embodiment, the means that can be used in the medical imaging apparatus 10 is not limited, for example, other various medical image acquisition means such as CT, X-Ray, PET, MRI may be used.

As described above, when performing a robot operation, it is possible to obtain data including various surgical information in the surgical image taken during the surgical procedure or the control of the surgical robot. As described above, the present invention is to provide more meaningful information to the medical staff undergoing surgery by using the surgical image or the surgical data that can be obtained in the surgical process.

Hereinafter, for convenience of description, a "computer" will be described as performing a surgical information providing method using a surgical image according to an exemplary embodiment disclosed herein. "Computer" may mean the server 100 or the controller 30 of FIG. 1, but is not limited thereto and may be used to encompass a device capable of performing computing processing. For example, the computer may be a computing device provided separately from the device shown in FIG. 1.

In addition, the embodiments disclosed below may not be applicable only in connection with the robotic surgery system illustrated in FIG. 1, but may be applied to all kinds of embodiments that may acquire and utilize a surgical image in a surgical procedure.

2 and 3, in the surgical information providing method using a surgical image according to an embodiment of the present invention, obtaining a surgical image for a specific operation (S100), the specific surgery corresponding to the surgical image Recognizing a surgical step in (S200), estimating the organ candidate group including at least one organ that can be extracted in the surgical step (S300), and based on the positional relationship between organs in the organ candidate group And specifying an organ region in the surgical image (S400). Hereinafter, a detailed description of each step will be described.

The computer may acquire a surgical image for a specific surgery (S100).

When performing a specific surgery on the subject, the medical staff may perform the actual surgery on the subject directly, or perform minimally invasive surgery using a laparoscopic or endoscope as well as the surgical robot described in FIG. 1. .

In this case, the computer may acquire a surgical image photographing a scene including a surgical operation performed in the surgical procedure, a surgical tool related thereto, a surgical site, and the like. In an embodiment, the computer may acquire a surgical image photographing a scene including a surgical site and a surgical tool that is currently undergoing surgery from a camera that enters the body of the subject.

The surgical image may include one or more image frames. Each image frame may represent a scene in which a surgical operation is performed, including a surgical site, a surgical tool, and the like of a surgical subject. For example, the surgical image may be composed of image frames in which a surgical operation is recorded for each scene (scene) according to time during the surgical procedure. Alternatively, the surgical image may be composed of image frames that record each surgical scene according to the spatial movement such as the surgical site or the position of the camera during the surgery.

The computer may recognize a surgical workflow stage in a specific surgical process corresponding to the surgical image acquired in step S100 (S200).

In one embodiment, the computer may perform learning using deep learning on a surgical image (eg, at least one image frame) acquired during a specific surgical procedure, and the context information of the surgical image through the learning. Can be derived. In addition, the computer may recognize a surgery stage corresponding to the surgery image on a specific surgery procedure including at least one surgery stage based on the context information of the surgery image. A detailed process thereof will be described with reference to FIG. 4.

Referring to FIG. 4, the computer may learn a surgical image (ie, at least one image frame) acquired at a specific surgery.

In an embodiment, first, the computer may perform learning using a convolutional neural network (CNN) for each image frame included in the surgical image (S210). For example, the computer may learn characteristics of the surgical image by inputting each image frame to at least one layer (eg, a convolution layer). As a result of this learning, the computer can infer what each image frame represents or represents.

Next, the computer may perform learning using a recurrent neural network (RNN) on the surgical image (that is, at least one image frame) derived as a result of learning using the CNN (S220). In an embodiment, the computer may receive the surgical image in units of frames and learn what the surgical image in the frame means by using an RNN (eg, an LSTM method). For example, the computer receives a surgical image (eg, the first image frame) of the current view and at least one surgical image (eg, the second to nth image frames) of the previous view and uses the RNN (eg, LSTM). Learning may be performed and context information of the surgical image (ie, the first image frame of the current view) of the current view may be derived as the learning result. Here, the context information is information indicating what the surgical image means, and may include information related to a specific surgical operation in the surgical procedure.

The computer may recognize the meaning (ie, context information) of the surgery image based on the learning result of the surgery image, and recognize the surgery stage corresponding to the surgery image. In one embodiment, the computer may recognize the operation step for each image frame based on the context information for each image frame derived as a learning result of the RNN (S230).

At this time, the specific surgical process may be composed of at least one surgical step. In one embodiment, at least one surgical step may be predefined for each particular surgery. For example, each operation step may be classified according to the time course of a specific operation, or each operation step may be classified to correspond to each operation part based on the operation site. Alternatively, each operation stage may be classified based on the position of the camera or the movement range of the camera during a specific operation, or each operation based on a change (eg, replacement) of a surgical tool during a specific operation. The steps may be categorized.

In addition, the surgical stages thus classified may be configured in a hierarchical structure. In one embodiment, a specific surgical procedure may be classified step by step from the lowest hierarchy (level) to the highest hierarchy (level) to form a hierarchical structure. Here, the lowest layer is composed of the smallest units representing the surgical procedure, and may include a minimum operation operation having a meaning as one minimum operation. For example, the computer may recognize a surgical operation that represents one constant motion pattern as a minimum operation motion such as cutting, grabbing, moving, etc., and configure it as the lowest hierarchy (ie, the lowest surgical step). In addition, the computer may be configured in a higher layer by grouping at least one minimal surgery operation according to a specific classification criteria (eg, time course, surgical site, camera movement, change of surgical tool, etc.). For example, when each of the minimum surgery operations indicating grabbing, moving, cutting, and the like of a surgical tool is performed in a predetermined order, and the predetermined order is connected, the minimum operation can be grouped into one higher layer when it has meaning as a specific operation. have. As another example, when each minimal surgical operation such as clipping, moving, or cutting is performed continuously, it may be recognized that this is a surgical operation for cutting blood vessels and may be grouped into one higher layer. As another example, when each minimal surgical operation such as grabbing, lifting, cutting, or kicking is performed in succession, it may be recognized that this is a surgical operation to remove fat and grouped into one higher layer. As such, when it is recognized as an upper operation operation (ie, an upper operation step) having a specific meaning from lower operation procedures (that is, lower operation steps), it may be classified into a higher layer and this process may be repeated. It can be done with Thus, a hierarchical structure can be finally formed up to the highest hierarchy (ie, the highest surgical stage) of a specific surgical procedure. For example, certain surgical procedures may have a hierarchical structure, such as a tree.

That is, since the computer can estimate the meaning of the surgical image by deriving context information for each image frame, the computer can recognize the specific operation stage to which the image frame belongs among the predefined operation stages for the specific operation. . In this case, the recognized specific surgery stage may include a hierarchical layer (eg, first level) hierarchically configured for a specific surgery, and a specific layer (eg, second to nth levels) among the highest hierarchy (n level). It can mean any one of the surgical operation) belonging to any one level. That is, the computer may recognize the operation step belonging to a specific layer on the hierarchical structure of the operation process through the context information for each image frame.

Referring back to FIGS. 2 and 3, the computer may estimate an organ candidate group including at least one organ that may be extracted from the surgery step recognized in step S200 (S300).

As described above, the specific surgery process may be composed of predetermined surgery steps, and thus, various pieces of information (eg, information about a specific surgery operation, a specific surgery part, a specific surgery tool, a camera position, etc.) may be obtained from each predetermined surgery step. Can be extracted. In one embodiment, the computer may extract the organ-organ information observable at each operation stage to generate a long-term candidate group for each operation stage. Alternatively, the computer may obtain a long-term candidate group for each surgery stage that is previously generated.

For example, as shown in FIG. 3, the computer may recognize first to nth surgical steps for each image frame. In this case, the computer may extract organs that may be observed at the time of surgery from each of the first to n th surgical steps, and may construct an organ candidate group including the extracted organs for each of the surgical steps. For example, the computer recognizes a k-stage operation (eg, left hepatic artery (LHA) incision stage) from a specific surgical image (k-th image frame), and uses the stomach, liver, and spleen as organs observable in the recognized k-stage operation. It can be estimated and composed of long-term candidate groups.

The computer may specify an organ region in a surgical image (that is, at least one image frame) based on information on at least one organ in the organ candidate population estimated in step S300 (S400).

In one embodiment, the computer may specify an organ region in the surgical image based on at least one of the positional relationship between organs in the organ candidate population estimated from the operation stage for the surgical image and texture information of the organs. Can be. A detailed process thereof will be described with reference to FIG. 5.

Referring to FIG. 5, the computer may calculate positional information of at least one organ included in a surgical image by learning a positional relationship between organs in an organ candidate group (S410).

In one embodiment, the computer may construct a neural network (eg, a backbone network) that includes at least one layer (eg, a convolution layer) to learn the positional relationship between organs in the organ candidate population. For example, the computer may input a surgical image (that is, a specific image frame) and an organ candidate group into the backbone network, and determine the positional relationship between organs in the organ candidate group through learning in the first convolutional layer. The computer may calculate location information of at least one organ included in the surgical image, based on a learning result about the positional relationship between organs in the organ candidate group. Here, the position information of the organ may use coordinate values of a specific point (for example, a center point) among regions where organs are distributed in the surgical image.

Each organ has its own fixed position inside the body. In addition, since the types of organs included in the organ candidate group can be known, location information of organs in the organ candidate group can also be obtained. Thus, the computer can perform the learning based on fixed locations of organs in the organ candidate population. That is, organs having a fixed position may be photographed as if they exist in different positions according to the field of view (FOV) of the camera, the angle of the camera, the position of the camera, etc., but the positional relationship between the organs is maintained. do. In other words, since the spatial topological relationship between organs is maintained, the computer can learn the spatial topological relationship between organs in the organ candidate group. The computer can recognize the organ placement existing in the surgical image by learning the spatial topological relationships between these organs. In addition, the computer may calculate the position information of the organs in the surgical image based on the organ placement.

The computer may learn texture information about organs in the organ candidate group to calculate texture information of at least one organ included in the surgical image (S420).

In one embodiment, the computer may include at least one layer (eg, a convolution layer) in a neural network (eg, a backbone network) to learn texture information about organs in the organ candidate population. For example, the computer may input a surgical image (ie, a specific image frame) and an organ candidate group into the backbone network, and derive texture information of organs in the organ candidate group through learning in the second convolutional layer. The computer may calculate texture information of at least one organ included in the surgical image based on the learning result of the texture information of the organs in the organ candidate group.

The computer may specify an organ region in which an organ exists in the surgical image based on the location information of the organs in the surgical image calculated in step S410, and detect an area corresponding to the texture information of the organ calculated in step S420. S430).

In one embodiment, the computer recognizes the organ at the location based on the location information of the organs present in the surgical image, and in the surgical image having the same or similar texture information using the texture information calculated for the recognized organs The distribution area can be detected. For example, if the computer calculates position information of three organs (eg, A, B, and C organs) in the surgical image through learning, the computer recognizes the A organ at the position of the A organ in the surgical image. The A organ region matching the texture information of the A organ may be detected in the surgical image based on the texture information of the A organ calculated through learning. In addition, the computer can detect the B organ region and the C organ region by repeating the same process for the B organ and the C organ. Thus, the computer can finally specify each organ region in which each organ included in the surgical image is present.

According to the surgical information providing method using the surgical image according to an embodiment of the present invention described above, the computer may display the specific organ region in the surgical image to provide to the medical staff performing the actual surgery in real time. For example, when the surgical image is output on the screen provided to the medical staff during the actual surgery, the computer can specify the organ region present in the surgical image through the steps S100 ~ S400, so that the specified organ region It can be displayed on the corresponding surgical image output on the screen. Accordingly, it is possible to more effectively provide meaningful information (important long-term information) through the surgical image in the surgical process.

In addition, the computer can match the organ region specified in the surgical image on the virtual body model. In addition, the computer may perform the simulation based on the long-term region matched on the virtual body model.

Here, the virtual body model may be three-dimensional modeling data generated based on medical image data (eg, medical images taken through CT, PET, MRI, etc.) previously photographed inside the body of the patient. . For example, the model may be modeled in accordance with the body of the surgical subject, and may be corrected to the same state as the actual surgical state. Medical staff can perform rehearsals or simulations using a virtual body model that is implemented in the same way as the physical state of the subject, and can experience the same state as during the actual surgery. In addition, when performing a virtual surgery using a virtual body model, virtual surgery data including rehearsal or simulation behavior for the virtual body model can be obtained. For example, the virtual surgery data may be a virtual surgery image including a surgical site on which a virtual surgery is performed on a virtual body model, or may be data recorded on a surgical operation performed on the virtual body model.

In one embodiment, the computer can obtain the type of organs present in the surgical image, the location information of the organs, the positional relationship between the organs, the texture information of the organs, etc. Match exactly on the model. This matching enables the same situation in the virtual body model that is currently being performed in a real surgery. Accordingly, the computer may perform simulation by applying surgical tools and surgical operations used in the surgical procedure actually performed by the medical staff based on the matched organ region on the virtual body model. In other words, the simulation can be performed in the same way as the current physical surgery is performed through the virtual body model, and it can play a role of assisting medical staff to perform the surgery more accurately and effectively. In addition, during the re-surgery or learning can be reproduced the same surgical process as the actual surgical situation, thereby improving the learning effect of the medical staff.

In addition, the computer may acquire simulation data through simulation on a virtual body model, and generate cue sheet data on a corresponding surgical procedure based on the obtained simulation data.

In an embodiment, the computer may acquire simulation data that records a target surgical site, a surgical tool, a surgical operation, and the like by performing a simulation using a virtual body model. The computer may generate cuesheet data about the actual surgical procedure being performed by the medical staff based on the simulation data containing this record.

Here, the cue sheet data may be data consisting of information in which the surgical procedures are arranged in order according to time based on the surgical operation performed at the time of surgery. In one embodiment, the cue sheet data may include surgical information on a surgical operation that may be performed as a meaningful surgical operation or a minimum unit. In this case, the surgical information may include surgical tool information, surgical site information, surgical operation information, and the like.

Such cuesheet data can be used to implement the same surgical operations as the actual surgical procedure, through which can be used to evaluate the operation or to be used as a learning model later. In addition, it is possible to repeatedly reproduce the surgical process using the cue sheet data, and through this it can be effectively utilized in the same or similar actual surgery.

Referring to FIG. 6, the processor 210 may include a connection passage (eg, a bus or the like) that transmits and receives a signal with one or more cores (not shown) and a graphic processor (not shown) and / or other components. ) May be included.

The processor 210 according to an embodiment executes one or more instructions stored in the memory 220 to perform a method of providing surgery information using the surgery image described with reference to FIGS. 2 to 5.

In one example, the processor 210 acquires a surgical image for a specific surgery by executing one or more instructions stored in the memory 220, recognizing a surgery step in the specific surgery corresponding to the surgical image, Estimating an organ candidate group including at least one organ that can be extracted in a surgical step, and specifying an organ region in the surgical image based on a positional relationship between organs in the organ candidate group. have.

On the other hand, the processor 210 is a random access memory (RAM) and a ROM (Read-Only Memory) for temporarily and / or permanently storing signals (or data) processed in the processor 210. , Not shown) may be further included. In addition, the processor 210 may be implemented in the form of a system on chip (SoC) including at least one of a graphic processor, a RAM, and a ROM.

The memory 220 may store programs (one or more instructions) for processing and controlling the processor 210. Programs stored in the memory 220 may be divided into a plurality of modules according to their functions.

The surgical information providing method using the surgical image according to the embodiment of the present invention described above may be implemented as a program (or an application) to be executed by being combined with a computer which is hardware and stored in a medium.

The above-described program includes C, C ++, JAVA, machine language, etc. which can be read by the computer's processor (CPU) through the computer's device interface so that the computer reads the program and executes the methods implemented as the program. Code may be coded in the computer language of. Such code may include functional code associated with a function or the like that defines the necessary functions for executing the methods, and includes control procedures related to execution procedures necessary for the computer's processor to execute the functions according to a predetermined procedure. can do. In addition, the code may further include memory reference code for additional information or media required for the computer's processor to execute the functions at which location (address address) of the computer's internal or external memory should be referenced. have. Also, if the processor of the computer needs to communicate with any other computer or server remotely in order to execute the functions, the code may be used to communicate with any other computer or server remotely using the communication module of the computer. It may further include a communication related code for whether to communicate, what information or media should be transmitted and received during communication.

The stored medium is not a medium for storing data for a short time such as a register, a cache, a memory, but semi-permanently, and means a medium that can be read by the device. Specifically, examples of the storage medium include, but are not limited to, a ROM, a RAM, a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, and the like. That is, the program may be stored in various recording media on various servers to which the computer can access or various recording media on the computer of the user. The media may also be distributed over network coupled computer systems so that the computer readable code is stored in a distributed fashion.

The steps of a method or algorithm described in connection with an embodiment of the present invention may be implemented directly in hardware, in a software module executed by hardware, or by a combination thereof. Software modules may include random access memory (RAM), read only memory (ROM), erasable programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory, hard disk, removable disk, CD-ROM, or It may reside in any form of computer readable recording medium well known in the art.

In the above, embodiments of the present invention have been described with reference to the accompanying drawings, but those skilled in the art to which the present invention pertains may realize the present invention in other specific forms without changing the technical spirit or essential features thereof. I can understand that. Therefore, it should be understood that the embodiments described above are exemplary in all respects and not restrictive.

Claims

In the surgical information providing method using a surgical image performed by a computer,

Obtaining a surgical image for a specific surgery;

Recognizing a surgery step in the specific surgery corresponding to the surgery image;

Estimating an organ candidate group including at least one organ that can be extracted in the surgery step; And

And specifying an organ region in the surgical image based on the positional relationship between the organs in the organ candidate group.
The method of claim 1,

Recognizing the operation step,

Performing learning based on at least one surgical image of a previous time and a surgical image of a current time;

Deriving context information on the surgical image through the learning; And

And recognizing a surgery stage corresponding to the surgery image among the surgery stages determined for the specific surgery based on the context information on the surgery image.
The method of claim 2,

Recognizing the operation step,

When the specific surgery has a hierarchical structure including the lowest hierarchy to the highest hierarchy, a surgical image characterized in that any one operation operation belonging to a specific hierarchy on the hierarchy is recognized as a surgical step corresponding to the surgical image. Surgical information provided method.
The method of claim 1,

The step of specifying the organ region in the surgical image,

Calculating positional information of organs included in the surgical image by learning a positional relationship between organs in the organ candidate group; And

And specifying an organ region in which the organ exists in the surgical image based on the location information of the organ.
The method of claim 4, wherein

The step of specifying the organ region in the surgical image,

Calculating texture information of organs included in the surgical image by learning texture information of organs in the organ candidate population; And

And detecting an organ region corresponding to texture information of the organ in the surgical image based on the location information of the organ.
The method of claim 1,

Surgical information providing method using a surgical image further comprising the step of providing to the user by displaying the specified organ region on the surgical image.
The method of claim 1,

Matching the specified organ region on a virtual body model; And

Simulating based on the matched organ region on the virtual body model,

The virtual body model,

Surgical information providing method using the surgical image, characterized in that the 3D modeling data generated based on the medical image data taken inside the body of the specific surgery target.
The method of claim 7, wherein

The simulating step,

Based on the matched organ region on the virtual body model is simulated by applying the surgical instruments and surgical operations used during the specific surgery,

Surgical information providing method using the surgical image further comprises the step of obtaining the simulation data through the simulation to generate the cue sheet data for the specific surgery.
Memory for storing one or more instructions; And

A processor for executing the one or more instructions stored in the memory,

The processor executes the one or more instructions,

Obtaining a surgical image for a specific surgery;

Recognizing a surgery step in the specific surgery corresponding to the surgery image;

Estimating an organ candidate group including at least one organ that can be extracted in the surgery step; And

And specifying an organ region in the surgical image based on a positional relationship between organs in the organ candidate population.
A computer program, coupled to a computer, which is hardware, stored on a recording medium readable by a computer so as to perform the method of claim 1.