KR101926123B1 - Device and method for segmenting surgical image - Google Patents

Abstract

Disclosed is a method for segmenting a surgical image comprising the steps of: obtaining a surgical image by using a computer; recognizing at least one object included in each of one or more frames of the surgical image; determining a position and a motion of an imaging device to take a photograph of the surgical image and each of the at least one recognized object; and segmenting the surgical image into at least one first classification unit group capable of recognizing an operation of a predetermined first classification unit based on the determined result.

Description

≪ Desc / Clms Page number 1 > DEVICE AND METHOD FOR SEGMENTING SURGICAL IMAGE &

The present invention relates to a surgical image segmentation method and apparatus.

In the surgical procedure, development of techniques capable of providing information for assisting the surgeon of the doctor is required. In order to provide information for assisting the operation, it is necessary to be able to recognize the operation.

Therefore, it is required to develop a technique in which a computer can recognize a surgical operation from a surgical image.

In recent years, deep learning has been widely used in the analysis of medical images. Deep learning is defined as a set of machine learning algorithms that try to achieve high levels of abstraction (a task that summarizes key content or functions in large amounts of data or complex data) through a combination of several nonlinear transformation techniques. Deep learning can be viewed as a field of machine learning that teaches computers how people think in a big way.

SUMMARY OF THE INVENTION It is an object of the present invention to provide a surgical image segmentation method and apparatus.

The problems to be solved by the present invention are not limited to the above-mentioned problems, and other problems which are not mentioned can be clearly understood by those skilled in the art from the following description.

According to an aspect of the present invention, there is provided a surgical image segmentation method comprising: obtaining a surgical image by a computer; recognizing at least one object included in each of one or more frames of the surgical image; The method of claim 1, further comprising the steps of: determining a position and a motion of each of the at least one recognized object; and determining, based on the determination result, Segmentation into one group of classification units.

The step of dividing into the first classification unit group may include a step of dividing the position and the movement of each of the photographing apparatus and the one or more objects based on a result of the determination as to the position and the motion of the photographing apparatus and the one or more objects, And dividing the surgical image into at least one second classification unit group that can be recognized as an operation of the set second classification unit.

The step of dividing into the second group of classification units may include a step of recognizing an operation of the predetermined third classification unit based on the at least one second classification unit operation recognized for each of the photographing apparatus and the at least one object Dividing the surgical image into one or more third classification unit groups, wherein the third classification unit group includes one or more of the second classification unit groups.

Recognizing an operation of the first classification unit from the surgical image; Wherein the step of recognizing the operation of the first classification unit comprises the step of recognizing the motion of the first classification unit based on a positional relationship between the body part and the at least one object included in the surgical image, And recognizing the operation of the mobile terminal.

The recognizing operation of the first classification unit may include acquiring modeling information corresponding to the body part, performing a surgical simulation including the modeling information and the virtual one or more objects, The one or more virtual objects moving according to the movement of the one or more objects, and recognizing the operation of the first classification unit using the simulation result.

The dividing into the first classification unit group may include recognizing an operation of the predetermined fourth classification unit based on the operation of the at least one first classification unit recognized for each of the at least one first classification unit group Dividing the surgical image into one or more fourth classification unit groups, wherein the fourth classification unit group includes one or more of the first classification unit groups.

The method according to claim 1, further comprising recognizing an operation of at least one of the first classification units corresponding to each of the at least one first classification unit group classified as the classified one or more first classification units, And recognizing an operation of at least one of the fourth classification units corresponding to each of the unit groups.

The first classification unit may include information on one or more predetermined operations, and each of the predetermined one or more operations may be assigned a code capable of identifying each of the one or more predetermined operations.

The method may further include recognizing an event occurring in the surgical image and dividing the surgical image into one or more event groups including the recognized event.

According to an aspect of the present invention, there is provided an apparatus for segmenting a surgical image, the apparatus comprising: a memory for storing one or more instructions; and a processor for executing the one or more instructions stored in the memory, An imaging device that acquires a surgical image by executing one or more instructions, recognizes one or more objects included in each of the one or more frames of the surgical image, captures the surgical image, And divides the surgical image into at least one first classification unit group capable of recognizing an operation of a predetermined first classification unit, based on the determination result.

A computer program stored in a recording medium readable by a computer to perform a surgical image segmentation method according to the disclosed embodiments in combination with a computer, which is hardware, is provided to solve the above-described problems.

Other specific details of the invention are included in the detailed description and drawings.

According to the disclosed embodiment, it is possible to divide and recognize the operations included in the surgical image more accurately by dividing and recognizing the surgical image through the stepwise division, and it is possible to divide and recognize the operations included in the surgical image even if the type of surgery or the physical condition of the patient is different. There is an effect that a surgical image segmentation and recognition method can be provided.

The effects of the present invention are not limited to the above-mentioned effects, and other effects not mentioned can be clearly understood by those skilled in the art from the following description.

1 is a diagram illustrating a robot surgery system in accordance with the disclosed embodiment.
2 is a flowchart illustrating a method of segmenting a surgical image according to an embodiment.
3 is a flowchart illustrating a method of spatially segmenting a surgical image according to an embodiment.
4 is a diagram showing an example of a surgical image.
5 is a flowchart illustrating a method of temporally dividing an operation image according to an embodiment.
6 is a flowchart illustrating an example of segmenting and recognizing a surgical image step by step.
7 is a diagram showing an example of a method of recognizing a surgical operation step by step.

BRIEF DESCRIPTION OF THE DRAWINGS The advantages and features of the present invention, and the manner of achieving them, will be apparent from and elucidated with reference to the embodiments described hereinafter in conjunction with the accompanying drawings. It should be understood, however, that the invention is not limited to the disclosed embodiments, but may be embodied in many different forms and should not be construed as limited to the embodiments set forth herein, Is provided to fully convey the scope of the present invention to a technician, and the present invention is only defined by the scope of the claims.

The terminology used herein is for the purpose of illustrating embodiments and is not intended to be limiting of the present invention. In the present specification, the singular form includes plural forms unless otherwise specified in the specification. The terms " comprises "and / or" comprising "used in the specification do not exclude the presence or addition of one or more other elements in addition to the stated element. Like reference numerals refer to like elements throughout the specification and "and / or" include each and every combination of one or more of the elements mentioned. Although "first "," second "and the like are used to describe various components, it is needless to say that these components are not limited by these terms. These terms are used only to distinguish one component from another. Therefore, it goes without saying that the first component mentioned below may be the second component within the technical scope of the present invention.

Unless defined otherwise, all terms (including technical and scientific terms) used herein may be used in a sense that is commonly understood by one of ordinary skill in the art to which this invention belongs. In addition, commonly used predefined terms are not ideally or excessively interpreted unless explicitly defined otherwise.

As used herein, the term "part" or "module" refers to a hardware component, such as a software, FPGA, or ASIC, and a "component" or "module" performs certain roles. However, "part" or " module " is not meant to be limited to software or hardware. A "module " or " module " may be configured to reside on an addressable storage medium and configured to play back one or more processors. Thus, by way of example, "a" or " module " is intended to encompass all types of elements, such as software components, object oriented software components, class components and task components, Microcode, circuitry, data, databases, data structures, tables, arrays, and variables, as used herein. Or " modules " may be combined with a smaller number of components and "parts " or " modules " Can be further separated.

The terms spatially relative, "below", "beneath", "lower", "above", "upper" And can be used to easily describe a correlation between an element and other elements. Spatially relative terms should be understood in terms of the directions shown in the drawings, including the different directions of components at the time of use or operation. For example, when inverting an element shown in the figures, an element described as "below" or "beneath" of another element may be placed "above" another element . Thus, the exemplary term "below" can include both downward and upward directions. The components can also be oriented in different directions, so that spatially relative terms can be interpreted according to orientation.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

As used herein, the term "image" may refer to multi-dimensional data composed of discrete image elements (e.g., pixels in a two-dimensional image and voxels in a 3D image). For example, the image may include a medical image or the like of the object obtained by the CT photographing apparatus.

As used herein, an " object "may be a person or an animal, or part or all of a person or an animal. For example, the subject may comprise at least one of the following: liver, heart, uterus, brain, breast, organs such as the abdomen, and blood vessels.

As used herein, the term "user" may be a doctor, a nurse, a clinical pathologist, a medical imaging specialist, or the like, and may be a technician repairing a medical device.

In the present specification, the term " medical image data " is a medical image captured by a medical image capturing apparatus, and includes all medical images capable of realizing a body of a subject as a three-dimensional model. "Medical image data" may include a computed tomography (CT) image, a magnetic resonance imaging (MRI), a positron emission tomography (PET) image, and the like.

As used herein, the term " virtual body model " refers to a model generated based on medical image data in accordance with an actual patient's body. The " virtual body model " may be generated by modeling the medical image data in three dimensions as it is, or may be corrected after modeling as in actual surgery.

As used herein, the term " detailed operation " means a minimum unit of a surgical operation divided according to a specific criterion.

As used herein, the terms " computer " and " device " encompass all of the various devices that can perform computational processing to provide results to a user. For example, the computer may be a smart phone, a tablet PC, a cellular phone, a personal communication service phone (PCS phone), a synchronous / asynchronous A mobile terminal of IMT-2000 (International Mobile Telecommunication-2000), a Palm Personal Computer (PC), a personal digital assistant (PDA), and the like. In addition, when the head mounted display (HMD) device includes a computing function, the HMD device can be a computer. Also, the computer may correspond to a server that receives a request from a client and performs information processing.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a diagram illustrating a robot surgery system in accordance with the disclosed embodiment.

Referring to FIG. 1, there is shown a simplified schematic representation of a system capable of performing robotic surgery in accordance with the disclosed embodiments.

1, the robotic surgery system includes a medical imaging apparatus 10, a server 20, a control unit 30 provided in an operating room, a display 32, and a surgical robot 34. Depending on the embodiment, the medical imaging equipment 10 may be omitted from the robotic surgery system according to the disclosed embodiment.

In one embodiment, the surgical robot 34 includes a photographing device 36 and a surgical tool 38.

In one embodiment, robotic surgery is performed by the user controlling the surgical robot 34 using the control unit 30. [ In one embodiment, robot surgery may be performed automatically by the control unit 30 without user control.

The server 20 is a computing device including at least one processor and a communication unit.

The control unit 30 includes a computing device including at least one processor and a communication unit. In one embodiment, the control unit 30 includes hardware and software interfaces for controlling the surgical robot 34.

The photographing apparatus 36 includes at least one image sensor. That is, the photographing device 36 includes at least one camera device, and is used for photographing a body part of the object, that is, a surgical part. In one embodiment, the imaging device 36 includes at least one camera coupled with a surgical arm of the surgical robot 34.

In one embodiment, the image photographed at the photographing device 36 is displayed on the display 340. [

In one embodiment, the surgical robot 34 includes one or more surgical tools 38 that can perform cutting, clipping, anchoring, grabbing, etc., of the surgical site. The surgical tool 38 is used in combination with the surgical arm of the surgical robot 34.

The control unit 30 receives information necessary for surgery from the server 20, or generates information necessary for surgery and provides the information to the user. For example, the control unit 30 displays on the display 32 information necessary for surgery, which is generated or received.

For example, the user operates the control unit 30 while viewing the display 32 to perform the robot surgery by controlling the movement of the surgical robot 34.

The server 20 generates information necessary for robot surgery using the medical image data of the object (patient) photographed beforehand from the medical imaging apparatus 10, and provides the generated information to the control unit 30. [

The control unit 30 provides the information received from the server 20 to the user by displaying the information on the display 32 or controls the surgical robot 34 using the information received from the server 20. [

In one embodiment, the means that can be used in the medical imaging equipment 10 is not limited, and various other medical imaging acquiring means such as CT, X-Ray, PET, MRI and the like may be used.

In the disclosed embodiment, the surgical image obtained in the photographing device 36 is transmitted to the control section 30. [

In one embodiment, the control unit 30 may segment the surgical image obtained during the operation in real time.

In one embodiment, the control unit 30 transmits a surgical image to the server 20 during or after surgery.

The server 20 can divide and analyze the surgical image.

Hereinafter, a method of dividing and analyzing a surgical image by the server 20 or the control unit 30 will be described in detail with reference to the drawings. Some or all of the surgical image segmentation method according to the embodiment disclosed herein may be performed in the server 20 or the control unit 30, respectively.

Hereinafter, for convenience of explanation, it is assumed that the " computer " performs the surgical image segmentation method according to the embodiment disclosed herein. The " computer " may mean both the server 20 and the control unit 30. [

In one embodiment, the surgical image may be segmented by various criteria. As an example, a surgical image may be segmented based on the type of object included in the image. The division method based on the kind of object requires a step in which the computer recognizes each object.

The object recognized in the surgical image includes the human body, the object introduced from the outside, and the object generated by itself. The human body includes a body part taken by medical imaging (e.g., CT) followed by surgery and a body part not taken.

For example, a body part photographed by a medical imaging includes an organ, a blood vessel, a bone, a tendon, etc., and such a body part can be recognized based on a 3D modeling image generated based on a medical image.

Specifically, the position, size, and shape of each body part are recognized in advance by a 3D analysis method based on a medical image. The computer defines an algorithm that can grasp the position of a body part corresponding to a surgical image in real time, and based on the information, information on the position, size, and shape of each body part included in the surgical image The body part that is not captured by the medical imaging includes the omentum and the like, which is not captured by the medical image, so it is necessary to recognize it in real time during the operation. For example, the computer can determine the location and size of the omentum using image recognition methods, and predict the location of the vessel in the presence of blood vessels within the omentum.

Objects introduced from the outside include, for example, surgical tools, gauzes, clips, and the like. Since it has predetermined morphological characteristics, it can recognize the computer in real time through image analysis during surgery.

Internally generated objects include, for example, bleeding from the body part. This allows the computer to recognize images in real time during image analysis during surgery.

The movements of organs and organs included in body parts and the causes of objects are all caused by the movement of objects from outside.

Thus, in addition to recognizing each object, a surgical image can be segmented based on the motion of each object. In one embodiment, the surgical image may be segmented based on the motion, i.e., action, of the externally introduced object.

The computer judges the type of each object recognized in the surgical image. The computer determines a motion of each object, that is, a motion of each object based on a predetermined operation defined in advance, a series of operations, The action can be recognized.

The computer recognizes the type of each action, and also recognizes the cause of each action. The computer can divide the surgical image based on the recognized action, and it can recognize from each detailed operation to the type of the whole operation through the stepwise division.

Furthermore, the computer determines the type of predefined operation corresponding to the surgical image from the judgment of the action. When determining the type of surgery, information about the entire surgical procedure can be obtained. If there are multiple surgical processes for the same kind of surgery, one surgical process may be selected based on the doctor's choice, or based on actions recognized up to a certain point in time.

The computer can recognize and predict the surgical stage based on the acquired surgical procedure. For example, if a particular step in a series of surgical procedures is recognized, then the steps following it can be predicted or candidates for possible steps can be culled. Therefore, it is possible to greatly reduce the error rate of the surgical image recognition caused by the occlusion or the like. Further, when the surgical image deviates greatly from the predictable surgical stage by more than a predetermined error range, it may be recognized that a surgical error situation has occurred.

In addition, the computer can make a judgment on each action based on the recognition of each action. For example, a computer can recognize the necessity and effectiveness of each action.

Specifically, the computer can make a judgment as to whether each action was necessary or unnecessary. In addition, the computer can determine whether each action was performed efficiently, even if each action was required. This is used to provide an operative report, eliminate unnecessary operations in the surgical procedure, and streamline inefficient operations.

As described above, the surgical image is largely divided into components including body parts (organ and omentum), objects introduced from the outside, objects generated inside, actions, types of surgery, necessity and efficiency of each action . That is, instead of recognizing the surgical image as a whole, the computer divides the surgical image into a component unit that minimizes mutual overlapping, including all elements of the surgical image as much as possible. Based on the divided component units, By recognizing, the surgical image can be recognized more specifically and more easily.

Hereinafter, a method for dividing a surgical image and recognizing a surgical operation step by step, based on an action, among the above-described components will be described in detail.

2 is a flowchart illustrating a method of segmenting a surgical image according to an embodiment.

In step S110, the computer acquires a surgical image.

In one embodiment, a surgical image is acquired from the imaging device 36 shown in FIG. The surgical image includes one or more frames.

In one embodiment, the surgical image comprises a portion of a body part of a subject, i.e., a surgical site. In one embodiment, the surgical image comprises one or more objects. In one embodiment, the one or more objects included in the surgical image include one or more surgical tools 38. In addition, one or more objects included in the surgical image may include tools and consumables used in surgery, such as gauze or clip.

In step S120, the computer recognizes one or more objects included in each of one or more frames of the surgical image obtained in step S110.

In one embodiment, the computer obtains information about the location of the imaging device 36 and the one or more surgical tools 38 from the surgical robot 34, and based on the acquired information, The position can be judged and recognized. In addition, the computer can determine the kind of one or more objects based on the information obtained from the surgical robot 34. [

In one embodiment, the computer may perform image analysis of the surgical image to recognize one or more objects. The computer may also perform image analysis of the surgical image to determine the type of one or more recognized objects.

In one embodiment, the computer can perform learning based on a surgical image labeled with the location and type of one or more objects. The types of learning methods are not limited, and various machine learning methods such as instructional learning, non-instructional learning, and reinforcement learning can be used. The computer can perform image analysis based on the learning results and determine the location and type of one or more objects.

In one embodiment, the computer may use the image analysis results for the surgical image and the information obtained from the surgical robot 34 to determine the location and type of one or more objects.

In step S130, the computer determines the position and movement of each of the one or more objects recognized in step S120. Further, the computer determines the position and movement of the photographing device 36 for photographing the surgical image.

In this specification, " determining position and motion " is understood as a concept including recognizing a state in which there is no change in position and motion. Accordingly, even when the computer judges or recognizes only one of the position and the motion of the photographing apparatus or the object, it can be understood that the other one (i.e., the motion or the position) is also determined to be inferable.

In one embodiment, the computer can determine which portion of the object is being photographed by determining the position and motion of the imaging device 36. [

In one embodiment, the computer may generate a virtual body model that includes the surgical site of the object based on the surgical site image of the object photographed from the medical imaging equipment 10. [ For example, the virtual body model can generate a 3D modeling image including the surgical site of the object.

The computer can determine the positional relationship between the camera and the virtual body model by matching the virtual body model with the actual object and reflecting the position of the camera to the virtual body model information.

The computer can determine the position and movement of the camera and determine information on which part of the user's body part the camera is shooting based on the virtual body model.

The computer can determine the position and movement of one or more objects (e.g., surgical tool 38) based on information obtained from the surgical robot 34. In addition, the computer may determine the positional relationship between the one or more objects and the virtual body model by reflecting the position and motion of one or more objects on the virtual body model information.

In another embodiment, the computer performs image analysis of the surgical image taken by the imaging device 36, thereby obtaining information on the body part included in the surgical image, that is, information of the body part taken by the imaging device 36 The position and the shape can be judged.

Likewise, the computer can use image analysis methods to determine the location and movement of one or more objects.

In one embodiment, the location of the imaging device 36 and one or more objects may be represented by two-dimensional or three-dimensional coordinates.

In addition, the motion of the photographing device 36 and one or more objects can be expressed by the direction, distance, speed, and the like of the motion on two-dimensional or three-dimensional coordinates.

In step S140, based on the determination result in step S130, the computer divides the surgical image acquired from the photographing device 36 into at least one first classification unit group capable of recognizing the operation of the predetermined first classification unit )do. In one embodiment, each group may comprise one or more frames.

In one embodiment, the first classification unit may be a concept that is spatially divided, or may be a concept that is temporally divided. The concept of each division will be described in detail below with reference to the drawings.

In the disclosed embodiment, the surgical operation may be stepwise divided to have a hierarchy of steps. For example, the surgical operation can be divided into a large class, a middle class, a small class, and the like, and can be classified into three types according to the type of operation, the segment according to each operation type, the subsegment and the component But it is not limited thereto.

In one embodiment, the computer divides the surgical image into the smallest classification units and divides the surgical image into smaller, larger, classification units based thereon.

In one embodiment, the computer may be aimed at segmenting the surgical image by itself, or it may be the purpose of segmenting and recognizing the surgical operation based on each of the segmented portions. According to the disclosed embodiment, the computer divides and recognizes the operation operation, may store only the recognition result, or may divide and store the divided surgical image corresponding to each recognized operation operation. In addition, the computer may store the result of recognizing the surgical operation, and may also store information indicating an interval of the image corresponding to each divided surgical operation.

The operation of the predetermined first classification unit includes predefined surgical operations, and includes a preset name for each operation and predetermined code assigned to identify each operation.

In one embodiment, the computer obtains information about the type, location, and movement of the imaging device 36 and one or more objects in step S130. The computer recognizes the operation of the predetermined first classification unit included in the surgical image based on the obtained information.

For example, the operation of the first classification unit may include cropping, catching, moving, and the like.

In one embodiment, the computer may learn the operation of each classification unit using a surgical image labeled with the name or code of each operation. The method used by the computer for learning is not limited, and can be learned by the above-described machine learning method.

In one embodiment, the computer may be learned to output the surgical operation corresponding to each surgical image, with the surgical image itself being an input.

In another embodiment, the computer may learn from the surgical image to input the result of judging the position and motion of the imaging equipment 36 and one or more objects determined according to step S130, and output the corresponding surgical operation, But is not limited thereto.

Hereinafter, a method of spatially dividing and recognizing a surgical image will be described in detail with reference to the drawings.

3 is a flowchart illustrating a method of spatially segmenting a surgical image according to an embodiment.

The method shown in Fig. 3 can be understood as a specific embodiment of step S140 shown in Fig.

In step S210, the computer determines the position and movement of each of the photographing apparatus and one or more objects included in the surgical image. The method described in connection with FIG. 2 may be used as a method by which the computer determines the position and movement of each of the imaging device and one or more objects.

In step S220, the computer determines the operation of each of the photographing apparatus 36 and one or more objects based on the determination result of the position and the motion of each of the photographing apparatus 36 and the one or more objects, The first and second classification unit groups can be divided into at least one second classification unit group.

That is, the second classification unit is a classification unit that defines the operation of each of the photographing apparatus 36 and one or more objects included in the surgical image. For example, the operation of the second classification unit may include moving, rotating, redirecting, gripping, cutting, moving, and clipping operations of the camera.

In one embodiment, the computer can determine the position of one or more objects in the acquired surgical image, and determine the motion of each of the one or more objects by separately analyzing only the portion containing the recognized one or more objects.

For example, the computer recognizes the position of the joint of the surgical arm corresponding to each surgical tool, the coordinate of the end of each surgical tool, and the like, and tracks the motion of each surgical tool by tracking the same.

Thus, the computer can divide only a portion of the recognized region of one or more objects, and process the divided region alone, without separately processing the entire surgical image, thereby individually determining the operation of each of the one or more objects.

Referring to FIG. 4, an example of a surgical image is shown.

Referring to FIG. 4, one or more objects are included in a surgical image 300. For example, each of the one or more objects may include one or more surgical tools 310 and 320 and one or more clips 330.

In one embodiment, the computer divides some of the regions 302 and 304 of the locations where the surgical tools 310 and 320 are recognized from each frame of the surgical image 300, May be processed to determine the operation of each surgical tool 310 and 320.

As the frame is switched, the computer recognizes the movement of the surgical tools 310 and 320 so that the areas 302 and 304 where the surgical tools 310 and 320 are recognized also move accordingly and the area 302 of the moved position And 304, and the operation of the surgical tool 310 can be recognized by processing the divided surgical images.

In one embodiment, the surgical image 300 may include one or more body parts. For example, the surgical image 300 may include a blood vessel 340 and an organ (350, e.g., stomach).

In one embodiment, the computer can recognize the motion of one or more objects based on the positional relationship between the body part and one or more objects included in the surgical image 300 and the motion of the one or more objects.

For example, the computer may recognize that the surgical tool 310 is moved toward the blood vessel 340 and performs an operation of cutting the blood vessel of the clipped region to the clip 330.

The computer can determine the positional relationship between the surgical tool 310 and the blood vessel 340 in the surgical image 300 through image processing. The computer recognizes the operation of cutting the blood vessel 340 by the surgical tool 310 from the surgical image 300 through the image processing and recognizes the result of the cutting of the blood vessel 340, ) Can be judged to have been performed.

In one embodiment, the computer may use the virtual body model to determine the operation of each surgical tool 310 and 320.

For example, a computer can perform a surgical simulation using a virtual body model and a virtual object. The surgical simulation can be performed using the virtual body model in the same manner as the actual surgical situation or the surgical image.

For example, in a surgical simulation, one or more virtual objects move according to the motion of one or more objects included in the actual surgical image, or follow the motion of one or more objects included in the actual surgical image.

The computer recognizes or determines the movement of one or more objects included in the surgical image, receives the recognized or determined information, and moves the virtual object based on the information.

In addition, the computer may determine the operation of one or more objects included in the surgical image 300, receive information that determines the operation of the object (e.g., a name or code corresponding to the operation of the second classification unit) The motion of the virtual object can be determined based on the motion of the virtual object.

The computer can determine the correlation of the virtual body model with motion or motion of one or more virtual objects and determine the operation of one or more objects included in the surgical image 300 from the correlation.

For example, when the computer recognizes that the surgical tool 310 has performed the trimming operation, the computer determines through the virtual body model simulation that the surgical tool 310 has performed the trimming operation at the position where the blood vessel 340 is clipped And it is recognized that the surgical tool 310 has performed an operation of cutting the clipped position of the blood vessel 340.

In step S230, based on the operation of the second classification unit recognized on the basis of the second classification unit group divided in step S220, the computer determines whether the operation of the one or more third classification units Surgical images are divided into groups. The third classification unit group includes one or more second classification unit groups.

That is, the third classification unit group is a classification unit for recognizing the operation of the entire system, including the operation of the photographing apparatus and the operation of one or more objects included in the surgical image.

For example, the third classification unit group may be a classification unit for recognizing what an operation involving the operation of one or more objects included in the same frame means and dividing a surgical image.

For example, when the surgical image is divided and recognized based on the second classification unit, the operation of each surgical tool can be recognized, but it may be difficult to know the cause of the operation or the target of the operation.

The third classification unit determines the operation operation corresponding to the operation image by using the operation of the second classification unit recognized from one or more objects and the position where each operation occurs. For example, the operation tool 320 may perform an operation of holding and lifting a specific part, and the operation tool 310 may perform an operation of cutting the same. It is difficult to know the cause or purpose of judgment individually, but if we use it together, we can judge the cause and purpose of the whole surgical operation.

Thus, the third classification unit group includes one or more second classification unit groups.

In one embodiment, the computer may be trained to recognize the operation of the third classification unit from the operation of one or more second classification units. The learning data may be individually labeled for each classification unit, or may be labeled only for a part of the surgical image or a part of the surgical image based on the surgical image. In this case, the computer can analyze the surgical image and generate necessary learning data.

5 is a flowchart illustrating a method of temporally dividing an operation image according to an embodiment.

In step S410, the computer recognizes the operation of the first classification unit.

In one embodiment, the first classification unit may refer to a unit of a method for temporally segmenting a surgical image. For example, the first classification unit may mean a minimum unit of a method of temporally dividing a surgical image, but is not limited thereto.

In one embodiment, the operation of the first classification unit includes recognized surgical operations based on the motion of one or more objects contained in each frame or one or more frames by a spatial segmentation method as described in connection with FIG. can do.

For example, the first classification unit means a unit of temporal division, and the temporally divided surgical image can be spatially re-divided and analyzed through the second classification unit and the third classification unit.

In another embodiment, the operation of the first classification unit may be described as a set of each of the second classification unit operations of one or more objects included in the surgical image, described in connection with FIG.

In one embodiment, in recognizing the operation of the first classification unit, it may further include determining the location and movement of each of the one or more objects, as well as the cause of each location and movement.

In addition to the location and movement of each of the one or more objects, the computer may use the location of each location and the cause of the motion together to determine the surgical operation.

In step S420, the computer divides the surgical image into one or more fourth classification unit groups capable of recognizing the operation of the predetermined fourth classification unit based on the one or more first classification unit operations recognized in step S410. The fourth classification unit group includes one or more first classification unit groups.

In one embodiment, the fourth classification unit may refer to a unit of a method of temporally segmenting the surgical image. For example, the fourth classification unit may refer to an upper unit successive to the first classification unit in the method of stepwise dividing the surgical image based on the temporal concept, but is not limited thereto.

In one embodiment, the computer may recognize the operation of the fourth classification unit, which includes the operation of one or more first classification units.

In one embodiment, the computer is trained to recognize the operation of the fourth classification unit based on the operation of one or more first classification units. The learning method that can be used by the computer is not limited, and the above-described machine learning method can be utilized.

In one embodiment, the learning data may be labeled for each classification unit, or may be labeled for only a portion of the surgical image or a portion of the surgical image based on the surgical image. In this case, the computer can analyze the surgical image and generate necessary learning data.

For example, it can be seen that when a series of operations including capturing, moving, cutting, etc. of each surgical instrument are performed in a predetermined order, this is a specific operation of the surgery. For example, when detailed operations such as clipping, moving, and cropping are performed continuously, it can be determined that this is an operation of cutting a blood vessel. As another example, when operations such as catching, lifting, cutting, and kicking are performed continuously, it can be determined that this is an action to remove fat.

With this principle, the computer can perform learning to allow the operation of the first classification unit to recognize the operation of the fourth classification unit based on information performed in a predetermined order.

The computer may divide the surgical image into fourth classification units based on the learning result and recognize the operation of the fourth classification unit corresponding to the divided portions.

Referring to FIG. 6, an example in which a surgical image is segmented and recognized step by step is shown.

Referring to FIG. 6, the computer can sequentially recognize the operations of the first classification unit, the fourth classification unit, the fifth classification unit, and the sixth classification unit included in the surgical image according to steps S510, S520, S530, and S540 .

In one embodiment, the first classification unit is a component unit, the fourth classification unit is a subsegment unit, the fifth classification unit is a segmentation unit, the sixth classification unit is a segmentation unit, May be an operation unit.

Referring to FIG. 7, an example of a stepwise recognition of a surgical operation is shown.

Referring to FIG. 7, the surgical operation is divided into a first classification unit 610 and recognized, and the fourth classification unit 620, the fifth classification unit 630, and the sixth classification unit 640 are stepwise And a method of recognizing the divided data is schematically shown.

In one embodiment, each code shown in FIG. 7 may refer to a code that can identify actions included in each classification unit.

By way of example and not limitation, the operation of the first classification unit may include catching, cutting, moving, etc., and the operation of the fourth classification unit may include vascularization, fat removal, Long term resection, long term resection, suture, etc., and the operation of the sixth classification unit may include gastric cancer surgery.

In other words, when gastric cancer surgery is taken as an example, each operation of gastric cancer surgery can largely include laparotomy, gastrectomy, long-term connection and suture, and each operation is more concretely a vasectomy, And the connection of some parts of other organs and the like, and each operation can be more specifically embodied by cutting the blood vessels, removing obstacles such as fat, etc., and this can be accomplished by more simple operations such as movement, Can be further specified.

According to the disclosed embodiment, the hierarchy can be used in reverse, and the operation can be divided into the minimum detail unit, and the computer can be learned to recognize the upper operation step by step using the divided result.

It is relatively difficult to accurately recognize the surgical operation corresponding to the upper stage by image processing the surgical image without such a stepwise approach. The surgical site is different for each patient, each disease differs in shape, and the operation patterns are different depending on the type of operation.

According to the disclosed embodiment, it is possible to start from a recognition of a detailed operation (for example, cutting, catching, etc.) less affected by the physical condition of the patient or the type of operation, It is possible to provide a learning model capable of recognizing the upper operation operation which is meant by a larger unit of operation and further recognizing the type of surgery.

Using the learning model according to the disclosed embodiment, it is possible to provide a surgical motion recognition model that can be applied to the patient regardless of the physical condition of the patient or the type of surgery, and, if necessary, Or may provide a surgical motion recognition model that is tailored to the type of condition or surgery.

In one embodiment, the computer can recognize an event that occurs in a surgical image. For example, the event includes a surgical error situation, such as bleeding. When an event occurs, the computer can recognize this through image processing of the surgical image.

When the event is recognized, the computer may divide the surgical image into one or more event groups including recognized events. The divided event groups may be managed separately, included in a classification unit according to the disclosed embodiment, or may be utilized as an independent classification unit for analysis of a surgical operation.

In one embodiment, the computer can determine the cause of the event based on the recognized event and the surgical operation before and after the event was recognized.

For example, when an event occurs, the computer may generate learning data for analyzing the cause of the event by storing the operations of the predetermined classification unit before and after the occurrence of the event together with information on the event.

The computer can perform learning using the generated learning data, and learn the correlation between the operation and the events of each classification unit.

Based on the learning result, the computer can determine the cause of the event occurrence and provide feedback to the user.

In one embodiment, the computer may perform learning for optimization of surgical operations based on operation of a given classification unit.

For example, the computer can learn an optimized sequence and method for performing the operation of each classification unit according to the physical condition of the patient and the type of surgery.

In one embodiment, the computer may perform learning for optimization of the surgical operation based on the operation of the first classification unit. For example, the computer may obtain one or more reference surgery information. The computer can perform learning based on the order of the operation operations included in the one or more reference operation information and determine the order of the optimized operation operations for each operation according to the learning results.

Since the operation of the first classification unit is a minimum unit operation commonly applied in any operation, when learning is performed based on the first classification unit, the order of the optimized operation operations regardless of the type of operation and the body condition of the patient Can be obtained. Likewise, it is also possible to obtain an optimized learning model according to the type of surgery and the patient's physical condition through fine adjustment to the learned model.

8 is a block diagram of an apparatus 700 according to an embodiment.

The processor 710 may include one or more cores (not shown) and a connection path (e. G., Bus, etc.) to transmit and receive signals to and / or from a graphics processing unit .

The processor 710 according to one embodiment performs the surgical image segmentation method described with respect to Figures 1-7 by executing one or more instructions stored in the memory 720. [

For example, the processor 710 may be configured to acquire a surgical image by executing one or more instructions stored in memory, to recognize one or more objects included in each of the one or more frames of the surgical image, And a controller for determining the position and motion of each of the recognized one or more objects and dividing the surgical image into at least one first classification unit group capable of recognizing the operation of the predetermined first classification unit based on the determination result .

The processor 710 may include a random access memory (RAM) (not shown) and a read-only memory (ROM) for temporarily and / or permanently storing signals (or data) , Not shown). In addition, the processor 710 may be implemented as a system-on-chip (SoC) including at least one of a graphics processing unit, a RAM, and a ROM.

The memory 720 may store programs (one or more instructions) for processing and control of the processor 710. Programs stored in the memory 720 can be divided into a plurality of modules according to functions.

As described above, the surgical image segmentation method according to an embodiment of the present invention may be implemented as a program (or an application) to be executed in combination with a hardware computer and stored in a medium.

The above-described program may be stored in a computer-readable medium such as C, C ++, JAVA, machine language, or the like that can be read by the processor (CPU) of the computer through the device interface of the computer, And may include a code encoded in a computer language of the computer. Such code may include a functional code related to a function or the like that defines necessary functions for executing the above methods, and includes a control code related to an execution procedure necessary for the processor of the computer to execute the functions in a predetermined procedure can do. Further, such code may further include memory reference related code as to whether the additional information or media needed to cause the processor of the computer to execute the functions should be referred to at any location (address) of the internal or external memory of the computer have. Also, when the processor of the computer needs to communicate with any other computer or server that is remote to execute the functions, the code may be communicated to any other computer or server remotely using the communication module of the computer A communication-related code for determining whether to communicate, what information or media should be transmitted or received during communication, and the like.

The medium to be stored is not a medium for storing data for a short time such as a register, a cache, a memory, etc., but means a medium that semi-permanently stores data and is capable of being read by a device. Specifically, examples of the medium to be stored include ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage, and the like, but are not limited thereto. That is, the program may be stored in various recording media on various servers to which the computer can access, or on various recording media on the user's computer. In addition, the medium may be distributed to a network-connected computer system so that computer-readable codes may be stored in a distributed manner.

The steps of a method or algorithm described in connection with the embodiments of the present invention may be embodied directly in hardware, in software modules executed in hardware, or in a combination of both. The software module may be a random access memory (RAM), a read only memory (ROM), an erasable programmable ROM (EPROM), an electrically erasable programmable ROM (EEPROM), a flash memory, a hard disk, a removable disk, a CD- May reside in any form of computer readable recording medium known in the art to which the invention pertains.

While the present invention has been described in connection with what is presently considered to be practical exemplary embodiments, it is to be understood that the invention is not limited to the disclosed embodiments, but, on the contrary, You will understand. Therefore, it should be understood that the above-described embodiments are illustrative in all aspects and not restrictive.

10: Medical imaging equipment
20: Server
30:
32: Display
34: Surgical robot
36:
38: Surgical tools

Claims (11)

The computer obtains the surgical image, wherein the surgical image is generated by photographing the inside of the body during laparoscopic surgery or robot surgery, or acquiring a surgical image, which is a body internal image generated by performing a virtual operation on the virtual body model step;
Wherein the computer recognizes one or more objects included in each of the one or more frames of the surgical image, the object being any one of a body organ or tissue, an object introduced from the outside, ;
The computer calculates a spatial classification unit by determining a position or motion of a part of the surgical tools included in the recognized one or more objects, wherein the spatial classification unit is a unit for defining a motion of some surgical instruments among the whole surgical tools, Classification unit calculation step;
The computer segmenting one or more image frames into at least one first classification unit group capable of recognizing an operation of a predetermined first classification unit based on one or more of the spatial classification units, A first classification unit group dividing step, which is a group of one or more image frames including at least one of the spatial classification units defining a minimum operation unit to be performed at the time of surgery;
The computer recognizing the operation of the first classification unit based on at least one of the spatial classification units;
The computer is classified into at least one fourth classification unit group capable of recognizing the operation of the predetermined fourth classification unit based on the operation of the at least one first classification unit recognized for each of the at least one first classification unit group , And the fourth classification unit group includes at least one of the first classification unit groups continuously performed; And
And recognizing a surgical stage of at least one of the fourth classification units corresponding to each of the at least one fourth classification unit group based on the operation of the at least one first classification unit,
The spatial classification unit calculation step may include:
The computer generating a plurality of divided regions by spatially dividing an image frame included in the first classification unit based on a position of a specific surgical tool;
Dividing the spatial classification unit based on a movement of a surgical tool performed in each of the plurality of divided regions; And
And the computer recognizing the operation of the surgical tool included in each of the divided spatial classification units. delete The method according to claim 1,
The surgical tool motion recognition step may include:
The computer recognizing coordinates of a joint of a surgical arm corresponding to each surgical tool in each image frame and coordinates of an end position of each surgical tool; And
And recognizing movement of each surgical tool by tracking the position coordinates and the end position coordinates of the joint in successive image frames. The method according to claim 1,
Wherein the step of recognizing the operation of the first classification unit comprises:
Wherein the operation of the first classification unit is recognized based on a positional relationship between the body part included in the surgical image and the at least one object, and a motion of the at least one object. 5. The method of claim 4,
Wherein the step of recognizing the operation of the first classification unit comprises:
Obtaining modeling information corresponding to the body part;
Performing simulation of a surgical operation including the modeling information and the virtual one or more objects, wherein the one or more virtual objects move in accordance with movement of the one or more objects in the simulation; And
Recognizing the operation of the first classification unit using the result of the simulation; Wherein the method comprises the steps of: The method according to claim 1,
Wherein the first classification unit includes information on one or more predetermined operations,
Wherein the predetermined one or more operations are each assigned a code capable of identifying each of the one or more predetermined operations. The method according to claim 1,
Before the fourth classification unit group classification step,
Recognizing an event generated in at least one image frame included in the surgical image;
Dividing the surgical image into one or more event groups including the recognized events; Further comprising the steps of: The method according to claim 1,
After the fourth classification unit group classification step,
Further comprising classifying a fifth classification unit group corresponding to an upper layer based on at least one of the fourth classification units in which the computer is continuous. 9. The method of claim 8,
After the fifth classification unit group classification step,
Generating a hierarchical structure for the entire operation based on at least one of the first classification unit, the at least one fourth classification unit and the at least one fifth classification unit. A memory for storing one or more instructions; And
And a processor executing the one or more instructions stored in the memory,
The processor executing the one or more instructions,
Obtaining a surgical image;
Recognize one or more objects included in each of one or more frames of the surgical image,
The computer calculates a spatial classification unit by determining a position or motion of a part of the surgical tools included in the recognized one or more objects,
Segmenting one or more image frames into at least one first classification unit group capable of recognizing an operation of a predetermined first classification unit based on one or more of the spatial classification units,
Recognizing an operation of the first classification unit based on at least one of the spatial classification units,
Classifying the group into at least one fourth classification unit group capable of recognizing the operation of the predetermined fourth classification unit based on the operation of the at least one first classification unit recognized for each of the at least one first classification unit group,
Recognizing a surgical stage of at least one of the fourth classification units corresponding to each of the at least one fourth classification unit group based on the operation of the at least one first classification unit,
The surgical image is an image of the inside of the body generated by photographing the inside of the body during laparoscopic surgery or robotic surgery, or by performing virtual surgery on the virtual body model,
The object is any one of a body organ or tissue, an object introduced from the outside, or an object newly generated in the surgical procedure,
The spatial classification unit is a unit that defines the motion of some of the surgical instruments among all the surgical instruments,
Wherein the first classification unit group is a group of one or more image frames including at least one spatial classification unit that defines a minimum operation unit to be performed at the time of surgery,
Wherein the fourth classification unit group includes at least one first classification unit group continuously performed,
In calculating the spatial classification unit,
Generating a plurality of divided regions by spatially dividing an image frame included in the first classification unit based on a position of a specific surgical tool,
Dividing the spatial classification unit based on movement of a surgical tool performed in each of the plurality of divided regions,
And recognizes the operation of the surgical tool contained in each of the divided spatial classification units. A computer program stored in a computer readable recording medium in combination with a computer which is hardware and which is capable of performing the method of claim 1. Description:

Images