WO2021206517A1

WO2021206517A1 - Intraoperative vascular navigation method and system

Info

Publication number: WO2021206517A1
Application number: PCT/KR2021/004532
Authority: WO
Inventors: 김하진; 허성환; 박성현
Original assignee: (주)휴톰
Priority date: 2020-04-10
Filing date: 2021-04-09
Publication date: 2021-10-14
Also published as: KR102457585B1; KR20210126243A

Abstract

An intraoperative vascular navigation method and system are disclosed. A method by which a system performs intraoperative vascular navigation, according to one embodiment, comprises the steps of: constructing, from two-dimensional medical image data of an object, a three-dimensional vascular model by using a vascular learning model; recognizing, from the photographed surgical image, the surgical stage of the object by using a surgical stage learning model; extracting, from the constructed three-dimensional vascular model, a main blood vessel corresponding to the recognized surgical stage, and matching same with a navigation image; and providing the navigation image to a user, wherein the navigation image includes the blood vessels branched from the main blood vessel, and the branched blood vessels can be added or removed on the basis of a user input.

Description

Intraoperative vascular navigation method and system

The present invention relates to a method and system for vascular navigation during surgery. More specifically, the present invention recognizes individual stages of an image during surgery through a learning model, and superimposes a patient's blood vessel image extracted based on medical image data (eg, CT image) on the recognized staged surgical image. It relates to a method and system for providing a modeled screen.

In general, when doctors establish a patient's surgical plan prior to surgery, they refer to 2D medical images such as CT (Computed Tomographic) or MRI (Magnetic Resonance Imaging) images of the patient for information about the surgical site to plan the operation. stand up In this case, it is difficult to match the location of the lesion existing inside the patient's organs or the relationship with the blood vessels according to the location of the lesion with the two-dimensional medical image, and there is no means to use it as organ imaging information before surgery. There are many limitations in determining the location of the lesion and the distribution of surrounding blood vessels.

In order to solve this problem, a technology for providing CT or MRI images as a 3D image is being studied.

Even if a surgical plan is planned using medical images implemented as 3D augmented reality images, doctors using the surgical navigation system may cause unwanted bleeding or damage organs if they cannot accurately recognize the location of different blood vessels for each person. there is a risk of making

The problem to be solved by the present invention is to recognize an individual phase of an image during surgery through a surgical stage learning model, and the patient's blood vessels extracted based on medical image data (eg, CT image) in the recognized staged surgical image. It is to provide a 3D modeled screen overlaid with images.

In addition, it is possible to construct a patient-specific three-dimensional anatomical model using a blood vessel learning model from the patient's medical image data, and to allow the operator to add or exclude branching blood vessels according to selection in the surgical image of a specific stage.

The problems to be solved by the present invention are not limited to the problems mentioned above, and other problems not mentioned will be clearly understood by those skilled in the art from the following description.

In order to solve the above problems, the intraoperative blood vessel navigation method performed by the system according to an embodiment of the present invention includes the steps of: constructing a three-dimensional blood vessel model using a blood vessel learning model from two-dimensional medical image data of an object; , Recognizing the surgical stage of the object using the surgical stage learning model in the photographed surgical image, extracting major blood vessels corresponding to the recognized surgical stage from the constructed 3D vascular model and matching the navigation image and providing the navigation image to a user, wherein the navigation image includes a branched blood vessel from the main blood vessel, and the branched blood vessel can be added or removed based on a user input.

In this case, the navigation image to which the main blood vessels are matched may include an image in which an object is 3D modeled using the medical image data or the photographed surgical image.

In addition, the surgical step learning model may be machine-learned by defining at least one or more steps necessary for the operation of the object as a label using surgical data, inputting a learning image for each defined label, and the surgical data is It may include tracking data for the movement of the robot arm or data obtained from a surgical image frame.

In addition, the blood vessel learning model may assign blood vessel types by sequentially applying blood vessel bifurcation points after modeling veins and arteries based on medical image data.

In addition, the step of matching the major blood vessels to the navigation image includes collecting the location information of the surgical image screen, and synchronizing the three-dimensionally modeled major blood vessels and the object to the surgical image based on the surgical step and location information. (SYNC) can be done.

In addition, the step of providing the navigation image may highlight and display the extracted major blood vessels in the navigation image. In this case, the step of highlighting and displaying the main blood vessel includes displaying a list including the extracted main blood vessel item on the navigation image, and when a specific item is selected from the list, a scene of the navigation image is displayed on the selected specific item It is possible to move to a scene of major blood vessels corresponding to , and highlight and display major blood vessels corresponding to the specific item in the moved scene. In addition, the step of highlighting and displaying the main blood vessels includes a first highlight display method for displaying the main blood vessels by zooming in at a preset magnification, a second highlight display method for displaying the main blood vessels with a preset color, and the main blood vessels. A third highlight display method for displaying the border of blood vessels by blinking, a fourth highlight display method for displaying text indicating the main blood vessels on the main blood vessels, and a fifth highlight display method for displaying the main blood vessels in a plurality of angles At least one or two or more of them may be highlighted in a combined form.

In addition, the present invention for solving the above problems is a computer program stored in a computer-readable recording medium that is combined with a computer, which is hardware, so as to perform a blood vessel navigation method during surgery, wherein the computer program is a two-dimensional (2D) object. A process of constructing a three-dimensional blood vessel model using a blood vessel learning model from medical image data, a process of recognizing a surgical stage of an object using a surgical stage learning model in a photographed surgical image, and corresponding to the recognized surgical stage performing a process of extracting major blood vessels from the constructed three-dimensional blood vessel model and matching them to a navigation image, and providing the navigation image to a user, wherein the navigation image includes blood vessels branching from the major blood vessels, The branched blood vessel may be added or removed based on a user input.

An intraoperative blood vessel navigation system according to an embodiment of the present invention for solving the above-described problems includes a medical image photographing device for photographing a surgical image, a display unit for providing a surgical navigation image to a user, and one or more processors and a control unit including at least one memory in which instructions for causing the at least one processor to perform an operation when executed by the one or more processors are stored, wherein the operation performed by the control unit is performed in two-dimensional medical image data of an object An operation for constructing a three-dimensional blood vessel model using the blood vessel learning model, an operation for recognizing the surgical stage of an object using the surgical stage learning model in the photographed surgical image, and a major blood vessel corresponding to the recognized surgical stage an operation of extracting from the constructed 3D blood vessel model and matching to a navigation image, and an operation of providing the navigation image to a user, wherein the navigation image includes a branched blood vessel from the main blood vessel, and the branched blood vessel may be added or removed based on user input.

According to an embodiment of the present invention, various information can be provided to the user by matching the three-dimensionally modeled blood vessel image to the surgical step-by-step image recognized during surgery.

In addition, by constructing a patient-specific three-dimensional anatomical model through machine learning, and allowing branching blood vessels to be added or excluded according to the user's selection in the surgical image of a specific stage, the operation process is facilitated, and the user can have different characteristics for each person. By accurately recognizing the location of blood vessels, it is possible to reduce the likelihood of causing unwanted bleeding or damaging organs.

Effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the following description.

1 is a view showing a robotic surgery system according to an embodiment of the present invention.

2 is a flowchart illustrating a method for vascular navigation during surgery according to an embodiment of the present invention.

3A to 3C are diagrams for explaining a blood vessel division method according to an embodiment of the present invention.

4 is a view for explaining an example of modeling a branch of a blood vessel according to an embodiment of the present invention.

5 is a flowchart for explaining a surgical step learning model according to an embodiment of the present invention.

6 is a diagram for explaining an example of matching blood vessel navigation images according to an embodiment of the present invention.

7 is a view for explaining an example of modeling blood vessel flow according to an embodiment of the present invention.

8A and 8B are diagrams for explaining an example of a blood vessel navigation image according to an embodiment of the present invention.

Advantages and features of the present invention and methods of achieving them will become apparent with reference to the embodiments described below in detail in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in various different forms, and only these embodiments allow the disclosure of the present invention to be complete, and those of ordinary skill in the art to which the present invention pertains. It is provided to fully understand the scope of the present invention to those skilled in the art, and the present invention is only defined by the scope of the claims.

The terminology used herein is for the purpose of describing the embodiments and is not intended to limit the present invention. As used herein, the singular also includes the plural unless specifically stated otherwise in the phrase. As used herein, “comprises” and/or “comprising” does not exclude the presence or addition of one or more other components in addition to the stated components. Like reference numerals refer to like elements throughout, and "and/or" includes each and every combination of one or more of the recited elements. Although "first", "second", etc. are used to describe various elements, these elements are not limited by these terms, of course. These terms are only used to distinguish one component from another. Accordingly, it goes without saying that the first component mentioned below may be the second component within the spirit of the present invention.

Unless otherwise defined, all terms (including technical and scientific terms) used herein will have the meaning commonly understood by those of ordinary skill in the art to which this invention belongs. In addition, terms defined in a commonly used dictionary are not to be interpreted ideally or excessively unless specifically defined explicitly.

In this specification, "image" may mean multi-dimensional data composed of discrete image elements (eg, pixels in a 2D image and voxels in a 3D image). For example, the image may include a medical image of the object obtained by the CT imaging apparatus.

As used herein, an “object” may be a human or an animal, or a part or all of a human or animal. For example, the object may include at least one of organs such as liver, heart, uterus, brain, breast, abdomen, and blood vessels.

As used herein, a “user” may be a medical professional, such as a doctor, a nurse, a clinical pathologist, or a medical imaging specialist, and may be a technician repairing a medical device, but is not limited thereto.

As used herein, "medical image data" is a medical image captured by a medical imaging device, and includes all medical images that can be implemented as a three-dimensional model of the body of an object. "Medical image data" may include a computed tomography (CT) image, a magnetic resonance imaging (MRI), a positron emission tomography (PET) image, and the like.

As used herein, the term "virtual body model" refers to a model generated to match the actual patient's body based on medical image data. The "virtual body model" may be generated by modeling medical image data in 3D as it is, or may be corrected after modeling to be the same as during actual surgery.

As used herein, “virtual surgical data” refers to data including rehearsal or simulation actions performed on a virtual body model. “Virtual surgical data” may be image data on which rehearsal or simulation is performed on a virtual body model in a virtual space, or data recorded on a surgical operation performed on the virtual body model. In addition, "virtual surgery data" may include learning data for learning the surgical learning model.

As used herein, “actual surgical data” refers to data obtained by actual medical staff performing surgery. "Surgery data" may be image data of a surgical site taken in an actual surgical procedure, or may be data recorded on a surgical operation performed in an actual surgical procedure.

In the present specification, a surgical phase refers to a basic phase that is sequentially performed in the entire operation of a specific type of operation.

As used herein, the term “computer” includes various devices capable of providing a result to a user by performing arithmetic processing. For example, computers include desktop PCs, notebooks (Note Books) as well as smart phones, tablet PCs, cellular phones, PCS phones (Personal Communication Service phones), synchronous/asynchronous A mobile terminal of International Mobile Telecommunication-2000 (IMT-2000), a Palm Personal Computer (PC), a Personal Digital Assistant (PDA), and the like may also be applicable. Also, when a head mounted display (HMD) device includes a computing function, the HMD device may be a computer. In addition, the computer may be a server or a navigation system that receives a request from a client and performs information processing.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

Referring to FIG. 1 , a schematic diagram of a system capable of performing robotic surgery according to an embodiment is shown.

According to FIG. 1 , the robotic surgery system includes a medical image capturing device 10 , a navigation system 20 , and a controller 30 provided in an operating room, an image capturing unit 36 , a display 32 and a surgical robot 34 . include According to an embodiment, the medical imaging equipment 10 may be omitted from the robotic surgery system according to an embodiment.

In one embodiment, the robotic surgery may be performed by the user controlling the surgical robot 34 using the control unit 30, and may be automatically performed by the control unit 30 without the user's control (manipulation). .

The navigation system 20 is a computing device including at least one processor, a memory and a communication unit. In one embodiment, the navigation system 20 may include a data processing server outside the operating room and a graphic processing terminal device in the operating room, or may include only the graphic processing terminal device.

The control unit 30 includes a computing device including at least one processor, a memory, and a communication unit. In one embodiment, the controller 30 includes hardware and software interfaces for controlling the surgical robot 34 . Also, the control unit 30 may be divided into a user console device and a control device.

The image capturing unit 36 includes at least one image sensor. That is, the image capturing unit 36 includes at least one camera device, and is used to photograph the surgical site. In one embodiment, the imaging unit 36 is used in combination with the surgical robot (34). For example, the imaging unit 36 may include at least one camera coupled to the surgical arm of the surgical robot 34 .

In an embodiment, the image captured by the image capturing unit 36 is displayed on the display 32 .

The controller 30 receives information necessary for surgery from the navigation system 20 or generates information necessary for surgery and provides it to the user. For example, the controller 30 displays generated or received information necessary for surgery on the display 32 .

For example, the user performs robotic surgery by controlling the movement of the surgical robot 34 by manipulating the controller 30 while viewing the display 32 .

The navigation system 20 generates information necessary for robotic surgery using medical image data of an object (patient) photographed in advance from the medical imaging device 10 , and provides the generated information to the controller 30 .

The controller 30 may provide information received from the navigation system 20 to the user by displaying it on the display 32 , and control the operation of the surgical robot 34 using the received information.

In one embodiment, the means that can be used in the medical imaging equipment 10 is not limited, for example, CT, X-Ray, PET, MRI, etc., various other medical image acquisition means may be used.

Hereinafter, each step is described as being performed by a “computer” for convenience of explanation, but the subject performing each step is not limited to a specific device, and all or part of it is performed by the server 20 or the control unit 30 can be performed.

In an embodiment, the surgical image captured by the medical imaging device 10 may be divided according to various criteria. For example, the surgical image may be divided based on the type of object included in the image. The segmentation method based on the type of object requires the computer to recognize each object.

Objects recognized in the surgical image largely include a human body, an object introduced from the outside, and an object created by itself. The human body includes body parts that are imaged by medical imaging (eg, CT) prior to surgery and body parts that are not imaged.

For example, body parts photographed by medical imaging include organs, blood vessels, bones, tendons, and the like, and these body parts may be recognized based on a 3D modeling image generated based on the medical image.

Specifically, the position, size, shape, etc. of each body part are recognized in advance by a 3D analysis method based on a medical image. The computer defines an algorithm that can determine the position of the body part corresponding to the surgical image in real time, and based on this, information on the position, size, shape, etc. of each body part included in the surgical image without performing separate image recognition can be obtained.

In addition, body parts that are not photographed by medical imaging include omentum, etc., which are not photographed by medical imaging, so it is necessary to recognize them in real time during surgery. For example, the computer may determine the position and size of the omentum through the image recognition method, and if there is a blood vessel inside the omentum, the location of the blood vessel may also be predicted.

The objects introduced from the outside include, for example, surgical tools, gauze, clips, and the like. Since it has preset morphological characteristics, the computer can recognize it in real time through image analysis during surgery.

The object generated inside includes, for example, bleeding occurring in a body part. This can be recognized in real time by the computer through image analysis during surgery.

The movement of organs or omentum included in the body part, and the cause of the internal creation of an object, are all caused by the movement of an object introduced from the outside.

Therefore, in addition to recognizing each object, the surgical image may be divided into several surgical phases based on the movement of each object. In an embodiment, the surgical image may be divided based on a movement of an externally introduced object, that is, an action.

The computer determines the type of each object recognized in the surgical image, and the movement of each object, that actions can be recognized.

The computer can recognize the type of each action and further recognize the cause of each action. The computer can segment the surgical image based on the recognized action, and can recognize from each detailed surgical operation to the type of the entire operation through stepwise segmentation.

For example, a computer extracts feature information from a surgical image taken based on a surgical stage learning model that performs machine learning of a convolutional neural network method, and generates images for each surgical stage based on the feature information. It can be divided or recognized at which stage of surgery it is currently located.

Furthermore, the computer may determine a predefined type of operation corresponding to the operation image from the determination of the action. When determining the type of surgery, information on the entire surgical process may be acquired. When a plurality of surgical processes exist for the same type of surgery, one surgical process may be selected according to a selection of a doctor or based on actions recognized up to a specific time point.

The computer may recognize and predict the surgical stage based on the acquired surgical process. For example, when a specific step is recognized in a series of surgical processes, the subsequent steps may be predicted or candidates for possible steps may be selected.

Therefore, it is possible to significantly reduce the error rate of surgical image recognition caused by omentum. In addition, when the surgical image greatly deviates from the predictable surgical stage by more than a predetermined error range, it may be recognized that a surgical error situation has occurred. For example, if a surgical stage switching that deviates from a predetermined surgical process frequently occurs, it may be recognized that a surgical error situation has occurred.

In addition, based on the image recognition of the surgical stage, the computer extracts navigation information about the main vessels corresponding to each surgical stage and the vessels branched to the main vessels according to the blood flow, thereby assisting the user for effective surgery. can do.

In addition, the computer may determine whether bleeding has occurred due to a surgical error based on the image recognition of the surgical stage.

Specifically, the computer can determine the location, time, and magnitude of each bleeding. It can also determine whether surgery should be stopped due to bleeding. Accordingly, according to an embodiment, the computer may be used to provide data on an error situation and bleeding situation as a surgical result report, to exclude unnecessary operations or mistakes in the surgical process, and to streamline the surgical process.

In order to provide blood vessel navigation during surgery according to an embodiment, the calculation performed on the computer is the calculation of constructing a 3D blood vessel model using the blood vessel learning model in the 2D medical image data of the object, in the photographed surgical image, Calculation of recognizing the surgical stage of an object using the surgical stage learning model, extracting major blood vessels corresponding to the recognized surgical stage from the constructed 3D vascular model and matching the navigation image, and providing the navigation image to the user It can include operations provided.

In addition, in providing the navigation image, the computer may further include an operation for adding or removing a blood vessel branching from a main blood vessel based on a user input.

In addition, the navigation image to which the major blood vessels are matched may be a three-dimensional object image modeled from the photographed surgical image or two-dimensional medical image data of the object. In addition, the computer collects the position information of the captured surgical image screen, and performs an operation of synchronizing (SYNC) the three-dimensional modeled main blood vessels and the object to the surgical image based on the recognized surgical stage and position information. can

Hereinafter, a blood vessel navigation method during surgery will be described in more detail with reference to the drawings.

Each of the steps shown in FIG. 2 is time-series performed by the navigation system 20 or the controller 30 shown in FIG. 1 . Hereinafter, each step is described as being performed by a “computer” for convenience of explanation, but the subject performing each step is not limited to a specific device, and all or part of the navigation system 20 or the control unit 30 can be performed in

In step S200, the computer according to an embodiment builds a 3D blood vessel model by using the blood vessel learning model from the two-dimensional medical image data of the object.

Here, the blood vessel learning model may be to find and model major blood vessels, branched blood vessels and blood vessel flows based on main blood vessels, based on user-specific information and two-dimensional medical image data obtained from clinical data.

The blood vessel learning model will be described later with reference to FIGS. 3A to 3C and 4 .

Next, in step S210, the computer according to an embodiment recognizes the surgical stage of the object by using the surgical stage learning model in the photographed surgical image.

Here, the surgical step learning model may be machine-learned by defining at least one or more steps necessary for the operation of the object as a label using virtual surgery data, and inputting a learning image for each defined label.

In addition, the step of recognizing the operation stage of the object using the operation stage learning model may be recognizing the operation stage using the learned operation stage learning model based on the actual operation data.

In addition, the actual surgical data may include tracking data for the movement of the robot arm or data obtained from a surgical image frame.

Hereinafter, a surgical step learning model will be described with reference to FIG. 5 .

In step S220, the computer according to an embodiment extracts major blood vessels corresponding to the recognized surgical steps from the constructed 3D blood vessel model and matches them with the navigation image.

The navigation image to which the major blood vessels are matched may be a three-dimensionally modeled object image using the captured surgical image or the medical image data.

Hereinafter, an example of matching a blood vessel navigation image according to an embodiment will be described later with reference to FIG. 6 .

In step S230, the computer according to an embodiment provides the matched navigation image to the user. Here, in the navigation image, a blood vessel branched from a main blood vessel may be added or removed according to a blood vessel flow based on a user input.

Hereinafter, an example of a method of providing and manipulating a blood vessel navigation image to a user will be described later with reference to FIGS. 7 and 8A and 8B .

Referring to FIG. 3A , a computer according to an embodiment of the present invention generates 3D image data 301 of an object (organs and blood vessels) through a process of converting medical image data (CT image) into a 3D object. do. More specifically, after veins and arteries are modeled based on medical image data, vascular branch points are sequentially assigned blood vessel types.

In this case, labeling for separating major blood vessels may be performed for each operation step, and machine learning may be performed on each labeled blood vessel.

Taking the operation for stomach cancer as an example, the names of organs and blood vessels appearing in the surgical image are listed up, and the listed names are tagged on the image so that each learning model can perform machine learning. Then, each organ and blood vessel can be divided using the machine-learned blood vessel learning model. Accordingly, as shown in FIG. 3B , the user can check the list 302 of blood vessels labeled with the names of organs and blood vessels, and further, as shown in FIG. 3C , the user can check the image 303 tagged with the names can

For example, FIG. 4 is a diagram for explaining an example of modeling a branch of a blood vessel according to an embodiment of the present invention.

Referring to FIG. 4 , the computer according to an embodiment of the present invention may label a blood vessel name by separating a blood vessel from a 3D image through a blood vessel learning model and then generating a point for each branch point 401 of each blood vessel. Therefore, according to the intraoperative blood vessel navigation method according to an embodiment, since the blood vessel learning model is machine-learned based on the branch point, when the blood vessel navigation image is provided to the user, the blood vessel may be displayed to be added or removed based on the branch point. .

In addition, the computer according to an embodiment of the present invention can more accurately find and model major blood vessels, branched blood vessels and blood vessel flows based on main blood vessels, based on user-specific information and two-dimensional medical image data obtained from clinical data. may be

Therefore, the location of different blood vessels for each person can be accurately recognized from the medical image data through such machine learning, and the location of blood vessels can be accurately guided to the user during future surgery.

Referring to FIG. 5 , in step S500, the computer selects the type of surgery. The type of surgery may be automatically selected by recognizing an object or an object in the input image, or may be directly input by the user.

Next, in step S510, the computer defines the operation as a label for each stage using the actual operation data.

Each surgical phase may be divided automatically by recognizing images of camera movement, equipment movement, and organs, and the user may classify an image suitable for a standardized surgical process as a learning image and learn it in advance.

For example, in the case of stomach cancer surgery, preparation steps, exposing organs, cutting blood vessels, cutting the lower left organ of the stomach, cutting the lower right organs of the stomach, or cutting the connection between the duodenum and the stomach A surgical stage can be defined as about 21 stages, including stages.

Next, in step S520, the computer selects a learning image corresponding to the defined label. The learning image may be selected by automatically dividing the inputted surgical image by a computer, or the user may select and input the learning image in advance based on actual surgical data.

Next, in step S530, the computer may perform machine learning using a convolutional neural network based on the selected learning images. For example, when the frame of each surgical image defined in this way is learned using an action recognition model such as SlowFast Network, a highly accurate surgical stage learning model can be trained.

In this way, the trained surgical stage learning model is used to more accurately and automatically recognize the surgical stage during surgery.

According to an embodiment of the present invention, a computer may collect position information of a surgical image screen, and synchronize (SYNC) the three-dimensional modeled major blood vessels and an object to the surgical image based on the surgical stage and position information. .

Here, the navigation image to which the blood vessels are matched may be an image of an object (organs and blood vessels) that is 3D modeled using the photographed surgical image or the medical image data.

For example, as in the image shown in the screen 602 of FIG. 6 , the computer extracts the major blood vessels and the object of the surgical stage recognized in the gastric cancer surgery image 601 from the 3D blood vessel model, and then extracts the extracted major blood vessels and 3 The dimensionally modeled image of the object may be matched and provided to the user.

Referring to FIG. 7 , in the case of a three-dimensional blood vessel model according to an embodiment of the present invention, it may be modeled to enable navigation of surrounding blood vessels according to the flow of blood vessels based on main blood vessels. In the case of the three-dimensional blood vessel model according to an embodiment, since each blood vessel branching around the main blood vessel is labeled and learned, the flow and depth are divided for each blood vessel recognized from the medical image data based on the branch point. Because it can be classified.

Therefore, as shown in FIG. 7 , the computer defines each step for displaying blood vessels by depth so that the guidance of the surrounding blood vessels can be performed centering on the selected main blood vessel 701 , and defines the blood vessel flow for each depth. A decision 702 is made. In addition, according to a user's selection, peripheral blood vessels may be added or removed according to the flow for each depth to be displayed.

8A and 8B show an example of a blood vessel navigation image according to an embodiment of the present invention.

As shown in FIG. 8A , the computer may display only the major blood vessels 801 synchronized with the surgical image on the navigation image screen.

As shown in FIG. 8A, through a user interface that removes (802) or transparent (804) organs or peripheral blood vessels from the initial registration image 802 according to the user's selection, the vascular navigation image required by the user in the surgical stage can provide In addition, it is also possible to display the branched blood vessels according to the flow of blood vessels from the main blood vessels step by step.

In addition, the computer identifies a plurality of points of interest (eg, major blood vessels) through analysis of surgical requirements included in information required for surgery, and reconstructs the identified plurality of major parts. After this, a list of the plurality of main parts is displayed on the blood vessel navigation image, and when a specific main part is selected from the list, the scene (or viewpoint) of the vessel navigation image is selected as the scene (or viewpoint) of the selected specific main part. It can be moved, and the specific main part can be highlighted and displayed within the scene of the moved specific main part.

In this case, the highlight display is a first highlight display method in which the specific main part is displayed prominently within the scene of the specific main part, and the specific main part is displayed by zooming in at a preset magnification. A second highlighting method for displaying a main part with a preset color, a third highlighting method for displaying the edge of the specific main part by blinking, and a third highlighting method for displaying text indicating the specific main part on the specific main part At least one or a combination of two or more of the fourth highlighting method and the fifth highlighting method of displaying the specific main part in a plurality of angles may be highlighted. That is, the computer can highlight and mark the parts of the virtual body model that need to be looked at carefully in the surgical operation.

In this case, the part of the virtual body model that needs to be carefully observed in the surgical operation may be a bifurcation point of a specific blood vessel and a starting point of a specific blood vessel. As an example, the starting point position of a specific blood vessel entering a specific organ may be displayed in the above-described highlighting method.

The computer may define in advance the main parts of the virtual body model that need to be carefully viewed in the surgical operation, and show a list of items of the defined main parts to the user.

Through this, the user can identify the part of the virtual body model that needs to be carefully viewed before or during the operation through the list, thereby compensating for the limitation in the process of minimally invasive surgery, which has a limited field of view, and related Damage to organs and blood vessels and side effects after surgery can be minimized.

The steps of a method or algorithm described in relation to an embodiment of the present invention may be implemented directly in hardware, as a software module executed by hardware, or by a combination thereof. A software module may contain random access memory (RAM), read only memory (ROM), erasable programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory, hard disk, removable disk, CD-ROM, or It may reside in any type of computer-readable recording medium well known in the art to which the present invention pertains.

As mentioned above, although embodiments of the present invention have been described with reference to the accompanying drawings, those skilled in the art to which the present invention pertains can realize that the present invention can be embodied in other specific forms without changing its technical spirit or essential features. you will be able to understand Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive.

Claims

A method for intraoperative vascular navigation performed by the system, comprising:

constructing a three-dimensional blood vessel model using the blood vessel learning model from the two-dimensional medical image data of the subject;

Recognizing the surgical stage of the object using the surgical stage learning model in the photographed surgical image;

extracting major blood vessels corresponding to the recognized surgical steps from the constructed 3D blood vessel model and matching them to a navigation image; and

Including; providing the navigation image to the user;

The navigation image includes blood vessels branching from the main blood vessels,

The branched blood vessel can be added or removed based on a user input, an intraoperative blood vessel navigation method.
The method of claim 1,

The navigation image to which the major blood vessels are matched includes an image in which an object is 3D modeled using the medical image data or the photographed surgical image.
The method of claim 1,

The surgical step learning model is machine-learned by defining at least one or more steps necessary for the operation of the object as a label using surgical data, and inputting a learning image for each defined label, a method for vascular navigation during surgery.
4. The method of claim 3,

The surgical data, including tracking data for the movement of the robot arm or data obtained from the surgical image frame, intraoperative blood vessel navigation method.
The method of claim 1,

The blood vessel learning model is, after modeling veins and arteries based on medical image data, sequentially applying blood vessel bifurcation points to give a blood vessel type, an intraoperative blood vessel navigation method.
The method of claim 1,

The step of matching the main blood vessels to the navigation image comprises:

Collecting the location information of the surgical image screen,

An intraoperative blood vessel navigation method of synchronizing (SYNC) the three-dimensionally modeled main blood vessel and the object to the surgical image based on the operation stage and location information.
The method of claim 1,

The step of providing the navigation image includes:

An intraoperative blood vessel navigation method for highlighting and displaying the extracted major blood vessels in the navigation image.
8. The method of claim 7,

The step of highlighting and displaying the main blood vessels is

Displaying a list including items of the extracted major blood vessels on the navigation image,

When a specific item is selected from the list, the scene of the navigation image is moved to the scene of the main blood vessel corresponding to the selected specific item,

In the moved scene, the main vessel corresponding to the specific item is highlighted and displayed, the intraoperative vessel navigation method.
9. The method of claim 8,

The step of highlighting and displaying the main blood vessels is

A first highlight display method for displaying the main blood vessels by zooming in at a preset magnification, a second highlight display method for displaying the main blood vessels with a preset color, and a third highlight display method for displaying the main blood vessels by blinking the edges of the main blood vessels At least one or a combination of two or more of a display method, a fourth highlight display method for displaying text indicating the main blood vessel on the main blood vessel, and a fifth highlight display method for displaying the main blood vessel in a plurality of angles Indicative, intraoperative vascular navigation method.
In combination with a computer that is hardware, a computer program stored in a computer-readable recording medium to perform a blood vessel navigation method during surgery,

The computer program is

a process of constructing a three-dimensional blood vessel model using a blood vessel learning model from two-dimensional medical image data of an object;

A process of recognizing a surgical stage of an object using a surgical stage learning model in the photographed surgical image;

a process of extracting major blood vessels corresponding to the recognized surgical stage from the constructed 3D blood vessel model and matching them to a navigation image; and

performing the process of providing the navigation image to the user,

The navigation image includes blood vessels branching from the main blood vessels,

The branched blood vessels may be added or removed based on a user input, a computer program.
medical imaging equipment for taking surgical images;

a display unit for providing a surgical navigation image to a user; and

one or more processors; and

When executed by the one or more processors, the one or more processors comprising a control unit including one or more memories stored in the instructions to perform an operation,

The operation performed by the control unit is,

an operation of constructing a 3D blood vessel model using a blood vessel learning model from the two-dimensional medical image data of an object;

an operation for recognizing the surgical stage of the object using the surgical stage learning model in the photographed surgical image;

an operation of extracting major blood vessels corresponding to the recognized surgical steps from the constructed 3D blood vessel model and matching them to a navigation image; and

Comprising the operation of providing the navigation image to the user,

The navigation image includes blood vessels branching from the main blood vessels,

The branched blood vessel can be added or removed based on a user input, an intraoperative blood vessel navigation system.
12. The method of claim 11,

The navigation image to which the main blood vessels are matched includes a three-dimensional object image modeled from the photographed surgical image or two-dimensional medical image data of the object.
12. The method of claim 11,

The operation performed by the control unit is,

Collecting the location information of the captured surgical image screen,

The intraoperative vascular navigation system further comprising an operation for synchronizing (SYNC) the three-dimensionally modeled main blood vessel and the object to the surgical image based on the recognized surgical stage and location information.