KR102195850B1 - Method and system for segmentation of vessel using deep learning - Google Patents

Abstract

According to an embodiment of the present invention, there is provided an image acquisition unit for obtaining a visible ray image and an indocyanin green infrared fluorescence angiography image; An image matching unit for generating matching information by matching a visible ray image and an indocyanine green infrared angiography image to the same location; A verification image production unit for producing ground truth information by converting the indocyanine green infrared fluorescence angiography image to thresholding or other types of algorithms; A training unit for supervised training a deep learning vessel classification algorithm based on the matching information and using the actual verification information as a correct answer image; Deep learning blood vessel classification system comprising a is provided. The field verification information production section, the training section, and the verification section can be integrated and operated as a single deep learning network.

Description

Deep learning based vessel classification method and system {METHOD AND SYSTEM FOR SEGMENTATION OF VESSEL USING DEEP LEARNING}

The present invention relates to a deep learning-based blood vessel classification method and system, and more particularly, a deep learning blood vessel classification algorithm with field verification information generated using an indocyanine green infrared fluorescence angiography image to provide a highly accurate blood vessel classification algorithm. It is a method and system for providing.

Machine learning algorithms can be divided into supervised learning and unsupervised learning according to the learning method. In supervised learning, labeled data is used for data optimization, and unlabeled data is used in self-study, so the two are distinct.

When using machine learning algorithms for automatic analysis of medical data such as electrophysiological signals such as electrophysiological signals such as conventional EEG analysis and medical images, in the case of supervised learning, the required result based on the result of the judgment of the expert in the field of the labeled data As the amount of data became enormous, the method based on the expert's judgment required a lot of time and cost, and the accuracy was not constant depending on the competence of the expert. According to the existing method, experts have provided data for supervising machine learning by directly marking pathological and non-pathological findings in EEG and medical images.

Conventionally, when image data or surgical result data related to postoperative prognosis is collected, the analysis method (or diagnosis method) and the detection threshold and criteria to be applied in the analysis method are directly designated by a medical expert, and Since the surgical result data was analyzed according to the analysis method and detection threshold, it was too dependent on the expert's judgment. And it can be expensive to collect large-scale expert opinions. In particular, according to the above method, it has been pointed out as a problem that it is fundamentally impossible to develop a method with higher accuracy than that of an expert because the diagnosis method for analyzing data is determined solely by the opinion of an expert.

Experts who do not analyze by applying machine learning algorithms, etc. after data on bio-signal-related findings, surgical results, or treatment results such as imaging findings, electrophysiological signals, etc. have become expert labeled data If a data analysis method through supervised learning without manual labeling by experts is developed based on true value information that can be obtained without labeling, it is possible not only to prevent false diagnosis by experts, but also to accumulate related data. As it is accumulated, it will be possible to develop medical images and data analysis methods with higher accuracy.
The background technology of the present invention is disclosed in Korean Patent Application Publication No. 10-2016-0064562 (published on June 8, 2016, title of the invention: a method and apparatus for modeling the structure of a coronary artery from a 3D CTA image).

An object of the present invention is to develop an algorithm for classifying blood vessels in surgical images without manually marking images for segmentation and classification based on deep learning.

Another object of the present invention is to develop an algorithm for classifying blood vessels according to types of blood vessels.

An object of the present invention is to improve the accuracy of a surgery-related application system, a surgical navigation system, and a robot through a blood vessel classification algorithm, and to prevent accidental damage to blood vessels during surgery, thereby improving safety.

According to an embodiment of the present invention, an image acquisition unit for acquiring a visible ray image and an indocyanine green infrared fluorescence angiography image; An image matching unit for generating matching information by matching the visible ray image and the indocyanine-green infrared angiography image to the same location; A verification image production unit for producing ground truth information by dividing and thresholding the indocyanine-green infrared fluorescence angiography image by contrast artery, vein, or entire vessel phase, and then merging them if necessary; A training unit for supervised training a deep learning vessel classification algorithm based on the matching information and using the actual verification information as a correct answer image; Deep learning blood vessel classification system comprising a is provided.

In the present invention, the image acquisition unit may simultaneously acquire the visible light image and the indocyanine green infrared fluorescence angiography image.

In the present invention, the image matching unit uses one or more of resizing, coregistration, deformation, area selection, and frame movement to provide the visible light image and the guide. Cyanine green infrared fluorescence angiography images can be matched.

In the present invention, the image matching unit may divide the indocyanine green infrared fluorescence angiography image for each area or perform co-registration based on a three-dimensional visual field relationship. This is because even though the positions of the visible light recording camera and the indocyanin green infrared image recording camera are similar, there may be slight differences in the three-dimensional space.

In the present invention, the verification image production unit may threshold the indocyanin green infrared fluorescence angiography image using an Otsu's method or another algorithm.

In the present invention, the deep learning blood vessel classification algorithm includes semantic image segmentation such as a full-convolution neural network, a comvolution auto-encoder, and additionally, a structure-aware analysis, a topology, and a hierarchical structure analysis ( Hierarchical structure) may be based on a deep learning algorithm at a possible level.

In the present invention, the deep learning vessel classification system may verify the deep learning vessel classification algorithm by applying unused data to the trained vessel classification algorithm.

According to an embodiment of the present invention, an image acquisition unit for acquiring a visible ray image and an indocyanine green infrared fluorescence angiography image; A verification image production unit for producing ground truth information by dividing, thresholding, and merging the indocyanin green infrared fluorescence angiography image by contrast phase;

A training unit for supervised training a deep learning vessel classification algorithm using the actual verification information as a correct answer image; A verification unit for verifying the deep learning blood vessel classification algorithm by applying unused data to the trained blood vessel classification algorithm; Deep learning blood vessel classification system comprising a is provided.

According to an embodiment of the present invention, an image acquisition step of obtaining a visible ray image and an indocyanine green infrared fluorescence angiography image; An image matching step of generating matching information by matching the visible light image and the indocyanine green infrared angiography image to the same location; A verification image production step of producing ground truth information by thresholding the indocyanine green infrared fluorescence angiography image; A training step of supervised training a deep learning vessel classification algorithm based on the matching information and using the actual verification information as a correct answer image; Deep learning vessel classification method comprising a is provided.

In the present invention, in the image acquisition step, the visible light image and the indocyanin green infrared fluorescence angiography image may be simultaneously acquired.

In the present invention, the image matching step includes the visible light image and the visible ray image using one or more methods of resizing, coregistration, deformation, area selection, and frame movement. Indocyanin green infrared fluorescence angiography images can be matched.

In the present invention, in the image matching step, the indocyanine green infrared fluorescence angiography image may be divided by region or co-registration may be performed based on a three-dimensional visual field relationship.

In the present invention, in the step of producing the verification image, the indocyanin green infrared fluorescence angiography image may be thresholded using an Otsu's method.

In the present invention, the deep learning blood vessel classification algorithm may be based on a full convolutional neural network.

In the present invention, the deep learning vessel classification method may verify the deep learning vessel classification algorithm by applying unused data to the trained vessel classification algorithm.

According to an embodiment of the present invention, an image acquisition step of obtaining a visible ray image and an indocyanine green infrared fluorescence angiography image; A verification image production step of producing ground truth information by dividing and thresholding the indocyanin green infrared fluorescence angiography image for each phase and then merging them if necessary; A training step of supervised training a deep learning blood vessel classification algorithm using the actual verification information as a correct answer image; A verification step of verifying the deep learning blood vessel classification algorithm by applying unused data to the trained blood vessel classification algorithm; Deep learning vessel classification method comprising a is provided.

Application stage: This method is applied to improve the accuracy of surgical navigation, prevent damage to blood vessels during surgery, and use it to set surgical targets for blood vessels.

Also provided is a computer-readable recording medium for recording a computer program for executing the method according to the present invention.

According to the present invention, since indocyanine green infrared fluorescence angiography images are used as labeled ground truth data, a deep learning algorithm for classifying blood vessels can be trained without manual marking by an expert.

According to the present invention, the types of blood vessels can be distinguished using temporal characteristics in which an artery in an indocyanin green infrared fluorescence angiography image is first imaged and later capillaries, veins, and entire blood vessels are imaged.

According to the present invention, it is possible to improve the accuracy of surgical navigation location information with blood vessel location information accurately identified on the surgical field of view, prevent damage to blood vessels during surgery, and set surgical targets for blood vessels.

1 is a schematic diagram of a configuration example of a brain surgery and other surgery and vascular image analysis system according to an embodiment of the present invention. Through the network, various devices can be operated at times, places, and equipment during or outside surgery.
2 is a diagram showing the configuration of a server internal processor according to an embodiment of the present invention.
3 is a time series diagram illustrating a deep learning-based blood vessel distinction method according to an embodiment of the present invention.
4 shows a visible ray image and an indocyanin green infrared fluorescence angiography image according to an embodiment of the present invention.
5 shows an example of matching according to an embodiment of the present invention.
6 illustrates semantic segmentation learning according to an embodiment of the present invention.
7 is a diagram showing an example of classifying blood vessel types according to an embodiment of the present invention;
8 is an example of improving the accuracy of surgical navigation using the present invention.
9 is an illustration of a method of setting a blood vessel damage risk warning and a surgical robot position limit and a blood vessel target surgery target using the present invention.

DETAILED DESCRIPTION OF THE INVENTION The detailed description of the present invention to be described below refers to the accompanying drawings, which illustrate specific embodiments in which the present invention may be practiced. These embodiments are described in detail sufficient to enable those skilled in the art to practice the present invention. It is to be understood that the various embodiments of the present invention are different from each other but need not be mutually exclusive. For example, specific shapes, structures, and characteristics described herein may be changed from one embodiment to another and implemented without departing from the spirit and scope of the present invention. In addition, it should be understood that the positions or arrangements of individual elements in each embodiment may be changed without departing from the spirit and scope of the present invention. Therefore, the detailed description to be described below is not made in a limiting sense, and the scope of the present invention should be taken as encompassing the scope claimed by the claims of the claims and all scopes equivalent thereto. Like reference numerals in the drawings indicate the same or similar elements over several aspects.

Hereinafter, various embodiments of the present invention will be described in detail with reference to the accompanying drawings in order to enable those of ordinary skill in the art to easily implement the present invention.

1 is a diagram illustrating an example of a network environment according to an embodiment of the present invention.

The network environment of FIG. 1 shows an example including a plurality of user terminals 110, 120, 130, 140, a server 150, and a network 160. 1 is an example for explaining the invention, and the number of user terminals or the number of servers is not limited as shown in FIG. 1.

The plurality of user terminals 110, 120, 130, and 140 may be a fixed terminal implemented as a computer device or a mobile terminal. The plurality of user terminals 110, 120, 130, and 140 may be terminals of an administrator controlling the server 150. Examples of the plurality of user terminals 110, 120, 130, and 140 include smart phones, mobile phones, navigation, computers, notebook computers, digital broadcasting terminals, personal digital assistants (PDAs), and portable multimedia players (PMPs). ), tablet PC, etc. For example, the user terminal 1 110 may communicate with other user terminals 120, 130, 140 and/or the server 150 through the network 160 using a wireless or wired communication method.

The communication method is not limited, and short-range wireless communication between devices as well as a communication method using a communication network (for example, a mobile communication network, a wired Internet, a wireless Internet, a broadcasting network) that the network 160 may include may be included. For example, the network 160 includes a personal area network (PAN), a local area network (LAN), a campus area network (CAN), a metropolitan area network (MAN), a wide area network (WAN), and a broadband network (BBN). , Internet, and the like. In addition, the network 160 may include any one or more of a network topology including a bus network, a star network, a ring network, a mesh network, a star-bus network, a tree or a hierarchical network, etc. Not limited.

The server 150 is a computer device or a plurality of computer devices that communicates with a plurality of user terminals 110, 120, 130, 140 and a network 160 to provide commands, codes, files, contents, services, etc. Can be implemented.

For example, the server 150 may provide a file for installing an application to the user terminal 1 110 accessed through the network 160. In this case, the user terminal 1 110 may install an application using a file provided from the server 150. In addition, by accessing the server 150 under the control of an operating system (OS) included in the user terminal 1 110 and at least one program (for example, a browser or an installed application), the service provided by the server 150 Content can be provided. As another example, the server 150 may establish a communication session for data transmission/reception, and may route data transmission/reception between a plurality of user terminals 110, 120, 130, and 140 through the established communication session.

The method and system for distinguishing deep-learning blood vessels according to an embodiment of the present invention relates to a method and system for detecting, discriminating, and demarcating a blood vessel in a surgical image through deep learning. In more detail, the deep learning vessel classification method and system according to an embodiment of the present invention provides information on the boundary of an object as an image using semantic segmentation to provide a fully convolutional neural network or a combol A deep learning algorithm such as a convolutional autoencoder is supervised training.

A learning image is required for such instructional training. According to the present invention, the blood vessel information image for learning may be based on an indocyanin green infrared fluorescence angiography image. In addition, the classification of arteries, arterioles, veins, predetermined veins, capillaries, etc., using different types of blood vessels displayed according to the temporal phase of the indocyanin green infrared fluorescence angiography image Training and classification by (class) are possible. The present invention can be applied to various fields of surgery and microscopic images, and in particular, may be used to distinguish cerebrovascular blood vessels in brain surgery microscopic images, but is not limited thereto. Hereinafter, the present invention will be described in more detail, focusing on the internal configuration of the server 150.

2 is a block diagram illustrating an internal configuration of a server according to an embodiment of the present invention.

In FIG. 2, the internal configuration of the server 150 will be described. The server 150 may include a memory 221, a processor 222, a communication module 223, and an input/output interface 224. The memory 221 is a computer-readable recording medium and may include a permanent mass storage device such as a random access memory (RAM), read only memory (ROM), and a disk drive. In addition, the memory 221 may store an operating system and at least one program code (for example, a browser installed and driven in the user terminal 1 110 or a code for the above-described application). These software components may be loaded from a computer-readable recording medium separate from the memory 221 using a drive mechanism. Such a separate computer-readable recording medium may include a computer-readable recording medium such as a floppy drive, a disk, a tape, a DVD/CD-ROM drive, and a memory card. In another embodiment, software components may be loaded into the memory 221 through the communication module 223 rather than a computer-readable recording medium. For example, at least one program is a program installed by files provided through the network 160 by a file distribution system (for example, the server 150 described above) for distributing the installation files of developers or applications (example It may be loaded into the memory 221 based on the above-described application).

The processor 222 may be configured to process instructions of a computer program by performing basic arithmetic, logic, and input/output operations. Instructions may be provided to the processor 222 by the memory 221 or the communication module 223. For example, the processor 222 may be configured to execute a command received according to a program code stored in a recording device such as the memory 221. The processor 222 may include an image acquisition unit 111, an image matching unit 112, a verification image production unit 113, and a training unit 114.

The communication module 223 may provide a function for the user terminal 1 110 and the server 150 to communicate with each other through the network 160, and other user terminals (for example, user terminal 2 120) or other A function for communicating with a server (for example, the server 150) may be provided. For example, a request generated by the processor of the user terminal 1 110 according to a program code stored in a recording device such as a memory may be transmitted to the server 150 through the network 160 under control of a communication module. Conversely, control signals, commands, contents, files, etc. provided under the control of the processor 222 of the server 150 are transmitted through the communication module 223 and the network 160 to the communication module of the user terminal 1 110. It may be received through the user terminal 1 (110). For example, control signals or commands of the server 150 received through the communication module may be transmitted to a processor or memory.

The input/output interface 224 may be a means for an interface with an input/output device. For example, the input device may include a device such as a keyboard or a mouse, and the output device may include a device such as a display for displaying a communication session of an application. As another example, the input/output interface 224 may be a means for an interface with a device in which input and output functions are integrated into one, such as a touch screen. As a more specific example, the processor 222 of the server 150 inputs/outputs a service screen or contents configured using data provided by the user terminal 2 120 in processing a command of a computer program loaded in the memory 221 It may be displayed on the display through the interface 224.

In addition, in other embodiments, the server 150 may include more components than those of FIG. 2. However, there is no need to clearly show most of the prior art components. For example, the server 150 may further include other components such as a transceiver, a Global Positioning System (GPS) module, a camera, various sensors, and a database.

Hereinafter, the present invention will be described in more detail for each configuration of the processor 222 of FIG. 2.

3 is a time series diagram illustrating a deep learning-based blood vessel distinction method according to an embodiment of the present invention.

First, the image acquisition unit 111 simultaneously acquires a visible light image and an indocyanin green infrared fluorescence angiography image during surgery (S1).

Next, the image matching unit 112 matches the visible light image and the indocyanine green infrared angiography image to the same location (S2). In the following specification of the present invention, a visible ray image may be referred to as a first image, and an indocyanine green infrared fluorescence angiography image may be referred to as a second image.

Next, the verification image production unit 113 processes the indocyanin green infrared fluorescence angiography image with thresholding, Otsu's method, other types of machine learning or deep learning algorithms, etc. to provide ground truth image. To produce (S3). Deep learning algorithms used here include a deep convolutional neural network, a deep recurrent neural network, and a deep autoencoder, and data of multiple image frames can be used. A Siamese or fusion structured deep network algorithm can be used.

Next, the training unit 114 supervised training the deep-rowing vessel classification algorithm based on the matching information and using the actual verification information as the correct answer image (S4). In this case, a fully convolutional neural network, a convolutional autoencoder, a hypercolumn, or an image segmentation network capable of structure-aware, etc. An algorithm capable of semantic segmentation can be used. In addition, it is possible to simultaneously process as a single deep learning network by integrating it as an input of a Siamese structured network without putting much of the process of producing the above actual verification information.

Next, apply new data that has not been used for training to the trained deep learning vessel classification algorithm to verify the data, and then apply the verified deep learning vessel classification algorithm to actual surgical field analysis or other application, adjustment for neuronavigation accuracy. (Fig. 8), operating microscope virtual reality application, surgical assistance and automatic warning system (Fig. 9), vascularture and vascular flow analysis system), structure-aware network, etc.

Hereinafter, the present invention and examples of application of the present invention will be described in more detail, focusing on each configuration.

First, the image acquisition unit 111 simultaneously acquires a visible ray image and an indocyanin green infrared fluorescence angiography image during surgery. Indocyanine green infrared image reveals blood vessels well, but it is difficult for humans to understand other anatomical structures other than blood vessels, and visible ray images show blood vessels, but discriminating and discriminating only blood vessels without additional information is currently in machine learning and It is not easy with a deep learning algorithm. In addition, even when an expert in the imaging field such as a surgeon sees it, the identification of blood vessels is not clear, and thus, sometimes refer to the indocyanine green infrared image information. Visible light imaging is suitable for identifying different anatomical structures. Visible light images are continuously used during surgery, unlike indocyanine green images, which are temporarily contrasted only when indocyanine green is injected intravenously. Accordingly, according to an embodiment of the present invention, a deep learning blood vessel classification system aims to classify blood vessels in a visible ray image and add a mark to facilitate human recognition. At this time, the position of the added mark can be obtained by training a deep learning algorithm based on the indocyanin green infrared fluorescence angiography image. To this end, it is necessary to match the visible light image and the indocyanin green infrared fluorescence angiography image. Therefore, it is necessary to take a picture at the same time or simultaneously with a visible ray image and an indocyanin green infrared fluorescence angiography image within a short time.

Next, the image matching unit 112 matches the visible ray image and indocyanine green infrared angiography to the same location. To this end, the image matching unit 112 resizes the first image and the second image, coregistration, deformation, area selection, frame movement, or other deep learning based It matches to the same location through automatic and manual processes using feature analysis and coregistration algorithm.

4 shows a visible ray image and an indocyanin green infrared fluorescence angiography image according to an embodiment of the present invention. 4A is a visible ray image, and (b) is an indocyanin green infrared fluorescence angiography image.

Although not clearly visible in FIGS. 4A and 4B, the images may not match even if images captured using different devices are taken at the same time. According to an embodiment of the present invention, in order to identify a blood vessel, two images must be matched to enable proper deep learning training, and thus a process of performing matching may be provided.

5 shows an example of matching according to an embodiment of the present invention.

Referring to FIG. 5, (a) of FIG. 5 is an indocyanin green infrared fluorescence angiography image, (b) is a visible ray image, and (c) is an image matching (a) and (b). Referring to the indocyanin green infrared fluorescence angiography image of (a), the part marked with v may be the cerebrovascular blood vessels imaged by indocyanin green (ICG) near-infrared angiography. As shown in (a), indocyanin green infrared fluorescence angiography can be used to determine the location of the contrasted blood vessel. On the other hand, in the visible ray image of (b), it is difficult to determine the location of blood vessels. Accordingly, the deep learning blood vessel classification system according to an embodiment of the present invention trains the blood vessel location using an indocyanine green infrared fluorescence angiography image in order to easily display the blood vessel location on a visible ray image. That is, in order to train deep learning on blood vessel location information using indocyanine green infrared fluorescence angiography, it is necessary to match the visible light image with the indocyanine green infrared fluorescence imaging image.

In more detail, according to an embodiment of the present invention, the image matching unit 112 may use techniques such as resizing, coregistration, deformation, area selection, and frame movement. have. More specifically, in the embodiment of FIG. 5, in order to synchronize the frame time, in order to synchronize the frame time, one frame is moved by 2 frames in consideration of the frame parallax of the video in which the images (a) and (b) are captured. Additionally, a matching image as shown in (c) was generated by performing 2D coregistration on the images (a) and (b).

In more detail, for more accurate matching, since there is a positional difference between the indocyanin green infrared fluorescence angiography image and the visible light image, the image may be divided by region or co-registration may be performed in consideration of a three-dimensional visual field relationship. In this case, according to an embodiment of the present invention, image matching may be performed using an automatic rigid coregistration algorithm. Or it can be matched through other feature-based or deep learning-based algorithms.

Next, the verification image production unit 113 produces a ground truth image by thresholding the indocyanin green infrared fluorescence angiography image. Thresholding means specifying a threshold value used in the field of image processing. Thresholding is a method of image segmentation, and as an example, each pixel of an image may be represented by only one of two colors. For example, if the image intensity is greater than a certain constant, it may be represented as a black pixel, and if the image intensity is less than a certain constant, it may be represented as a white pixel. The above-described thresholding method is only an example, and various modifications may be made in keeping with the characteristic configuration of the present invention. Otsu's method and other machine learning and image processing convolutional or recurrent deep learning network techniques can be used to convert indocyanine green images into ground truth images.

In more detail, the thresholding of indocyanin green infrared fluorescence angiography images can be performed using machine learning algorithms such as Otsu's method. In addition, thresholding can be performed using supervised training of deep learning or autoendoding algorithm. When using deep learning-based supervised training, appropriate thresholding of indocyanin green infrared fluorescence angiography image It is better to learn a fully convolutional neural network or a convolutional autoencoder for indocyanine green infrared fluorography images using the result drawn by an expert as ground truth information. It is possible.

In addition, the image to which the indocyanin green infrared fluorescence angiography image is thresholded may be a ground truth image or ground truth information. The actual verification image or the actual verification information is an image or information indicating where the actual blood vessel is, and a deep learning blood vessel identification algorithm may be trained using the actual verification image or information. According to an embodiment of the present invention, an actual verification image or information may be generated using an indocyanin green infrared fluorescence angiography image without manual marking by an expert. That is, as shown in (c) of FIG. 5, the actual verification information generated by thresholding is matched with a visible ray image, so that the location information of the blood vessel may be used as the actual verification information for deep learning training. In addition, since the process of producing such field verification information can be automated, deep learning training can be performed with a high level of accuracy by providing a large variety of accurate field verification information that cannot be reached manually.

Next, the training unit 114 supervised training the deep learning algorithm using the thresholded indocyanin green infrared fluorescence angiography image as ground truth information. That is, the training unit 114 supervises and learns the cerebrovascular position based on the correct answer image based on the indocyanin green infrared fluorescence angiography image in order to semantic segmentation of the cerebrovascular position. At this time, during supervised learning, a correct answer image may be generated from the actual verification information using the matching information, that is, the information in which the visible light image and the indocyanin green infrared fluorescence angiography image are matched, and supervised learning may be performed with the correct answer image. . In addition, the process of producing actual verification information is not required, and an indocyanine green image and a visible ray image can be simultaneously input and processed as a single deep learning network through the input of the Siamese structure network.

6 illustrates semantic segmentation learning according to an embodiment of the present invention.

Referring to FIG. 6, (a) of FIG. 6 is an image of a brain surgery microscope (visible ray image), (b) is an image of a correct answer, and (c) is an example of semantic segmentation learning results. As described above, in the method for classifying a blood vessel type according to an embodiment of the present invention, by determining actual verification information in consideration of matching information as the correct answer image, the shape of the blood vessel is trained based on the correct answer images. When training is performed, the semantic segmentation result as shown in (c) of FIG. 6 can be obtained. As can be seen in the photo example of FIG. 6, the method of distinguishing the type of blood vessel of the present invention learned from actual verification information can show good semantic segmentation results.

In addition, the processor 222 of the present invention may additionally include a verification unit, and the verification unit verifies the deep learning vessel classification algorithm by applying new data that has not been used for training to the trained deep learning vessel classification algorithm of the present invention. , The proven deep learning blood vessel classification algorithm can be used in actual surgery.

On the other hand, indocyanin green infrared fluorescence angiography is where indocyanin green material, that is, a fluorescent material, is injected from an artery, starts from a large artery of the brain through the aorta and carotid artery, passes through the arterioles, passes through capillaries, and exits through a vein or vein It features. Therefore, in the case of using indocyanin green infrared fluorescence angiography, there is a difference in the time period in which each blood vessel is contrasted according to the timing of the passing of the fluorescent material, and the present invention distinguishes it and generates separate field verification information to distinguish deep learning blood vessels. You can train the algorithm. Accordingly, the deep learning blood vessel classification method according to an embodiment of the present invention can distinguish blood vessel types.

In more detail, when acquiring a visible ray image and an indocyanine green infrared fluorescence angiography image, the image acquisition unit 111 may acquire an image in consideration of timing at which the fluorescent material passes through each blood vessel. In more detail, the image acquisition unit 111 may obtain an image for each time interval after dividing the time interval in consideration of timing at which the fluorescent material passes through each blood vessel.

For example, so that the type of blood vessel is appropriately classified for each time section, the first time section through which the fluorescent material passes through the aorta, the second time section through which the fluorescent material passes through the carotid artery... are divided into sections, and each time section You can acquire an image every time. To this end, the image acquisition unit 111 may mark a change in brightness of the entire acquired image as a graph over time. The method of distinguishing the type of blood vessel in the indocyanine green image is not limited to the above method, and the image features of a summary dimension of the timing and pattern of angiography with a deep recurrent neural network and a deep convolution autoencoder, etc. dimensionality feature) can be used to classify the type of blood vessel by the type of the indocyanine green image. Indocyanin green infrared fluorescence angiography images generated by time intervals as described above or by other methods may indicate the type of blood vessel corresponding to the time interval. Accordingly, the training unit may train a blood vessel classification algorithm based on the blood vessel type corresponding to each time section, taking into account that the actual verification information is information on the corresponding blood vessel type.

In addition, if the fluorescent material is uniformly distributed over time, the entire blood vessel may be imaged, so the image acquisition unit 111 acquires the indocyanine green infrared fluorescence angiography image in which the entire blood vessel is imaged after a predetermined time elapses. Can be done, and actual verification information may be generated based on this. In this case, the verification image production unit 113 may exclude the actual verification information of the arterial phase, which is the actual verification information of the time interval corresponding to the artery, from the actual verification information in which the entire blood vessel is imaged, and based on this You can get the actual verification information for.

7 is a diagram showing an example of classifying blood vessel types according to an embodiment of the present invention.

Referring to FIG. 7, (a) of FIG. 7 is an indocyanin green infrared fluorescence angiography image of an artery phase, and (b) is an indocyanin green infrared fluorescence angiography image of an entire blood vessel. That is, only arteries are displayed in the indocyanin green infrared fluorescence angiography image of (a), and both arteries and veins are displayed in the indocyanine green infrared fluorescence angiography image of (b). In addition, (c) of FIG. 7 is actual verification information obtained by thresholding (a) the image and (d) the (b) image. If you perform the operation of removing the display portion of the image from (d) to (c), you can obtain the location information of the vein. As described above, according to an embodiment of the present invention, the overall accuracy of the blood vessel classification algorithm may be improved by performing training through subdivision by blood vessel type.

The detection of cerebrovascular blood vessels in a brain surgery microscope or visible light image disclosed in the blood vessel classification system according to an embodiment of the present invention may be used for blood vessel avoidance or blood control in various surgeries.

In the existing researched blood vessel detection method, for deep learning training of surgical images, ground truth information is produced by manual marking of structures to be classified, so a lot of time and money are spent for training. do. In particular, in the case of a biological structure that is difficult to distinguish, such as blood vessels, there is a problem that the accuracy of the manual marquee is also poor, and the more complicated the structure, the more expensive an expert's input time increases, and thus a problem exists.

On the other hand, the method for classifying blood vessels according to an embodiment of the present invention can reduce the labor input of experts as in the prior art by obtaining actual verification information for semantic segmentation using indocyanin green infrared fluorescence angiography image, The accuracy of actual verification information can be improved. In addition, deep learning algorithms can be trained by providing actual verification information having a high level of accuracy that is impossible with manual marking to various types of blood vessels.

8 is an example of improving the accuracy of surgical navigation using the present invention.

In more detail, FIG. 8 is an example of a method of correcting location information of a navigation system using the present technology. If the present cerebrovascular segmentation method is applied separately to the left and right visual fields and the above information is applied to the existing three-dimensional visual field, the position of the blood vessel in the three-dimensional space can be derived. Through the above information, the difference in three-dimensional space with the blood vessel position estimated on the surgical navigation is calculated to correct the location information of the surgical navigation so that it matches the actual observed surgical field of view. Through the above method, it is possible to correct the position of the surgical navigation due to deformation of soft tissue and other causes. The application of the above technology is not limited to navigation, and can be used to correct the position of image information of the augmented reality system projected on image projection equipment such as a microscope for brain surgery or an endoscope monitor, and other virtual reality glasses through the position of blood vessels in the surgical field of view. .

9 is an illustration of a method of setting a blood vessel damage risk warning and a surgical robot position limit and a blood vessel target surgery target using the present invention.

In more detail, FIG. 9 is an example of an alarm and safety system for preventing blood vessel damage during surgery using the present technology. After grasping the location of the blood vessel in the surgical field in a three-dimensional space by the method as shown in FIG. 8, an alarm is generated to the surgeon or a surgical robot when a sharp or distal part of a surgical tool approaches or intersects the blood vessel with a conventional technique. It is possible to stop the operation of surgical tools such as the arm. This can prevent unintended damage to blood vessels. This system can be temporarily shut down when special surgical techniques such as vascular resection are required. And, conversely, when surgical techniques such as resection or ligation of blood vessels are needed, location information for setting targets for surgery on surgical robots, locations for vascular clip, delamination of surrounding tissues and tumors, etc., and appropriate access directions are provided. can do.

The embodiments according to the present invention described above may be implemented in the form of program instructions that can be executed through various computer components and recorded in a computer-readable recording medium. The computer-readable recording medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded in the computer-readable recording medium may be specially designed and configured for the present invention or may be known and usable to those skilled in the computer software field. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CD-ROMs and DVDs, and magnetic-optical media such as floptical disks. medium), and a hardware device specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of the program instructions include not only machine language codes such as those produced by a compiler but also high-level language codes that can be executed by a computer using an interpreter or the like. The hardware device can be changed to one or more software modules to perform the processing according to the present invention, and vice versa.

In the above, the present invention has been described by specific matters such as specific elements and limited embodiments and drawings, but this is provided only to help a more general understanding of the present invention, and the present invention is not limited to the above embodiments. Anyone with ordinary knowledge in the technical field to which the invention belongs can make various modifications and changes from these descriptions.

Accordingly, the spirit of the present invention is limited to the above-described embodiments and should not be defined, and all ranges equivalent to or equivalently changed from the claims to be described later as well as the claims to be described later are the scope of the spirit of the present invention. It will be said to belong to.

Claims (17)

An image acquisition unit for acquiring a visible ray image and an indocyanine green infrared fluorescence angiography image;
An image matching unit for generating matching information by matching a visible ray image and an indocyanine green infrared angiography image to the same location;
A verification image production unit for producing ground truth information by thresholding the indocyanin green infrared fluorescence angiography image;
A training unit for supervised training a deep learning vessel classification algorithm based on the matching information and using the actual verification information as a correct answer image;
Deep learning blood vessel classification system comprising a. An image acquisition unit for acquiring a visible ray image and an indocyanine green infrared fluorescence angiography image;
A verification image production unit for producing ground truth information by thresholding the indocyanin green infrared fluorescence angiography image;
A training unit for supervised training a deep learning vessel classification algorithm using the actual verification information as a correct answer image;
A verification unit for verifying the deep learning blood vessel classification algorithm by applying unused data to the trained blood vessel classification algorithm;
Deep learning blood vessel classification system comprising a. The method according to claim 1 or 2,
The image acquisition unit acquires the visible light image and the indocyanine green infrared fluorescence angiography image at the same time, a deep learning vessel classification system. The method of claim 1,
The image matching unit,
Matching the visible light image and the indocyanine green infrared fluorescence angiography image using one or more methods of resizing, coregistration, deformation, area selection, and frame movement , Deep learning vessel classification system. The method of claim 1,
The image matching unit divides the indocyanin green infrared fluorescence angiography image by region or performs co-registration based on a three-dimensional visual field relationship. The method according to claim 1 or 2,
The verification image production unit converts the indocyanin green infrared fluorescence angiography image into a field verification image for deep learning training using Otsu's method or other machine learning and deep learning algorithms, a deep learning vessel classification system. The method according to claim 1 or 2,
A deep learning vessel classification system that trains a deep learning algorithm capable of full convolutional neural network or other semantic image segmentation through a training data group. The method according to claim 1 or 2,
The deep learning blood vessel classification system verifies the deep learning blood vessel classification algorithm by applying unused data to the trained blood vessel classification algorithm. An image acquisition step of obtaining a visible ray image and an indocyanine green infrared fluorescence angiography image;
An image matching step of generating matching information by matching the visible ray image and the indocyanine green infrared angiography image to the same location;
A verification image production step of producing ground truth information by thresholding the indocyanine green infrared fluorescence angiography image;
A training step of supervised training a deep learning vessel classification algorithm based on the matching information and using the actual verification information as a correct answer image;
Deep learning vessel classification method comprising a. An image acquisition step of obtaining a visible ray image and an indocyanine green infrared fluorescence angiography image;
A verification image production step of producing ground truth information by thresholding the indocyanine green infrared fluorescence angiography image;
A training step of supervised training a deep learning vessel classification algorithm using the actual verification information as a correct answer image;
A verification step of verifying the deep learning vessel identification algorithm by applying unused data to the trained vessel identification algorithm;
Deep learning vessel classification method comprising a. The method of claim 9 or 10,
In the step of obtaining the image, the visible ray image and the indocyanin green infrared fluorescence angiography image are simultaneously acquired. The method of claim 9,
The image matching step,
Resizing, coregistration, deformation, area selection, frame movement, and deep learning algorithms using one or more methods of the visible light image and the indocyanine green infrared fluorescence imaging Deep learning vein classification method that matches images. The method of claim 9,
In the image matching step, the indocyanin green infrared fluorescence angiography image is segmented for each area or co-registration is performed based on a three-dimensional visual field relationship. The method of claim 9 or 10,
In the step of producing the verification image, the indocyanin green infrared fluorescence angiography image is thresholded using an Otsu's method. The method of claim 9 or 10,
The deep learning blood vessel classification method is based on a full convolutional neural network or another type of deep learning algorithm. The method of claim 9 or 10,
The deep learning blood vessel classification method is a deep learning blood vessel classification method for verifying the deep learning blood vessel classification algorithm by applying unused data to the trained blood vessel classification algorithm. A computer-readable recording medium recording a computer program for executing the method according to any one of claims 9, 10, 11 and 12.

Images