WO2020122672A1 - Apparatus and method for automatically segmenting blood vessels by matching fp image and fag image - Google Patents

Abstract

The present invention discloses an apparatus and a method for automatically segmenting blood vessels by matching an FP image and an FAG image. A method, according to one aspect of the present invention, may automatically extract a precise region of retinal blood vessels in an FP image by matching the FP image and an FAG image.

Description

Automatic blood vessel segmentation device and method using matching of fundus image and fluorescein angiography image

The present invention relates to an automatic blood vessel segmentation apparatus and method using matching of a fundus image and a fluorescein angiogram, and more specifically, a fundus image and a fluorescein angiogram that can automatically extract a precise region of retinal blood vessels in the fundus image. An automatic blood vessel segmentation apparatus and method using image matching.

This application claims priority based on Korean Patent Application No. 10-2018-0161131 filed on December 13, 2018, and all contents disclosed in the specification and drawings of the application are incorporated in this application.

Fundus imaging is one of the most used ophthalmic photographs in ophthalmology for diagnostic or documentation purposes. The fundus image is relatively similar to the fundus of the subject observed at the time of treatment, and thus is intuitive, and thus is used for examining ophthalmic diseases. On the other hand, clinicians intend to quantitatively analyze blood vessel characteristics based on these fundus images, and develop a system for diagnosing diseases based on the fundus image, but the accuracy of the technique for extracting blood vessel regions is still limited in accuracy.

(Patent Document 1) Korean Registered Patent No. 10-1761510 (announced on July 26, 2017)

The present invention is proposed to solve the above problems, matching the fundus image and the fluorescein angiography image capable of automatically extracting a precise region of retinal blood vessels in the fundus image by matching the fundus image and the fluorescein angiography image. It is an object of the present invention to provide an automatic blood vessel segmentation device and method.

Other objects and advantages of the present invention may be understood by the following description, and will be more clearly understood by an embodiment of the present invention. In addition, it will be readily appreciated that the objects and advantages of the present invention can be realized by means of the appended claims and combinations thereof.

Automatic blood vessel segmentation method using matching of the fundus image and the fluorescein angiogram according to an aspect of the present invention for achieving the above object, the patient's fundus image (FP image) and fluorescein angiography image (FAG image) ) Obtaining frames; Rigid registration of each frame of the fluorescein angiography image using a feature point matching technique; Performing vascular extraction of the matched fluorescein angiography image (FAG image) based on deep learning according to the characteristics of the fluorescein angiography image (FAG image); Integrating blood vessel extraction results of fluorescein angiography (FAG image) frames into an average value; Performing blood vessel extraction of the fundus image based on deep learning according to the characteristics of the fundus image; Performing a matching of a fluorescence fundus angiography image (FAG image) and a fundus image (FP image) from the extracted blood vessels; And dividing blood vessels based on the matched results.

Rigid registration of each frame of the fluorescein angiography image using a feature point matching technique uses RANSAC (RANdom Sample Consensus) during feature point detection, feature point descriptor extraction, and feature point matching. And performing registration of a fluorescein angiography image (FAG image).

The deep learning is a learned convolutional neural network (CNN), and the blood vessels of the fluorescein angiogram (FAG image) matched based on deep learning according to the characteristics of the fluorescein angiogram (FAG image). The step of performing the extraction is characterized by including a step of deriving a deep learning-based FAG vessel probability map (FAGVP) based on blood vessels in a fluorescein angiography image (FAG image).

Integrating the blood vessel extraction results of the fluorescein angiography images (FAG image) frames into an average value is a non-rigid matching based on a free-form deformation technique of a coordinate grid represented by a B-Spline model in FAGVP. and performing a rigid registration) and deriving a Maximum FAG Vessel Probability map (M-FAGVP) extracting the average value of the average FAG Vessel Probability map (A-FAGVP) of the matched FAGVP and the maximum value for each pixel location. Is done.

In accordance with the characteristics of the fundus image, the step of performing blood vessel extraction of the fundus image based on deep learning derives a deep learning-based Fundus Photo Vessel Probability map (FPVP) for matching between the fundus image (FP image) and A-FAGVP. It characterized in that it comprises a step.

The step of performing the matching of the fluorescence fundus angiography image (FAG image) and the fundus image (FP image) from the extracted blood vessels, using a chamfer matching (Chamfer Matching) technique from blood vessels derived from FPVP and A-FAGVP Performing rigid registration between image-fluorescein angiography (FP-FAG) images; And performing non-rigid registration between the final fundus image-fluorescein angiography image (FP-FAG) based on the free-form deformation technique of the coordinate grid represented by the B-Spline model. It is characterized by including.

The step of dividing blood vessels based on the matched result is a step of deriving a binary vessel segmentation mask by applying a hysteresis thresholding technique based on the probability value for each pixel of A-FAGVP. ; Removing noise by applying a connected component analysis technique to the derived vascular segmentation mask; And segmentation based on the matched A-FAGVP reinforced to a gap occurring in a vein.

An automatic vascular segmentation apparatus using matching of a fundus image and a fluorescein angiogram according to another aspect of the present invention for achieving the above object, the patient's fundus image (FP image) and fluorescein angiography image (FAG image) ) An image acquisition unit that acquires frames; A FAG matching unit for rigid registration of each frame of the fluorescein angiography image using a feature point matching technique; A FAG blood vessel extraction unit performing blood vessel extraction of a matched fluorescein angiographic image (FAG image) based on deep learning according to the characteristics of the fluorescence angiography image (FAG image); An integrating unit for integrating vascular extraction results of fluorescein angiography (FAG image) frames into an average value; A FP blood vessel extraction unit performing blood vessel extraction of the fundus image based on deep learning according to characteristics of the fundus image; A FAG-FP matching unit performing registration of a fluorescein angiography image (FAG image) and a fundus image (FP image) from the extracted blood vessels; And a blood vessel dividing unit for dividing blood vessels based on the matched results.

According to an aspect of the present invention, by matching the fundus image and the fluorescein angiography image and automatically extracting a precise region of retinal blood vessels in the fundus image, the effect of early diagnosis and treatment of blindness-induced disease and/or chronic vascular disease is obtained. have.

The effects obtainable in the present invention are not limited to the above-mentioned effects, and other effects not mentioned may be clearly understood by those skilled in the art from the following description. .

The following drawings attached to the present specification are intended to illustrate preferred embodiments of the present invention, and serve to further understand the technical idea of the present invention together with specific details for carrying out the invention, and thus the present invention It should not be interpreted as being limited only to the items described.

1 is a schematic configuration diagram of a device for matching a fundus image and a fluorescein angiography image according to an embodiment of the present invention,

2 is a diagram for a result of matching using a SIFT technique according to an embodiment of the present invention,

Figure 3 is a result of deriving the FAG Vessel Probability map (FAGVP) in the fluorescence fundus angiography image (FAG image) according to an embodiment of the present invention,

FIG. 4 shows the results of two images that are similarly matched by rigid registration according to an embodiment of the present invention, which are precisely registered through the B-Spline technique,

5 is a result of precisely segmented blood vessel images according to an embodiment of the present invention,

6 is a flow chart of a method of matching a fundus image and a fluorescein angiography image according to an embodiment of the present invention,

7 is a view showing a FAG Vessel Probability map using deep learning according to an embodiment of the present invention,

8 is a view showing the results of rigid body matching (left) and non-rigid matching (right) according to an embodiment of the present invention,

9 is a view showing an Average FAGVP map (left) and a Maximum FAGVP map (right) according to an embodiment of the present invention,

10 is a diagram showing a FAGVP map derived using FP image (left) and deep learning according to an embodiment of the present invention,

11 is a view showing the results of rigid body matching using chamfer matching according to an embodiment of the present invention,

12 is a view showing the results of non-rigid matching using the B-Spline technique according to an embodiment of the present invention.

The above objects, features, and advantages will become more apparent through the following detailed description in connection with the accompanying drawings, and accordingly, those skilled in the art to which the present invention pertains can easily implement the technical spirit of the present invention. There will be. In addition, in the description of the present invention, when it is determined that the detailed description of the known technology related to the present invention may unnecessarily obscure the subject matter of the present invention, the detailed description will be omitted. Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

Throughout the specification, when a part “includes” a certain component, it means that the component may further include other components, not to exclude other components, unless otherwise stated. Also, “… The term “part” means a unit that processes at least one function or operation, which may be implemented by hardware or software or a combination of hardware and software.

1 is a schematic configuration diagram of an apparatus for matching a fundus image and a fluorescein angiogram according to an embodiment of the present invention, and FIG. 2 is a diagram for a result of matching using a SIFT technique according to an embodiment of the present invention It is a drawing.

Referring to FIG. 1, an apparatus for matching a fundus image and a fluorescein angiography image according to the present embodiment includes an image acquisition unit 110, a FAG matching unit 120, a FAG blood vessel extraction unit 130, and an integration unit 140 ), FP blood vessel extraction unit 150, FAG-FP matching unit 160.

The image acquisition unit 110 acquires frames of a patient's fundus image (FP image) and a fluorescein angiography image (FAG image). At this time, the fundus image (FP image) can be obtained through a fundus imaging device for the examination of eye diseases in the ophthalmology, the fluorescein angiography image (FAG image) is injected with a fluorescent substance (Fluorescein) in a vein, a fluorescent substance It can be acquired through a device that optically photographs circulating through the retinal circulatory system and displays blood vessels. Meanwhile, the above-described image is composed of multiple frames according to the passage of time.

The FAG matching unit 120 matches each frame of the fluorescein angiography image (FAG image) composed of multiple frames with the passage of time using a feature point matching technique. More specifically, the FAG matching unit 120 performs rigid registration of a fluorescein angiography image using a Scale Invariant Feature Transform (SIFT) technique. The FAG matching unit 120 may perform matching of a fluorescence angiography image (FAG image) using RANSAC (RANdom Sample Consensus) during feature point detection, feature point descriptor extraction, and feature point matching. Fluorescein angiography (FAG image) changes in blood vessels and background over time. Therefore, the present invention utilizes a Scale Invariant Feature Transform (SIFT) technique to detect various features while minimizing image changes. Using SIFT (Scale Invariant Feature Transform) technique, various regional features such as optic disc, fovea, local vessel structure in fluorescein angiography image can be found. To minimize visual changes based on these features, features within two images are matched, and based on this, perspectivce transform based on RANSAC (RANDOM SAMPLE CONSENSUS) technique is estimated. Rigid registration between two images is performed with the estimated perspective transform. Continuously performing an image, SIFT-based registration is performed on the basis of the previously performed image. The result of this series of processes is as shown in FIG. 2. Through this process, all fluorescein angiography images (FAG images) are repeatedly registered (registration). After performing all fluorescein angiography images (FAG images), the finally registered results are registered based on the first image.

The FAG blood vessel extraction unit 130 performs blood vessel extraction of the matched fluorescein angiography image (FAG image) based on deep learning according to the characteristics of the fluorescein angiography image (FAG image). In this case, deep learning may be a learned convolutional neural network (CNN). More specifically, the FAG blood vessel extraction unit 130 may derive a deep learning-based FAG vessel probability map (FAGVP) based on blood vessels in a fluorescein angiography image (FAG image). Although the fluorescein angiographic image (FAG image) is forcibly registered from the perspective transform through the above-described FAG matching unit 120, the fluorescence angiography image (FAG image) changes over time, optic disc, Since the properties of the original fluorescein angiogram (FAG image) such as background are retained, it is necessary to remove other properties that interfere with registration in order to match the blood vessels very closely. Accordingly, the FAG blood vessel extraction unit 130 applies a deep learning (DL) based vascular region technique with high precision to perform blood vessel extraction of the matched fluorescein angiography image (FAG image). Deep learning-based vascular zoning techniques are known techniques, and other attributes except blood vessels can be removed with a very high probability. However, databases (DRIVE, STARE, CHASE, HRF) of the Retinal Vessel Segmentation techniques in the prior art are all fundus images (FP images) and have different characteristics from fluorescein angiography images (FAG images). In particular, the characteristics of blood vessels appear very different. When the fundus image is converted to grayscale, it is expressed with a lower pixel value than the periphery of the blood vessel. On the other hand, in the fluorescein angiography image (FAG image), it is expressed with a higher pixel value than the surroundings. Therefore, the existing fundus image (FP image) database is appropriately converted so that these opposite characteristics can be reflected in the fluorescein angiography image (FAG image). By inverse transforming the pixel value of the green channel in the fundus image based on the blood vessel, it can be made to have characteristics similar to the fluorescein angiography image. Based on the converted fundus image (FP image), the same ground truth (GT) is learned, and through this, a FAG vessel probability map (FAGVP) in a fluorescein angiography image (FAG image) is derived. The results obtained at this time can be seen that the blood vessels are estimated very precisely as shown in FIG. 3. 3, the upper left is a Color Fundus Photo Image, the upper right is a Gray Fundus Photo Image, the lower left is an Inverse Gray Fundus Photo, and the lower right is a FAG Image.

The integration unit 140 integrates the results of vascular extraction of the fluorescein angiography images (FAG image) frames as an average value. More specifically, the integration unit 140 performs non-rigid registration based on the B-Spline technique in FAGVP and derives an average FAG vessel probability map (A-FAGVP) of the matched FAGVP. In more detail, the integration unit 140 performs non-rigid registration based on the free-form deformation technique of the coordinate grid represented by the B-Spline model in FAGVP and average FAG Vessel Probability of the matched FAGVP The average value of map(A-FAGVP) can be derived. SIFT-based rigid registration through the FAG blood vessel extraction unit 130 described above is not precisely matched due to non-rigid peripheral structures or non-rigid movement. Therefore, in the present invention, non-rigid registration with high precision between blood vessels is performed from FAGVP with other structures removed. Non-rigid registration (non-rigid registration) using the B-Spline registration technique that calculates information by performing iterative matching from FAGVP that is fairly similarly matched through rigid registration, and reduces errors based on this -rigid registration).

The FP blood vessel extraction unit 150 performs blood vessel extraction of the fundus image based on deep learning according to the characteristics of the fundus image. More specifically, the FP blood vessel extraction unit 150 derives a deep learning based Fundus Photo Vessel Probability map (FPVP) for matching between the fundus image (FP image) and A-FAGVP. After non-rigid registration through the integration unit 140, average FAG Vessel Probability map (A-FAGVP) is calculated by calculating the average of the same pixel position on the time axis to include all changes in blood vessels over time. And a Maximum FAG Vessel Probability map (M-FAGVP) from which the maximum value for each pixel location is extracted. A-FAGVP, calculated as the average value for each pixel along the time axis, not only reflects changes in blood vessels over time, but also effectively suppresses small noise that may occur. Since the FP vascular extraction unit 150 according to the present embodiment is a matching technique based on blood vessels, the matching technique of the fluorescence fundus angiography image (FAG image) and the fundus image (FP image) is also performed in the fundus image (FP image). It is desirable to derive FPVP using deep learning (DL). On the other hand, the vessel segmentation (vessel segmentation) technique from the fundus image (FP image), there are a number of techniques known in advance, the present invention utilizes a DL technique learned from DRIVE, a database, one of the known techniques.

The FAG-FP matching unit 160 performs matching of the fluorescence fundus angiography image (FAG image) and the fundus image (FP image) from the extracted blood vessels. The reason for this matching is that the characteristics of blood vessels are different in the fluorescein angiography image (FAG image) and the fundus image (FP image). The FAG-FP matching unit 160 matches the fluorescence angiography image (FAG image) and the fundus image (FP image) based on the FPVP derived from the fundus image using the deep learning technique as described above. To perform.

The FAG-FP matching unit 160 may proceed through two processes of matching the fluorescence fundus angiography image (FAG image) and the fundus image (FP image). First, the FAG-FP matching unit 160 uses a chamfer matching technique from blood vessels derived from FPVP and A-FAGVP to provide rigid registration between fundus image-fluorescence angiography image (FP-FAG). ), and second, the FAG-FP matching unit 160 between the final fundus image-fluorescence fundus angiography image (FP-FAG) based on the free-form deformation technique of the coordinate grid represented by the B-Spline model. Non-rigid registration is performed. More specifically, the FAG-FP matching unit 160 binarizes the A-FAGVP and FPVP derived from the above-described process based on an appropriate threshold value, and performs rigid registration using a chamfer matching technique. . Second, the FAG-FP matching unit 160 performs non-rigid registration based on the free-form deformation technique of the coordinate grid represented by the B-Spline model for precise registration between blood vessels. .

Meanwhile, according to the above, A-FAGVP is registered based on FPVP. In the present invention, since the input sources for registration are all vessel probability maps, they have a part of information about blood vessels, so instead of the above-described SIFT technique, a chamfer matching technique is used. The SIFT technique detects a feature from an image and creates a descriptor. Then, a matching point is found from a feature descriptor, and a perspective transform using RANSAC (RANDOM SAMPLE CONSENSUS) is calculated based on this. These series of processes require a very large amount of computation, and are also effective in images with many complex local features. On the other hand, the chamfer matching technique is to calculate the distance of all pixels from the target and source binary image and to calculate the similarity between the two distance images based on this. Therefore, the chamfer matching technique takes a very small amount of computation and time compared to the SIFT technique, and is also effective for registration between vessels. On the other hand, the chamfer matching (Chamfer Matching) technique according to the present embodiment is a customizing (Chamfer Matching) technique that improves the conventional chamfer matching (Chamfer Matching) technique for calculating translation, considering the rotation (rotation) do.

Finally, the FAG-FP matching unit 160 performs non-rigid registration to match the final fundus image-fluorescence angiography image (FP-FAG). According to the above, non-rigid registration is performed based on a free-form deformation technique of a coordinate grid represented by a B-Spline model between fluorescein angiography images (FP-FAGs). The FAG-FP matching unit 160 is a coordinate represented by a B-Spline model in consideration of non-rigid motion in order to finally match a precise fundus image-fluorescence angiography image (FP-FAG). Non-rigid registration based on the free-form deformation technique of the grid is performed. The two images, which have already been similarly matched by rigid registration, derive very precise registration results as shown in FIG. 4 through the free-form deformation technique of the coordinate grid represented by the B-Spline model. In FIG. 4, the result of matching before the match between the top left image of the fundus image, the top right image of the blood vessel probability map of the fundus image, and the bottom left image of the fundus image map of the fundus image (white) and the vascular probability map (blue) of the fluorescein angiography image, right The bottom is the result after matching between the fundus probability map (white) of the fundus image and the blood vessel probability map (blue) of the fluorescein angiography image.

The blood vessel division unit 170 divides the blood vessel based on the matched result. According to this embodiment, since a very precise vessel segmentation mask that can be utilized as a ground truth (GT) must be derived from the matched A-FAGVP, the blood vessel segmentation unit 170 has a probability value for each pixel of the A-FAGVP Based on this, a binary threshold segmentation technique is applied to derive a binary vessel segmentation mask and noise is removed by applying a connected component analysis technique to the derived vessel segmentation mask. do. First, the blood vessel segmentation unit 170 must fill a hole occurring in a vein region because contrast is late in a vein in a fluorescein angiography image. Therefore, the blood vessel segmentation unit 170 detects a hole inside the vein in the matched A-FAGVP and reinforces the relatively low pixel probability. Thereafter, the blood vessel division unit 170 divides the thin blood vessel and the thick blood vessel separately by applying the concept of hysteresis threshold in the matched A-FAGVP. Then, very small regions are removed from the connected components to remove noise. On the other hand, in A-FAGVP, a crevice occurs mainly in the center of a vein. This occurs due to the time difference between the contrast reaching the artery and the vein. In addition, the average probability corresponding to the blood vessel wall appears to be quite high as the contrast first reaches the blood vessel wall close to the capillaries. On the other hand, since the center of the vein fills up in the short region almost at the end, the average probability is low. Therefore, in order to obtain the vessel segmentation mask, a gap generated in a vein must be filled. More specifically, first, in order to find a gap, A-FAGVP is set to a low threshold (t=0.3) and binarization (X) is performed. Next, the gap is filled from the binarized image using a morphological closing technique. Finally, by setting the high threshold value (t=0.7), A-FAGVP is binarized (Y), and then the difference (X-Y) between the two images is calculated and the negative value is discarded. The calculated subtracted image is a representation of a vein gap as a binary image. Thereafter, the probability of the subtracted image area corresponding to A-FAGVP is reinforced. At this time, in the present invention, it is reinforced with a fixed probability (p=0.5).

Next, the blood vessel dividing unit 170 is segmented based on the matched A-FAGVP reinforced up to the gap occurring in the vein. In this case, segmentation based on the concept of hysteresis, rather than evolution from a simple threshold, is performed. First, a binary vessel mask based on coarse vessels is obtained through the first threshold. Next, the second threshold is set to a low level so that a thin vessel can be detected to derive a second binary vessel mask. Thereafter, a vessel center line mask is derived from a second binay vessel mask through a skeletonization technique. The vessel center line mask, which is represented by a thin vessel at a terminal close to 1 in pixel thickness, is merged with a primary binary vessel mask in which thick vessels are derived. Since the vessel center line mask is merged, some broken regions can be connected in the primary binary vessel mask. Finally, very small areas that are not connected to the main vessels are removed through the connected component labeling technique.

According to the present invention described above, it is possible to obtain a result of a precisely segmented blood vessel image as shown in FIG. 5, and through this result image, early diagnosis and treatment of blindness-induced disease and/or full-blood vessel disease is possible. In FIG. 5A and B, the upper left is the entire fundus image, the upper right is the entire fundus image vascular region result, the lower left is from the left optic disc center enlarged fundus image, macular center (fovea center) enlarged fundus The lower right of the image is the result of enlarged vascular region of the optic disc center from the left and the enlarged vascular region of the fovea center.

6 is a flowchart of a method of matching a fundus image and a fluorescein angiography image according to an embodiment of the present invention.

The image matching method according to the present embodiment is performed in an apparatus for matching the above-described fundus image and fluorescein angiography image.

Referring to FIG. 6, first, frames of a patient's fundus image (FP image) and a fluorescence fundus angiography image (FAG image) are acquired (S510). At this time, the fundus image (FP image) can be obtained through a fundus imaging device for the examination of eye diseases in the ophthalmology, the fluorescein angiography image (FAG image) is injected with a fluorescent substance (Fluorescein) in a vein, a fluorescent substance It can be acquired through a device that optically photographs circulating through the retinal circulatory system and displays blood vessels.

Next, each frame of the fluorescein angiography image (FAG image) is matched using a feature point matching technique (S520). More specifically, rigid registration of a fluorescence angiography image (FAG image) is performed using a Scale Invariant Feature Transform (SIFT) technique. Since a fluorescein angiography image (FAG image) has a change in blood vessels and background over time, the present invention utilizes a Scale Invariant Feature Transform (SIFT) technique to detect various features while minimizing the change in the image.

Next, blood vessel extraction of the matched fluorescein angiography image (FAG image) is performed based on deep learning according to the characteristics of the fluorescein angiography image (S530). In this case, deep learning may be a learned convolutional neural network (CNN).

Next, the blood vessel extraction results of the fluorescein angiography images (FAG image) frames are integrated as an average value (S540). More specifically, non-rigid registration based on the free-form deformation technique of the coordinate lattice represented by the B-Spline model in FAGVP and average FAG Vessel Probability map (A-FAGVP of the matched FAGVP) ).

Next, blood vessel extraction of the fundus image is performed based on deep learning according to the characteristics of the fundus image (S550 ). More specifically, a deep learning-based Fundus Photo Vessel Probability map (FPVP) is derived for matching between the fundus image (FP image) and A-FAGVP.

Next, matching of the fluorescence fundus angiography image (FAG image) and the fundus image (FP image) is performed from the extracted blood vessels (S560). The reason for this matching is that the characteristics of blood vessels are different in the fluorescein angiography image (FAG image) and the fundus image (FP image). According to the present embodiment, matching of a fluorescein angiography image (FAG image) and a fundus image (FP image) is performed based on FPVP derived from a fundus image using a deep learning technique.

Next, the blood vessel is divided based on the matched result (S570). More specifically, the vessel segmentation mask is derived from the matched A-FAGVP using the vessel segmentation mask generation technique, and the matched A reinforced to the gap occurring in the vein -Segmentation based on FAGVP.

On the other hand, the applicant conducted the following experiment to confirm the result of automatic blood vessel segmentation using matching of the fundus image and fluorescein angiography image.

The experiment was conducted in hardware consisting of Intel(R) Core(TM) i7-6850K CPU @ 3.6GHz, 32G RAM, GeForce GTX 1080TI 11G, Ubuntu 16.04 LTS OS, python 2.7 development environment. The database used in the experiment is a total of 10 FP-FAG sets, consisting of a single FP image and multiple FAG images in one set. In addition, 20 FP images and GT (ground truth) of each train and test set of the published DRIVE Database were used for deep learning. First, for non-rigid matching between FAG images, perspective transform was calculated and matched through feature detection and feature matching using SIFT technique, and RANSAC technique. Next, deep learning was used to derive the vessel probability map of all FAG images. The input image of deep learning based vessel segmentation, which has been studied a lot, is FP. Therefore, we converted the FP image of the DRIVE Database to gray scale and then inverse transformed to show characteristics similar to the FAG image. After that, I learned deep learning using the inverted FP image as input. The network model used for learning was based on SSA-Vessel Segmentation, which showed the best performance by applying the scale space theory recently. When learning, the mean from the input image was subtracted by preprocessing and divided by standard deviation. Then, when testing from the learning network, the input image is changed to the FAG image to derive the result. The derived results show that even very finely thin blood vessels appear as shown in FIG. 7.

Then, for more precise matching using the FAG Vessel Probability map derived from the learned deep learning, non-rigid matching is performed through the free-form deformation technique of the coordinate grid represented by the B-Spline model of the free form deformation series. As shown in FIG. 8, the FAG matching results derived up to the non-rigid matching can see very precise results. The map that can synthesize the entire sequence should be derived using the FAGVP map, which has been completed up to the non-rigid matching. Therefore, we derived the results including information along the time axis as the Average FAGVP map and the Maximum FAGVP map. As shown in FIG. 9, the aggregated image shows that the blood vessels are estimated very precisely and precisely.

Now, for matching between the FP image and the FAG image, the vessel probability map of the FP image is derived using deep learning. As in the case of deriving the FAGVP map, the database used DRIVE, and the network model used the same model. When learning, the input was studied using the FP image of the DRIVE Database as it was, and the average was subtracted by preprocessing and divided by standard deviation. Based on the learned results, the results as shown in Fig. 10 were derived by using our FP image as an input.

The final process of matching, FP-FAG matching, is to match A-FAGVP map to FPVP map. The first step is FP-FAG rigid registration. We have already derived the vessel probability map for each image. Therefore, a binary image was generated from the vessel probability map, and rigid body matching was performed using chamfer matching. However, as shown in FIG. 11, it is matched at a similar position, but it can be seen that an error occurs slightly.

Therefore, we perform non-rigid matching using the free-form deformation technique of the coordinate grid represented by the B-Spline model, as in the FAG matching process. As shown in Fig. 12, the results of the non-rigid matching can be seen to be much more precise and precise than the previous results.

Methods according to an embodiment of the present invention may be implemented as an application or in the form of program instructions that can be executed through various computer components to be recorded in a computer-readable recording medium. The computer-readable recording medium may include program instructions, data files, data structures, or the like alone or in combination. The program instructions recorded on the computer-readable recording medium are specially designed and configured for the present invention, and may be known and available to those skilled in the computer software field. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CD-ROMs, DVDs, and magneto-optical media such as floptical disks. media) and hardware devices specifically configured to store and execute program instructions such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes produced by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. The hardware device may be configured to operate as one or more software modules to perform processing according to the present invention, and vice versa.

While this specification includes many features, such features should not be construed as limiting the scope of the invention or the claims. Also, features described in individual embodiments of the present specification may be implemented in combination in a single embodiment. Conversely, various features described in a single embodiment of the present specification may be individually implemented in various embodiments, or may be implemented in appropriate combination.

Although the operations in the drawings have been described in a specific order, it should not be understood that such operations are performed in a specific order as shown or in a series of sequences, or that all described actions are performed to obtain a desired result. In certain circumstances, multitasking and parallel processing may be advantageous. In addition, it should be understood that the division of various system components in the above-described embodiment does not require such division in all embodiments. The above-described app components and systems may be generally implemented as a package in a single software product or multiple software products.

The present invention described above, since it is possible for a person of ordinary skill in the art to which the present invention pertains, various substitutions, modifications and changes are possible without departing from the technical spirit of the present invention. It is not limited by the drawings.

Claims (8)

1. Obtaining frames of a patient's fundus image (FP image) and a fluorescence fundus angiography image (FAG image);

   Rigid registration of each frame of the fluorescein angiography image using a feature point matching technique;

   Performing vascular extraction of the matched fluorescein angiography image (FAG image) based on deep learning according to the characteristics of the fluorescein angiography image (FAG image);

   Integrating blood vessel extraction results of fluorescein angiography (FAG image) frames into an average value;

   Performing blood vessel extraction of the fundus image based on deep learning according to the characteristics of the fundus image;

   Performing a matching of a fluorescence fundus angiography image (FAG image) and a fundus image (FP image) from the extracted blood vessels; And

   A step of dividing blood vessels based on the matched results; Automatic blood vessel segmentation method using the matching of the fundus image and the fluorescein angiogram.
2. According to claim 1,

   The step of rigid registration of each frame of the fluorescein angiography image using a feature-point matching technique is:

   And performing matching of the fluorescence angiography image (FAG image) using RANSAC (RANdom Sample Consensus) during the feature point detection, feature point descriptor extraction, and feature point matching process. Automatic blood vessel segmentation method using matching.
3. According to claim 1,

   The deep learning is a learned convolutional neural network (CNN),

   In accordance with the characteristics of the fluorescein angiography image (FAG image), performing a blood vessel extraction of the matched fluorescein angiography image (FAG image) based on deep learning,

   Automated using the matching of the fundus image and the fluorescein angiography image, comprising deriving a deep learning-based FAG vessel probability map (FAGVP) based on blood vessels in the fluorescein angiography image (FAG image). Methods of dividing blood vessels.
4. According to claim 1,

   Integrating the results of blood vessel extraction of the fluorescein angiography (FAG image) frames into an average value,

   Performs non-rigid registration based on the free-form deformation technique of the coordinate grid represented by the B-Spline model in FAGVP and averages the average FAG Vessel Probability map (A-FAGVP) of the matched FAGVP and Automatic blood vessel segmentation method using matching of fundus image and fluorescein angiography image, comprising deriving a Maximum FAG Vessel Probability map (M-FAGVP) extracting the maximum value for each pixel location.
5. According to claim 1,

   In accordance with the characteristics of the fundus image, performing blood vessel extraction of the fundus image based on deep learning,

   Automatic matching using matching of the fundus image and fluorescein angiography image, comprising deriving a deep learning-based Fundus Photo Vessel Probability map (FPVP) for matching between the fundus image (FP image) and A-FAGVP. Methods of dividing blood vessels.
6. According to claim 1,

   The step of performing the matching of the fluorescence fundus angiography image (FAG image) and the fundus image (FP image) from the extracted blood vessels,

   Performing rigid registration between fundus image-fluorescence fundus angiography image (FP-FAG) using a chamfer matching technique from blood vessels derived from FPVP and A-FAGVP; And

   And performing a non-rigid registration between the final fundus image-fluorescence fundus angiography image (FP-FAG) based on the free-form deformation technique of the coordinate grid represented by the B-Spline model. Automatic blood vessel segmentation method using matching of the fundus image and fluorescein angiography image, characterized in that.
7. According to claim 1,

   The step of dividing the blood vessel based on the matched result,

   Deriving a binary vessel segmentation mask by applying a hysteresis thresholding technique based on the probability value for each pixel of A-FAGVP;

   Removing noise by applying a connected component analysis technique to the derived vascular segmentation mask; And

   Segmentation based on the matched A-FAGVP reinforced up to the gap occurring in the vein; automatic blood vessel segmentation method using matching of the fundus image and fluorescein angiography image.
8. An image acquisition unit acquiring frames of a patient's fundus image (FP image) and fluorescence fundus angiography image (FAG image);

   A FAG matching unit for rigid registration of each frame of the fluorescein angiography image using a feature point matching technique;

   A FAG blood vessel extraction unit performing blood vessel extraction of a matched fluorescein angiography image (FAG image) based on deep learning according to the characteristics of the fluorescence angiography image (FAG image);

   An integrating unit for integrating blood vessel extraction results of fluorescein angiography (FAG image) frames into an average value;

   A FP blood vessel extraction unit performing blood vessel extraction of the fundus image based on deep learning according to characteristics of the fundus image;

   A FAG-FP matching unit performing registration of a fluorescein angiography image (FAG image) and a fundus image (FP image) from the extracted blood vessels; And

   Automatic blood vessel segmentation device using the matching of the fundus image and the fluorescein angiography image comprising a; blood vessel segmentation unit for dividing blood vessels based on the matched results.

Images