KR20210154731A - Method for detecting change of fundus for longitudinal analysis of fundusimage and device performing the same - Google Patents

Abstract

Embodiments relate to a method for detecting a change of a fundus and to a device performing the same. The method include the steps of: acquiring first and second fundus images taken respectively at first and second time points of the eye of a subject; matching the first and second fundus images based on intraocular blood vessels; detecting the region of interest in the first fundus image and the region of interest in the second fundus image from the matched image; and, based on the region of interest image in the first fundus image and the region of interest in the second fundus image, detecting a fundus change from the first time point to the second time point.

Description

Fundus change detection method for longitudinal analysis of fundus images, and apparatus for performing the same

The present application relates to a technique for detecting a fundus change, and more particularly, a method of matching fundus images taken at different time points of the same subject based on blood vessels of each fundus image and detecting fundus changes in the registered photos, and the It is about the device that does it.

In order for a specialist to determine the disease, it is necessary to examine whether there is a change from the fundus image taken in the past as well as the fundus image taken at the present time. Therefore, there is a need for a technique that assists in expert judgment by detecting even minute changes through longitudinal analysis in two fundus images taken at different points in time.

Korean Patent Registration No. 10-1761510 (2017.07.26.)

According to one aspect of the present application, an apparatus for supporting longitudinal analysis of fundus images is provided by matching fundus images taken at different time points of the same subject based on blood vessels of each fundus image and detecting fundus changes in the registered photos. can do.

In addition, it is possible to provide a computer-readable recording medium recording the fundus change detection method and instructions for performing the same.

A registration method performed by a computing device including a processor according to an aspect of the present application includes: acquiring first and second fundus images obtained by photographing an eyeball of a subject at first and second time points, respectively; registering the first and second fundus images based on intraocular blood vessels; detecting the region of interest in the first fundus image and the region of interest in the second fundus image from the matched image; and detecting a fundus change from the first viewpoint to the second viewpoint based on the region of interest image of the first fundus image and the region of interest image in the second fundus image.

In one embodiment, the step of matching the first fundus image and the second fundus image; matching the first fundus image to the second fundus image based on the feature points of the first and second fundus images; generating a first blood vessel mask representing a blood vessel region in the first fundus image; generating a second blood vessel mask representing a blood vessel region in the second fundus image; and re-registering the first fundus image and the second fundus image by deformable registration of the first blood vessel mask and the second blood vessel mask.

In an embodiment, the generating of the first blood vessel mask from the first fundus image may include extracting the shape of blood vessels of at least a portion of the first fundus image using a blood vessel extraction model based on region characteristics within the first fundus image. to do; and generating a blood vessel mask having a shape corresponding to the blood vessel region. The blood vessel extraction model is trained to output blood vessels in an input image as a vessel probability map.

In an embodiment, the generating of the first blood vessel mask from the first fundus image may include: generating a binarized blood vessel mask by binarizing an image including the extracted blood vessel shape as a blood vessel region; detecting a valley between a blood vessel and an external boundary by inverting the image including the binarized blood vessel mask; and filling the inside of the valley so that the inside of the valley represents the blood vessel.

In an embodiment, the step of matching the first fundus image to the second fundus image based on the feature points of the first and second fundus images includes at least through a feature descriptor for extracting features that are invariant to the size and rotation of the image. extracting one feature point from the first and second fundus images; sampling at least one of the feature points extracted from the first and second fundus images; and rigid body registration of the first and second fundus images based on the sampled feature points.

In an embodiment, the region of interest includes at least one of an optic nerve papilla portion and a macular portion. The detecting of the one or more regions of interest in the first fundus image may include applying the matched first and second fundus images to a first region of interest estimation model to obtain regions of interest in the matched first and second fundus images. It is also possible to detect the positions of each. The first region of interest estimation model is modeled to output at least one of an x-coordinate and a y-coordinate of the optic nerve head portion, or at least one of an x-coordinate and a y-coordinate of the macula portion.

In an embodiment, the region of interest includes an abnormal finding region. Then, the detecting of the one or more regions of interest in the first fundus image may include applying the matched first and second fundus images to a second region of interest estimation model in the matched first and second fundus images. The positions of the regions of interest may be respectively detected. The second region of interest estimation model is modeled to classify whether the input fundus image belongs to a disease image group.

In an embodiment, the region of interest includes a region corresponding to a blood vessel. Then, the location of the blood vessel region may be the location of the blood vessel mask generated in the first fundus image.

In an embodiment, the calculating of the fundus change during the first time point and the second time point may include detecting the fundus change by applying the first and second fundus images to a change detection model. The change detection model extracts a first feature set and a second feature set from the first and second fundus images, respectively, calculates a correlation map between features of the first and second feature sets, and the Based on the correlation map, it is modeled to classify whether the pair of the first and second fundus images belongs to a group having a fundus change.

In an embodiment, the calculating of the fundus change during the first time point and the second time point comprises pixel colors of the detected area of interest in the first fundus image and the detected area of interest in the second fundus image. It may include the step of detecting a change by comparing in units.

A computer-readable recording medium according to another aspect of the present application may be readable by a computing device, and may store program instructions operable by the computing device. When the program instruction is executed by the processor of the computing device, the processor performs the fundus change detection method according to the above-described embodiments.

An apparatus according to another aspect of the present application includes: an image acquisition unit for acquiring a fundus image photographed by a fundus camera - The fundus image is a first fundus image of a subject photographed at a first time point and a second time point photographed including a second fundus image of the subject; a matching unit for generating a blood vessel mask representing a blood vessel region in the fundus image, and matching first and second fundus images based on the blood vessel masks of the first and second fundus images; and a fundus change detector configured to detect a fundus change from the first time point to the second time point based on the matching result.

The device according to an aspect of the present application may detect a fundus change from a plurality of fundus images of a subject taken at different time points. As a result, the device can support longitudinal analysis of the fundus condition of the subject.

The device may register a plurality of fundus images of the subject already provided. As a result, the device may generate a wide-angle fundus image, which has a wider viewing angle than one fundus image, as a registered image.

Also, the apparatus may register fundus images having different photographing ranges. As a result, the device may also support cross-sectional analysis.

In order to generate a visualization image such as a registration image for longitudinal/transverse analysis, there is no need to use a separate device such as a laser scanning device of OPTOS. In addition, the device may generate such a visualization image without being affected by the fundus imaging environment and/or imaging device.

Effects of the present application are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description of the claims.

In order to more clearly explain the technical solutions of the embodiments of the present application or the prior art, drawings necessary for the description of the embodiments are briefly introduced below. It should be understood that the following drawings are for the purpose of explaining the embodiments of the present specification and not for the purpose of limitation. In addition, some elements to which various modifications such as exaggeration and omission have been applied may be shown in the drawings below for clarity of description.
1 is a conceptual diagram of an apparatus for performing a fundus change detection method according to embodiments of a first aspect of the present application.
2 is a flowchart of a fundus change detection method according to an embodiment of the present application.
3 is a diagram illustrating a pair of input fundus images according to an embodiment of the present application.
4 is a flowchart of a registration process according to an embodiment of the present application.
5A and 5B are diagrams illustrating matching results based on feature points of blood vessels, according to an embodiment of the present application.
6A to 6D are diagrams illustrating the results of post-processing the extracted blood vessel mask.
7A and 7B are diagrams illustrating a result of deforming and matching a blood vessel mask according to an embodiment of the present application.
8 is a conceptual network structure diagram of a change detection model for detecting a fundus change, according to an embodiment of the present application.
9 shows a schematic network architecture of an optic disc/macular detection model, according to an embodiment of the present invention.
10 is a flowchart of a fundus image registration method for generating a wide-angle fundus image according to the second aspect of the present application.
FIG. 11 is a schematic diagram of a fundus input term matching method for generating the wide-angle fundus image of FIG. 10 .

The terminology used herein is for the purpose of referring to specific embodiments only, and is not intended to limit the present application. As used herein, the singular forms also include the plural forms unless the phrases clearly indicate the opposite. The meaning of "comprising," as used herein, specifies a particular characteristic, region, integer, step, operation, element and/or component, and includes the presence or absence of another characteristic, region, integer, step, operation, element and/or component. It does not exclude additions.

Although not defined otherwise, all terms including technical and scientific terms used herein have the same meaning as commonly understood by a person of ordinary skill in the art to which this application belongs. Commonly used terms defined in the dictionary are additionally interpreted as having a meaning consistent with the related technical literature and the presently disclosed content, and unless defined, are not interpreted in an ideal or very formal meaning.

Hereinafter, embodiments of the present application will be described in detail with reference to the drawings.

1 is a conceptual diagram of an apparatus for performing a fundus change detection method according to embodiments of a first aspect of the present application.

Referring to FIG. 1 , the device 10 is configured to acquire a fundus image from a fundus camera (not shown). The device 10 includes an image acquisition unit 100; matching unit 300; and a fundus change detection unit 500 .

The device 10 according to embodiments may be entirely hardware, entirely software, or may have aspects that are partly hardware and partly software. For example, the device may collectively refer to hardware equipped with data processing capability and operating software for driving the same. As used herein, terms such as “unit”, “module,” “device,” or “system” are intended to refer to a combination of hardware and software run by the hardware. For example, the hardware may be a data processing device including a central processing unit (CPU), a graphic processing unit (GPU), or another processor. In addition, software may refer to a running process, an object, an executable file, a thread of execution, a program, and the like.

The image acquisition unit 100 acquires a fundus image (or fundus photo) of the subject through a fundus camera photographing the subject's eyeball. In certain embodiments, the fundus camera transmits photographing data of the subject's eye to the device 10 through wired/wireless electrical communication. In other specific embodiments, a fundus camera may be integrated into the device 10 .

The fundus camera includes various fundus cameras capable of acquiring fundus images. For example, the fundus camera may include a mydriatic fundus camera, a non-mydriatic fundus camera, an OCT type fundus camera, and the like. The fundus image includes a retinal fundus image.

In an embodiment, the image acquisition unit 100 may acquire a first fundus image captured at a first time point and a second fundus image captured at a second time point for the same subject. For example, the second time point may be a time point that has elapsed from the first time point.

In certain embodiments according to the first aspect of the present application, the first fundus image and the second fundus image may be fundus images obtained by photographing a specific sub-region among the entire eye region of the subject.

The image acquisition unit 100 may acquire a binocular fundus image or a monocular fundus image of the subject as a first fundus image or a second fundus image. For example, the image acquisition unit 100 may acquire a pair of first binocular fundus images. The image acquisition unit 100 may acquire a first monocular fundus image. The image acquisition unit 100 may acquire a pair of second binocular fundus images. The image acquisition unit 100 may acquire a second monocular fundus image.

Also, the image acquisition unit 10 may acquire data related to the fundus image. For example, the data related to the fundus image may include subject identification information (eg, name, identity information, identifier, etc.) capable of identifying the subject of the fundus image, a photographing time, and/or information related to a photographing device. . In addition, label data indicating whether the obtained fundus image is the fundus image of the left eye or the fundus image of the right eye may be further acquired.

The image acquisition unit 100 provides the fundus image and/or data related to the fundus image of the subject to the matching unit 300 . Also, the image acquisition unit 100 may provide data related to the corresponding image while providing the fundus image.

In an embodiment, the image acquisition unit 100 may provide the matching unit 300 with a first fundus image captured at a first time point and a second fundus image captured at a second time point for the same subject.

The matching unit 300 generates a registered image by registering a plurality of fundus images of the subject. For example, the matching unit 300 may register the first fundus image with the second fundus image.

In an embodiment, the matching unit 300 may register the first fundus image and the second fundus image based on the blood vessel regions of the first and second fundus images. To this end, the matching unit 300 may extract a blood vessel region from the fundus image. In certain embodiments, the registration unit 300 may generate a vascular mask for registration.

The fundus change detection unit 500 detects a fundus change using the matched image. The fundus change may include whether a fundus change occurs, an amount of fundus change, and the like. The fundus change amount may include, for example, an area change, a color change, and the like.

In an embodiment, the fundus change detector 500 may detect a fundus change based on the ROI in the first fundus image and the ROI in the second fundus image. Each ROI is determined based on the matched image. For example, the fundus change detector 500 may detect the ROI of the first fundus image based on the coordinates of the matched image.

The fundus change detector 500 may detect the ROI by generating an ROI image of the fundus image from a portion of the fundus image. Then, the fundus change detection unit 500 may detect the fundus change by comparing two ROI images.

When the pair of the first pair of binocular fundus photos taken at the first time point and the second pair of binocular fundus photos captured at the second time point are used, the apparatus 10 may perform the above-described operation for each corresponding eye. For example, the device 10 detects a fundus change by matching the first fundus photo of the left eye and the second fundus photo of the left eye, or matching the first fundus photo of the right eye and the second fundus photo of the right eye to fundus Changes can also be detected.

The operation of the matching unit 300 and the fundus change detecting unit 500 will be described in more detail based on FIG. 2 and the like below.

The fundus change detection method according to another aspect of the present application may be performed by a computing device including a processor (eg, the device 10 of FIG. 1 ). Hereinafter, for clarity of explanation, the present application will be described in more detail based on embodiments performed by the apparatus 10 of FIG. 1 .

2 is a flowchart of a fundus change detection method according to an embodiment of the present application.

Referring to FIG. 2 , the method for detecting fundus changes includes: acquiring a fundus image obtained by photographing the subject's eye (S100)). The step (S100)) includes acquiring a first fundus image obtained by photographing the subject's eyeball at a first time point. The step (S100)) includes acquiring a second fundus image by photographing the subject's eyeball at a second time point.

In addition, some or all of the photographing area may be shared in the first fundus image and the second fundus image. The first fundus image and the second fundus image are not necessarily required to have identical photographing regions.

3 is a diagram illustrating a pair of input fundus images according to an embodiment of the present application.

For example, as shown in the upper part of FIG. 3 , one fundus image (eg, the first fundus image) may be a fundus image obtained by focusing an optic disc part. As shown in the lower part of FIG. 3 , another fundus image (eg, the second fundus image) may be a fundus image obtained by focusing a foveolar part. Also, vice versa.

Referring back to FIG. 2 , the method for detecting fundus changes includes: matching the first fundus image with the second fundus image based on feature points of the first fundus image and the second fundus image ( S300 ).

4 is a flowchart of a registration process according to an embodiment of the present application.

Referring to FIG. 4 , the step S300 may include matching the first fundus image and the second fundus image through feature point-based matching in the fundus image ( S310 ).

Step S310 may include extracting feature points in the fundus image. The feature point includes a point that is easily identified while being distinguished from the surrounding background in matching the image.

In an embodiment, the feature point may include a feature point related to a blood vessel. For example, a feature point included in the shape of a blood vessel or an edge may be included. In some embodiments, the feature points in the fundus image may include extracted blood vessels, for example, feature points of a blood vessel mask.

In step S310, feature points may be extracted from the input image (eg, the first blood vessel mask) through various feature extraction algorithms based on a feature descriptor. The feature extraction algorithm may include, for example, a Scale Invariaant Feature Trnasform (SIFT) that extracts a point having a maximum cornerity on an image axis and/or a scale axis based on Difference of Gaussian (DoG), but is limited thereto. it doesn't happen A feature point set including a plurality of feature points may be extracted in step S310 .

In step S310, the feature point-based matching may be performed in the pixel domain.

In the step S310, the first fundus image and the second fundus image by rigid registration of the first feature point set extracted from the first fundus image and the second feature point set extracted from the first fundus image, respectively You can also match images.

In an embodiment, the step S310 may include matching the first feature point set and the second feature point set using a perspective transformation matrix. In some embodiments, the step S310 may further use the RANSAC algorithm to match the first set of key points and the second set of key points. For example, in the process of matching the first feature point set and the second feature point set using the perspective transformation matrix, a value that maximizes the degree of matching between the feature points of each set is calculated through the RANSAC algorithm to perform feature point based matching. may be

5A and 5B are diagrams illustrating matching results based on feature points of blood vessels, according to an embodiment of the present application.

Referring to FIGS. 5A and 5B , the first fundus image and the second fundus image are matched based on the feature points of blood vessels ( S310 ).

Also, referring to FIG. 5B , by matching the feature points of blood vessels, a registered image in which the first fundus image and the second fundus image are registered and visualized as a result of FIG. 5A may be obtained ( S310 ).

Referring back to FIG. 4 , the step ( S300 ) includes: extracting a blood vessel region from the fundus image ( S330 ). The step (S330) may include extracting a blood vessel region from the first fundus image; and extracting the blood vessel region from the second fundus image. In step S330 , a mask having a shape corresponding to the blood vessel region (hereinafter, “vessel mask”) may be created in order to extract blood vessels. The blood vessel mask may include an outline of at least some blood vessels in the fundus image. In addition, the blood vessel mask may be configured to visually distinguish the outside of the blood vessel and the blood vessel region.

In an embodiment, the blood vessel region is extracted from the fundus image using a machine learning model learned based on region characteristics in the fundus image. The deep learning model is trained to output blood vessels in the inner region of the fundus image as input as a Vessel Probaibility Map.

The blood vessel extraction model is a machine learning model having a deep learning structure, comprising: a feature extraction layer; and a classification layer.

The feature extraction layer is configured to extract features used to classify a portion of an input image into a blood vessel class. When the blood vessel extraction model is a Convolutional Neural Network (CNN)-based model, the feature extraction layer may include a convolutional layer and/or a pooling layer.

The classification layer includes a classification layer configured to calculate a probability that a portion (eg, a pixel or a set of pixels) in an image is classified into a blood vessel class based on the extracted features. Then, for each pixel in the fundus image, a blood vessel probability map including the probability that the corresponding pixel is a blood vessel is calculated.

A blood vessel shape is determined based on the blood vessel probability map calculated in step S330 . In certain embodiments, a mask corresponding to a blood vessel shape based on the calculated blood vessel probability map may be generated as an eye blood vessel mask.

In some embodiments, the classification layer may be further configured to classify into a vascular class and a non-vascular class based on the calculated probability. In this case, an image of a blood vessel region including pixels classified into blood vessel classes may be generated in step S330 .

The blood vessel extraction model may have various structures based on a Convolutional Neural Network (CNN). The blood vessel extraction model includes: a layer for extracting features from an input image; and a classification layer configured to calculate a probability that a portion (eg, a pixel) in an image is classified into a blood vessel class based on the extracted features. The blood vessel extraction model may include, for example, an SSANet, or a Rential SSANet-based structure, but is not limited thereto.

The blood vessel extraction model is trained to calculate a probability that a portion (eg, a pixel) in an image belongs to a blood vessel group by using a training sample including a plurality of training fundus images. The blood vessel extraction model may be a model learned by various learning processes having the purpose of class classification or calculating a probability value for class classification of an input image part (eg, pixel).

For example, parameters of a blood vessel extraction model having a fundus image as an input may be updated and learned in a direction in which the cost function of the corresponding model is minimized. Here, the cost function represents the difference between the result value calculated by the model and the actual result value. Learning these parameters is commonly referred to as optimization. The parameter optimization may include, for example, Adaptive Moment Estimation (ADAM), Momentum, Nesterov Accelerated Gradient (NAG), Adaptive Gradient (Adagrad), RMSProp, and various gradient descent schemes.

Through this learning process, the blood vessel extraction model is trained to enhance the ability to calculate a higher probability value for actual blood vessels and a lower probability value for non-vascular vessels in the fundus image. In addition, due to the enhancement of the probability value calculation capability, the blood vessel region identification capability may also be enhanced.

A blood vessel shape is determined based on the blood vessel probability map extracted in step S330. In certain embodiments, a mask corresponding to a blood vessel shape based on the extracted blood vessel probability map may be generated as an eye blood vessel mask.

Also, in an embodiment, step S330 may further include post-processing the shape of the blood vessel based on the extracted blood vessel probability map. Then, it is possible to generate a more accurate and improved blood vessel mask than the blood vessel mask based only on the calculated blood vessel probability map. Post-processing includes binarization and/or redefining the boundaries of vessel shape.

6A to 6D are diagrams illustrating the results of post-processing the extracted blood vessel mask.

In the exemplary embodiment, the post-processing step may include: generating a binarized fundus mask by applying a hysteresis thresholding to the input image and binarizing it.

As shown in FIG. 6A , an image of a blood vessel region based on the blood vessel probability map calculated in step S330 may be generated. The double image technique is applied to the image of FIG. 6A .

The double threshold technique is a technique for determining target data using an upper threshold and a lower threshold. In order to prevent the error that filamentray vessels are cut off in the image, the blood vessel region of the input image is binarized based on the preset first threshold value (τl) and the second threshold value (τh) in step S300 . The second threshold value τh may be a higher threshold value than the first threshold value τl. Pixels exceeding the second threshold value τh are used as seeds for regions in which pixels with a higher probability than the lower threshold value τl grow. The first threshold value and the second threshold value may be empirical values, such as, for example, τ1 = 0.1, τh = 0.75, and the like.

By this double-threshold technique, the boundary of the vessel shape is grown as a region of the pixel having a higher probability than the first threshold value. A binarized vascular mask may be generated based on the growth results.

In addition, the post-processing step may include: fine-tuning the blood vessel mask to align the vessel boundary with respect to the image gradient of the fundus image.

To this end, the post-processing step may include: detecting a valley between a blood vessel and an external boundary in a binarized image; and filling the inside of the valley. If the binarized image is inverted, the region between the blood vessel and the outer boundary is more clearly distinguished. For example, as shown in FIG. 6C , a plurality of regions (eg, a region between a blood vessel and an external boundary indicating a blood vessel inner, outer boundary, and/or blood vessel thickness, etc.) that were identically expressed as a blood vessel region in FIG. 6B . are more clearly distinguished. In the inverted binarized image, a valley region between the blood vessel and the outer boundary is detected, and image processing is performed to fill the inside of the valley region. To do this, in the initial blood vessel mask shown in the partially enlarged view of FIG. 6A, the error part that was connected between the blood vessels is removed as shown in FIG. may be The process of filling the valley inside in step S330 may be performed, for example, through image erosion processing.

When the processes of step S330 are applied to the first fundus image and/or the second fundus image, the vascular mask of the first fundus image and/or the second fundus image is obtained (S330).

Referring back to FIG. 5 , the step ( S300 ) may include a step ( S350 ) of matching the first fundus image and the second fundus image by deforming and matching the first blood vessel mask and the second blood vessel mask. In certain embodiments, the step S350 may be performed after the feature point-based matching of the step S310.

Deformable registration is a non-linear form of registration in which a registration object can be freely deformed. For example, a matching target of a straight line may be deformable in a shape such as a curve, a rectangle, or a circle.

The transformation matching algorithm performed in step S350 may include, for example, a B-Spline algorithm. The B-Spline algorithm is a matching algorithm that iteratively optimizes in the direction of increasing pixel similarity between two images. An optimized matching result is calculated by determining the parameter that has the highest similarity.

In an embodiment, in order to register the first and second fundus images in operation S350, the first blood vessel mask and the second blood vessel mask may be deformed and matched. As described above in step S200 , when the first blood vessel mask and the second blood vessel mask are generated by the blood vessel extraction model, the deformation registration is performed based on the more precisely extracted blood vessel region.

For example, if the B-Spline algorithm is applied using the first blood vessel mask as the source shape and the second blood vessel mask as the target shape, one or more control points of the first blood vessel mask are extracted and the shape of the first blood vessel mask is set to the second shape. It can also be modified to have an optimization curve that maximizes the pixel similarity to the blood vessel mask. Then, the first blood vessel mask and the second blood vessel mask may be re-registered so that errors present in the feature point-based registration result are finely adjusted.

7A and 7B are diagrams illustrating a result of deforming and matching a blood vessel mask according to an embodiment of the present application.

Referring to FIG. 7A , a more sophisticated matching result may be obtained by performing deformation matching of the blood vessel mask in step S300 to fine-tune errors existing in the feature point-based matching result.

When only rigid body matching is performed, a portion of blood vessels may not be accurately matched (S310). For example, as shown in FIG. 5A , when only feature point-based matching is performed, as the distance from the optic nerve head increases, the matching may not be relatively precise. However, as shown in FIG. 7A , when deformation matching is further applied, the first fundus image and the second fundus image may be non-rigidly registered to obtain a precise matching result ( S350 ).

Also, as shown in FIG. 7B , it is also possible to obtain a registered image that visualizes the result of the registration of the first fundus image and the second fundus image so as to be finely adjusted.

A more accurate registered image may be obtained through two-stage registration consisting of such feature point-based registration and deformation registration.

The method for detecting fundus changes includes: comparing a first fundus image with a second fundus image based on a matched image, and detecting a fundus change from a first time point to a second time point ( S500 ).

Referring back to FIG. 2 , the method for detecting a fundus change may include: before step S500 , detecting a region of interest from the matched image ( S400 ).

The region of interest indicates a portion in which a significant fundus change is detected in the fundus image. In an embodiment, the region of interest may include an optic disc portion, a macular portion, and/or a vascular portion. Also, in some embodiments, the region of interest may further include an abnormal finding portion.

When the ROI includes the optic nerve head and/or the macula, the step S400 includes applying the matched image of the step S300 to the first ROI estimation model.

The first region of interest estimation model is configured to extract a feature from an input image to detect an object of interest. The first region of interest estimation model may be a deep learning model including a CNN-based structure.

In some embodiments, the first ROI estimation model may be a machine learning model including a residual block. The residual block has a structure in which a kind of skip connection (or referred to as a shortcut connection) is added to an existing stacked structure in a CNN-based model having a deep structure. . In the residual block, if the input is x, the final data to be learned is H(x). The H(x) is F(x) + x, which may be expressed as F(x) = H(x) - x. Therefore, the residual block learns the residual between the output H(x) and the input accumulated through the weight layer, and as a result, the result F(x) originally intended for learning can be obtained. As such, learning of a CNN model having a residual learning block structure that only needs to learn the residual may be referred to as residual learning.

The first region of interest estimation model may be, for example, a model including a ResNet-based structure, but is not limited thereto.

In an embodiment, the first region of interest estimation model is a model trained to output one or more of the x-coordinate and the y-coordinate of the optic disc part and/or at least one of the x-coordinate and the y-coordinate of the macular part in the matched image may be For example, the first region of interest estimation model may be configured to calculate a maximum of four coordinates from one input image (eg, a registered image).

The first region of interest estimation model may determine the optic nerve head portion from the input image based on the coordinates of the optic nerve head portion. Also, the first region of interest estimation model may determine the macula portion from the input image based on the coordinates of the macula portion.

Then, the position of the region of interest in the first fundus image and the position of the region of interest in the second fundus image are output based on the registered image, and eventually, the region of interest in the first fundus image and the region of interest in the second fundus image based on the registered image. This may be detected.

Since the learning process of the first ROI estimation model is similar to the above-described blood vessel extraction model, a detailed description thereof will be omitted.

When the region of interest includes a blood vessel portion, determining the blood vessel mask extracted in step S200 as the region of interest is included. By expressing the blood vessel mask extracted in step S200 as coordinates of the matched image, the blood vessel region of the ROI may be detected in the first fundus image and the second fundus image based on the matched image.

When the region of interest includes the region of abnormality, the step of applying the matched image of step S300 to the second region of interest estimation model is included.

The abnormal finding region includes a candidate region in which a change may occur when an eye-related disease occurs. The abnormal finding region may include, for example, a position of an image region activated when a disease occurs. That is, the abnormal finding region does not refer to the region of the abnormal finding target in which an abnormality actually appears in the captured fundus image.

In step S400 , the positions of the regions of interest may be respectively detected in the matched first and second fundus images by applying the matched first and second fundus images to a second region of interest estimation model.

The second region of interest estimation model is a machine learning model trained to classify whether an input fundus image belongs to a disease image group.

In an embodiment, the second region of interest estimation model performs global average pooling (GAP) pooling processing on a feature map through a convolutional layer, multiplies the feature map by a weight for each channel, and sums the feature map (Class Activation Map) It may be configured to calculate . Here, the channel is a factor that determines the input size of the fully connected layer.

Referring back to FIG. 2 , the method for detecting a fundus change includes: detecting a fundus change based on the ROI of the first and second fundus regions ( S500 ). When a fundus change is detected in step S500, fundus change information may be provided.

The change of the fundus includes whether the state of the fundus changes during the first time point and the second time point. In some embodiments, the fundus change may further include a fundus change amount.

In an embodiment, step S500 may include detecting a fundus change by applying the first and second fundus images to a change detection model. In certain embodiments, images of the ROI of the first and second fundus images may be used as input images for detecting fundus change.

In an embodiment, the change detection model may be configured to have the structure and function of a binary classification model (BCM) for classifying whether a pair of input images belongs to a conditionally positive group or a conditionally negative group. The input is a pair of fundus images taken at two viewpoints for each region of interest, and the output indicates the presence or absence of change.

8 is a conceptual network structure diagram of a change detection model for detecting a fundus change according to an embodiment of the present application.

Referring to FIG. 8 , the change detection model includes: a feature extraction layer; and a classification layer.

The feature extraction layer is configured to extract the first feature set and the second feature set from the pair of region-of-interest images of the first and second fundus images, respectively. When the change detection model has a CNN-based structure, the feature extraction layer may include one or more convolution layers. In addition, the feature extraction layer may further include one or more pooling layers.

The first set of features is a feature extracted from an image of a region of interest of the first fundus image, and the feature includes an image feature related to classifying whether a pair of input images belongs to a group having a fundus change.

In addition, the change detection model may further include: a correlation map generation layer, which calculates an inter-feature correlation map from the set of the first and second features.

The correlation map layer is configured to calculate a geometric correlation map from the first feature set and the second feature set extracted from the feature extraction layer. A correlation map of the first feature set and the second feature set is input to the classification layer.

The classification layer is configured to classify whether the pair of the first and second fundus images belongs to a group having a fundus change based on the extracted features. The classification layer may include a CNN layer, a fully connected layer, and/or a probability layer. The CNN layer and the fully connected layer are configured to estimate the presence/absence of fundus change from the correlation map.

The probability layer is configured to calculate a probability that the pair of input images belongs to the first group indicating that there is a change based on an output value output from a previous layer (eg, a fully connected layer).

In some embodiments, the change detection model may be further configured to perform activation processing before applying the output value of the previous layer to the probability layer. For example, the change detection model may apply the sigmoid function to the result that has passed through the fully connected layer, and then input the result to the probability layer.

The change detection model may classify a pair of input images into a first group or a second group based on the probability value. The change detection model may perform group classification according to a probability value based on a probability threshold. The probability threshold is a parameter determined by learning. For example, by setting 0.5 as an initial value, it may be adjusted to 0.5 or more or less according to the training data.

As such, when the pair of input images is classified into the first group in step S500, the fundus change is detected as having a fundus change. On the other hand, when the pair of input images is classified into the second group in step S500, there is no fundus change and no fundus change is detected.

In addition, in step S500, a fundus change may be detected through an image processing algorithm.

In another embodiment, the method may include detecting a change by comparing colors of the detected region of interest in the first fundus image and the detected region of interest in the second fundus image in units of pixels.

In operation S500 , the RGB color of the ROI may be normalized to detect a pixel-by-pixel change between the pixels of the first ROI and the second ROI corresponding to each other based on the matched image. For RGB normalization, for example, the first region of interest image and the second region of interest image may be normalized to the same RGB color through a histogram equalization algorithm.

In some embodiments, operation S500 may further include removing illumination through gamma correction after normalization.

Then, the pixel-wise change may be calculated using the two normalized images (eg, the first region of interest image and the second region of interest image) in operation S500 .

The change in units of pixels may include the presence or absence of change and/or the amount of change. The change amount may include a change area or a change color.

In this way, the fundus change detection in step S500 may be performed using an image processing technique and/or a deep learning model.

In addition, different progress may be made for each fundus change item. For example, in step S500 , the presence or absence of a fundus change may be performed through a change detection model, and the amount of fundus change may be performed through an image processing technique.

It will be apparent to one of ordinary skill in the art that the apparatus 10 may include other components not described herein. For example, the device 10 may include other hardware elements necessary for the operation described herein, including a network interface, an input device for data entry, and an output device for display, printing, or other data presentation. may be

As such, in certain embodiments according to the first aspect of the present application, the apparatus 10 obtains fundus change information of the specific sub-region using the first fundus image and the second fundus image obtained by photographing the specific sub-region. can also be detected.

Meanwhile, in other specific embodiments according to the second aspect of the present application, the apparatus 10 has a wide angle having a wider imaging range than the fundus images of the first side (ie, fundus images of a specific sub-region). Fundus change information in a wider ocular region may be detected using fundus images.

The apparatus 10 may be further configured to generate a wide-angle fundus image before detecting fundus change information. In certain embodiments, the apparatus 10 may include: a first wide-angle fundus image obtained by photographing a plurality of sub-regions at a first viewpoint, obtained by synthesizing a plurality of fundus images at a first viewpoint; and generating a second wide-angle fundus image obtained by photographing the plurality of sub-regions at a second viewpoint by synthesizing the plurality of fundus images at a second viewpoint; In addition, a wider range of fundus changes may be detected based on the first wide-angle fundus image and the second wide-angle fundus image.

Referring back to FIG. 1 , the device 10 according to the second aspect includes a classification unit 200 ; and/or may further include a synthesizing unit 400 .

The image acquisition unit 100 acquires a fundus image set including a plurality of fundus images obtained by photographing each sub-region of the entire fundus region of the target at the same viewpoint. The image acquisition unit 100 is a first fundus image (ie, 1-1 fundus image) in which the first sub-region of the entire fundus region of the object is photographed at the first time point, and the second sub-region is captured A first fundus image set including 1 fundus image (ie, 1-2 fundus images) and a first fundus image (ie, 1-nth fundus image) obtained by photographing the nth sub-region is acquired. Similarly, the image acquisition unit 100 acquires a second fundus image set taken at the second time point. In certain embodiments, the first fundus image set and the second fundus image set each include a fundus image to which each sub-region corresponds. For example, each of the first fundus image set and the second fundus image set includes fundus images (ie, the 1-1 fundus image and the 2-1 fundus image) corresponding to the first sub-region.

In certain embodiments, the plurality of fundus images constituting the first or second fundus image set includes a first group of fundus images including both an optic nerve head and a fovea in an image frame, and a second group including only the optic nerve head within an image frame. of the fundus image, the fundus image of the third group including only the fovea in the image frame, and the fundus image of the fourth group that does not include both the optic nerve papilla and the fovea in the image frame.

The classifier 200 classifies the plurality of fundus images into first to fourth groups by detecting the optic nerve papilla or fovea in each image frame of the plurality of fundus images captured at the same time point. The classifier 200 detects an optic disc or a fovea in each image frame in the first or second fundus image set.

In an embodiment, the classifier 200 may detect the optic nerve head or fovea in the image frame using a pre-trained optic nerve head/macular detection model. Also, the classifier 200 may calculate the two-dimensional position of the detected object on the image frame. The classification unit 200 calculates the two-dimensional positions of the optic nerve papilla and fovea as pixel coordinates on the image frame.

9 shows a schematic network architecture of an optic disc/macular detection model, according to an embodiment of the present invention.

Referring to FIG. 9 , the optic nerve papilla/macular detection model (hereinafter, “detection model”) includes a shared convolutional network; regional network; It includes an object detection network.

The shared convolutional network includes a plurality of convolutional layers, the output of which is shared by a localized network and an object detection network.

The convolution feature map output by the last shared convolution layer of the shared convolutional network is input to the region-specific network.

The regional designation network generates an intermediate feature map by applying a sliding window method to the input convolutional feature map, and predicts a region of interest in which a candidate object is likely to be located based on the intermediate feature map.

The sliding method is a method of searching for a desired object for all ranges of input data while sliding a window on the input data. This window slides along the input convolutional feature map. The feature window at each sliding position is mapped to a lower-dimensional feature. Then, the region-specific network generates an intermediate feature map of a lower dimension than the input convolutional feature map. The regional designation network predicts a region of interest based on the intermediate feature map.

In an embodiment, the region-specific network may predict the ROI so that the candidate object (ie, the optic disc or fovea) is located in the center of the bounding box. Then, the position of the bounding box corresponds to the position of the center, that is, the position of the candidate object.

In some embodiments, the region-specific network may predict a region of interest using a bounding box having a fixed size in advance.

The size of the optic disc and fovea generally tends to be constant. The size of the bounding box may be preset to a value obtained based on this tendency.

As shown in FIG. 9 , the regional-specific network predicts a region of interest that is likely to include the optic disc and fovea.

The object detection network includes a pooling layer and a fully connected layer.

The convolution feature map output by the last shared convolution layer of the shared convolutional network is also input to the object detection network. The input convolutional feature map is input to the pooling layer.

The pooling layer performs a pooling process on the input data based on the ROI prediction result by the regional designation network. The pooling layer is configured to output a feature vector when a feature map is input. Then, the feature vector for the region of interest in the image frame initially input to the detection model of FIG. 3 is input to the fully connected layer.

In some embodiments, when the size of the bounding box is fixed, the size of the intermediate feature map of the searched ROI is also fixed. Then, the object detection network acquires a feature vector of a fixed size and inputs it to the fully connected layer.

The fully connected layer is configured to predict the position of the optic nerve head and fovea by calculating the probability that the optic nerve head and fovea exist in the region of interest.

In an embodiment, the object detection network may output a result including the two-dimensional position and size of the bounding box on the input image frame, the class of each object (ie, the optic disc or fovea), and the confidence score for each class have. The confidence score may be a probability value or another conversion value based thereon.

The detection model detects the optic nerve head and fovea based on this result. The detection model detects the optic nerve head and fovea by selecting the region of interest having the highest value for the corresponding reliability for each class.

In one embodiment, the detection model selects the region of interest having the highest value for the corresponding reliability for each class, and when the reliability of the class of the selected region of interest is higher than a preset reliability threshold, the class of the selected region of interest It is also possible to detect an object of the optic nerve disc or fovea.

When the reliability is a probability value, the reliability threshold may be, for example, 0.9, but is not limited thereto.

The detection model is trained using a training data set including some or all of the first to fourth groups. The detection model is trained to have a parameter value for minimizing the difference between the actual value and the predicted value of the training fundus image.

The classification unit 200 automatically classifies the plurality of fundus images into first to fourth groups according to whether the optic nerve papilla or fovea is detected. The operation of the classification unit 200 will be described in more detail below with reference to FIG. 9 and the like.

The matching unit 300 of the apparatus 10 may extract at least one feature point for each fundus image in each set in order to generate a wide-angle fundus image. In addition, the matching unit 300 extracts a blood vessel region for each of a plurality of fundus images in order to match different fundus images. Here, each fundus image in the set and different fundus images mean images in which the captured sub-regions do not match. For example, fundus images in which the photographed sub-regions partially overlap are treated as individual fundus images in the set and different fundus images.

The matching unit 400 may generate a blood vessel mask indicating at least a portion of a blood vessel for each fundus image included in the first or second fundus image set. Since the operations of the matching unit 400 have been described above, a detailed description thereof will be omitted.

The synthesizing unit 500 generates a wide-angle fundus image at the specific time by synthesizing fundus images captured at the same specific time point and photographing different sub-regions. The operation of generating the wide-angle fundus image of the synthesizing unit 500 will be described in more detail below with reference to FIG. 10 and the like.

FIG. 10 is a flowchart of a fundus image registration method for generating a wide-angle fundus image according to the second aspect of the present application, and FIG. 11 is a schematic diagram of a fundus input term matching method for generating the wide-angle fundus image of FIG. 10 . 11B is a schematic partial enlarged view of step S1300 of FIG. 11A .

The fundus domain matching method (hereinafter, “matching method”) for generating the wide-angle fundus image includes: acquiring a plurality of fundus images (eg, by the image acquisition unit 100 ) ( S1100 ). Each of the plurality of fundus images may include an optic nerve disc and/or a macular pericentre, or may not include all of them.

The registration method includes: (eg, by the classification unit 200 ), for each of the plurality of fundus images, detecting the optic nerve papilla and/or fovea and classifying them into a plurality of groups ( S1210 ). The plurality of fundus images in step S1210 may be a first fundus image set or a second fundus image set.

In an embodiment, the optic nerve head and/or fovea may be detected in the image frame of the individual fundus images in each set using the pre-trained optic nerve head/macular detection model (S1210). The detection model of step S1210 and the detection operation of the optic nerve papilla and fovea using the same have been described above with reference to FIG. 9 , and detailed description thereof will be omitted.

In certain embodiments, the plurality of groups for the first or second fundus image set includes first to fourth groups. The step (S1210) includes: classifying the fundus image in which both the optic nerve papilla and the fovea are detected into a first group; classifying the fundus image in which only the optic nerve head is detected into a second group; classifying the fundus image in which only the fovea is detected into a third group; and/or classifying a fundus image in which neither the optic nerve papilla nor the fovea is detected into a fourth group. For example, in step S1210 , the plurality of images of step S1100 may be classified into first to fourth groups, respectively.

In addition, the matching method includes: (eg, by the classification unit 200 ) allocating a matching rank among fundus images within a group for each group ( S1230 ). The step (S1230) may include: allocating a matching priority to fundus images included in each of the partial or all groups with respect to some or all of the first to fourth groups; and/or allocating a matching order within the fourth group to the fundus images of the fourth group.

Hereinafter, for clarity of explanation, the apparatus 10 of the present application will be described in more detail as embodiments in which the first to fourth groups in the first fundus image set are classified into 1-1 to 1-4 groups. .

When a matching rank is assigned, the order of fundus images to be matched depends on the matching rank. For example, fundus images belonging to the 1-1 group are sequentially matched according to the matching order among fundus images in the 1-1 group. Alternatively, fundus images belonging to groups 1-4 may be sequentially registered according to the matching order among fundus images in groups 1-4.

In one embodiment, the step (S1230) may include: selecting each reference fundus image for each group with respect to at least one of the group 1-1 to group 1-3; and allocating a matching rank within the corresponding group to the fundus image for each corresponding group using the reference fundus image. For example, a reference fundus image for each of the group 1-1 to 1-3 is selected, and a matching rank within the group is assigned using the reference fundus image for each group (S1230).

The fovea and the optic disc have close anatomical characteristics. For this reason, since the fundus image in which the fovea is located in the central portion of the image frame generally also includes the optic disc, it can be treated as belonging to the 1-1 group.

In an embodiment, the reference fundus image of the 1-1 group is a fundus image having a minimum distance between the position of the optic nerve head detected for each fundus image among the fundus images belonging to the 1-1 group and the center position of the image frame. may be selected (S1230). When the reference fundus image of the 1-1 group that may be used as the initial image frame of the wide-angle fundus image is selected in this way, as described above, it is advantageous to start with the center of the inside of the eyeball as a reference in generating the wide-angle fundus image, and This is because it is possible to have all the conveniences of visual discrimination at the same time.

In addition, the registration order within the 1-1 group for the fundus images of the 1-1 group is based on the distance between the optic nerve head of the reference fundus image of the 1-1 group and another fundus image in the 1-1 group. may be assigned. This is because, as described above, the optic nerve discs are more easily distinguished visually. As the distance between the optic nerve head of another fundus image and the optic nerve head of the reference fundus image is closer, a higher matching priority is assigned (S1230).

The distance is, for example, ||p ^o _i -p ^o _ref || It may be calculated by Equation ( _2). Here, i is an index within the group, p ^o _i represents the pixel coordinates on the image frame of the optic nerve head of the i-th fundus image of the group 1-1, and p ^o _ref is the position of the optic nerve head on the image frame of the reference fundus image. Represents pixel coordinates.

Since the fundus images of the group 1-1 and group 1-2 include the optic nerve papilla, the reference fundus image of the group 1-1 may be used as the reference fundus image of the group 1-2. That is, when the reference fundus image of the 1-1 group is selected, the reference fundus image of the 1-2 group is also automatically selected.

The matching order within the 1-2 group for the fundus images of the 1-2 group is assigned based on the distance between the optic nerve head of the reference fundus image of the 1-2 group and another fundus image in the 1-2 group (S1230).

The reference fundus image of the group 1-3 is also similar to the process of selecting the reference fundus image of the group 1-1 described above, but is selected based on the position of the fovea instead of using the optic disc.

In an embodiment, the reference fundus image of the 1-3 group is a fundus image having a minimum distance between the position of the fovea detected for each fundus image and the central position of the image frame among fundus images belonging to the 1-3 group. may be selected (S1230).

In addition, the registration order within the 1-1 group for the fundus images of the 1-3 group is based on the distance between the optic nerve head of the reference fundus image of the 1-1 group and another fundus image in the 1-1 group. may be assigned. This is because, as described above, the optic nerve discs are more easily distinguished visually. The closer the distance between the fovea of another fundus image and the fovea of the reference fundus image is, the higher the matching priority is assigned (S1230).

Meanwhile, the matching order among fundus images in the 1-4 groups is assigned based on the matching degree of the feature points.

In an embodiment, the registration order among the fundus images in the 1-4 group is an initial fundus image (ie, the reference fundus image of the 1-1 group) with each feature point of the fundus images belonging to the group 1-4. Alternatively, it may be assigned based on the degree of matching with the feature points of the image already synthesized up to the previous matching rank. A higher matching degree is assigned a higher matching rank.

In alternative embodiments, the matching method may further include: (eg, by the classification unit 200 ) setting a matching rank between groups (not shown).

When it is assumed that a wide-angle fundus image expressing the region inside the eyeball is generated, it is advantageous to match and synthesize the images based on the center of the eyeball. The fovea has anatomical characteristics close to the center of the retina. Therefore, the more the fovea is included, the higher the priority of registration should be. However, the fovea has properties that are obscure to distinguish visually, such as in the form of slightly darker dots within the retina. Therefore, it is not easy to use only the fovea as a reference in image alignment. On the other hand, the optic nerve papilla is relatively visually distinguishable compared to the fovea.

For this reason, as for the matching rank between the groups, the highest matching rank between the groups is assigned to the 1-1 group including both the fovea and the optic disc. A matching rank among the lowest groups is assigned to the 1-4 groups. Groups 1-1 and 1-3 are assigned between them. For example, in the order of group 1-1, group 1-2, group 1-3, and group 1-4, or group 1-1, group 1-3, group 1-2 and the order of groups 1-4 may be set. Then, in order to generate a wide-angle fundus image, an initial fundus image is first selected from the 1-1 group, and the fundus images in the 1-1 group are selected according to the matching order of the fundus images in the 1-1 group. They are sequentially matched to fundus images, and a composite image is generated.

The step of setting the matching order between the groups may be performed before or after the step of acquiring a plurality of fundus images for generating a wide-angle fundus image ( S1100 ). For example, the step of setting the matching order between groups may be set before the step of synthesizing the matched fundus image to be described below ( S1400 ).

Then, the overall matching order may be assigned to all the plurality of fundus images of groups 1-1 to 1-4. For example, if a matching rank is assigned to fundus images in group 1-1, the first group having the highest matching rank in group 1-2? The fundus images may be sorted as having the next matching rank of the last fundus image having the lowest matching rank in the 1-1 group.

Referring to FIGS. 10 and 11B , the registration method includes: (eg, by the matching unit 300) matching the start fundus image with the fundus image of the next matching order based on the registration order (S1300); .

In certain embodiments, when the overall matching rank is assigned to all the plurality of fundus images of the 1-1 to 1-4 groups, the fundus image added to the already synthesized image may be a single fundus image.

Step S1300 will be described in more detail with reference to the reference fundus image of the 1-1 group having the highest matching rank and the fundus image of the next matching rank in the 1-1 group, which can be used as an initial montage. The source image of the montage, that is, the reference fundus image of the 1-1 group in the 1-1a fundus image set is the 1-1a fundus image, and the fundus image of the next matching rank is the 1-1b fundus image. is referred to

The step S1300 includes: generating a vascular mask of the 1-1a fundus image and the 1-1b fundus image (S1310). The blood vessel mask is generated by extracting the blood vessel region from the fundus image using a pre-trained segmentation model. Then, a 1-1a blood vessel mask representing at least a portion of the blood vessels in the 1-1a region and a 1-1b blood vessel mask representing at least a portion of the blood vessels in the 1-1b region are generated ( S1310 ).

A blood vessel shape is determined based on the blood vessel probability map extracted in step S1310. In certain embodiments, a mask corresponding to a blood vessel shape based on the extracted blood vessel probability map may be generated as an eye blood vessel mask.

The operation of generating the blood vessel mask by extracting the division model and the blood vessel region in step S1310 has been described above with reference to the operation of the matching unit 300 , and detailed description thereof will be omitted.

In addition, the step (S1300) may include: matching the 1-1a fundus image and the 1-1b fundus image based on the feature points of the 1-1a fundus image and the 1-1b fundus image (S1330); and matching the 1-1a fundus image and the 1-1b fundus image based on the generated blood vessel mask (S1350). Since the matching operation in step S1350 is performed after the feature point matching in step S1330, it is treated as a re-matching operation.

Step S1330 is a step of extracting a feature point in the fundus image; and matching the 1-1a fundus image and the 1-1b fundus image based on the extracted feature points.

The feature point includes a point that is easily identified while being distinguished from the surrounding background in matching the image. As described above, geometric feature points may be used as feature points. In an embodiment, the feature point may include a feature point related to the geometry of the blood vessel. For example, a feature point included in the shape of a blood vessel or an edge may be included.

In step S1330, feature points may be extracted from the input image (eg, 1-1a fundus image) through various feature extraction algorithms based on a feature descriptor. The feature extraction algorithm may include, for example, a Scale Invariaant Feature Trnasform (SIFT) that extracts a point having a maximum cornerity on an image axis and/or a scale axis based on Difference of Gaussian (DoG), but is limited thereto. it doesn't happen In step S1330, a feature point set including a plurality of feature points may be extracted.

In step S1330, the feature point-based matching may be performed in the pixel domain.

By rigid registration of the 1-1 feature point set extracted from the 1-1a fundus image and the 1-2 feature point set extracted from the 1-1a fundus image in the step S1330, respectively, The 1-1a fundus image and the 1-1b fundus image may be registered.

In an embodiment, the step S1330 may include matching the first-first feature point set and the first-second feature point set using a perspective transformation matrix. In some embodiments, the step ( S1230 ) may further use the RANSAC algorithm to match the first-first feature point set and the first-second feature point set. For example, in the process of matching the 1-1 feature point set and the 1-2 feature point set using the perspective transformation matrix, a value that maximizes the degree of matching between the feature points of each set is calculated through the RANSAC algorithm. Matching can also be performed.

After the matching in step S1330, the matching in step S1350 is additionally performed.

In an embodiment, the matching method may further include: after keypoint-based matching, verifying validity of the keypoint-based matching result ( S1340 ). Further matching in step S1350 is performed when validity is verified in step S1340.

In an embodiment, for validation, the fundus image having the next matching rank (eg, the 1-1b fundus image) may be subjected to homography transformation ( S1340 ). When the 1-1b fundus image is subjected to homography conversion, the feature points of the 1-1b fundus image are projected onto another fundus image (ie, the 1-1a fundus image). With respect to the 1-1b fundus image, if the change before and after homography transformation does not satisfy the validity condition, the 1-1b fundus image is not matched and is not used to generate a wide-angle fundus image.

In an embodiment, the validity condition may be set to be satisfied when a difference in area between image frames before and after homography transformation is less than a preset threshold. The threshold value is a pixel area difference of an image frame before/after homography conversion, and may be an area change amount itself or an area change rate. When the area change rate is used as a threshold, the threshold may be set to a value of approximately 15% to 5%. For example, the threshold may be 10%. Then, when the difference between before and after homography conversion of the 1-1b fundus image is greater than 10%, the 1-1b fundus image is excluded from the additional registration and synthesis target.

As a result, inaccuracies of registration, including the inappropriateness of applying 2D homography to represent the view transformation of a 3D spherical object, and failure of matching a limited number of feature points due to small overlap or insufficient texture, etc. The problem is at least partially resolved.

Referring again to FIGS. 10 and 11 , the 1-1a fundus image and the 1-1b fundus image are matched based on the blood vessel mask (or the post-processed blood vessel mask) in step S1350 .

In one embodiment, for re-registration after the feature point-based registration of step S1330, deformable registration is performed between the 1-1a blood vessel mask and the 1-1b blood vessel mask to match the 1-1a fundus image The 1-1b fundus image may be re-registered (S1350).

Deformable registration is a non-linear form of registration in which a registration object can be freely deformed. For example, a matching target of a straight line may be deformable in a shape such as a curve, a rectangle, or a circle.

The transformation matching algorithm used in step S1350 may include, for example, the B-Spline algorithm. The B-Spline algorithm is a matching algorithm that iteratively optimizes in the direction of increasing pixel similarity between two images. An optimized matching result is calculated by determining the parameter that has the highest similarity.

As described above in step S1310, when the 1-1a blood vessel mask and the 1-1b blood vessel mask are generated by the segmentation model, the deformation registration is performed based on the more precisely extracted blood vessel region.

For example, if the B-Spline algorithm is applied using the 1-1a blood vessel mask as the source shape and the 1-1b blood vessel mask as the target shape, one or more control points of the 1-1a blood vessel mask are obtained. After extraction, the shape of the 1-1a blood vessel mask may be modified to have an optimization curve that maximizes the pixel similarity to the 1-1b blood vessel mask. Then, the 1-1a blood vessel mask and the 1-1b blood vessel mask may be re-registered so that errors existing in the feature point-based matching result are finely adjusted.

In addition, the registration method includes a step (S1400) of synthesizing the registered fundus images to generate a wide-angle fundus image (eg, by the synthesizing unit 400). Based on the registration result of step S1350, the 1-1a fundus image and the 1-1b fundus image used for registration in step S1300 are synthesized into a partially overlapping single image (S1400). The synthesized single image has an image frame that is larger than the image frame of the synthesized 1-1a fundus image or 1-1b fundus image. Due to this, a composite image having a wider view than a photographing area of a single fundus image is generated (S1400).

As shown in FIG. 11 , the synthesized image of step S1400 includes the overlapping portion of the 1-1a fundus image and the 1-1b and other portions.

In an embodiment, the step of synthesizing the 1-1a fundus image and the 1-1b fundus image for generating the wide-angle fundus image includes: After B-spline optimization, a displacement vector from registration of the vascular mask calculating ; and applying the calculated displacement vector to the 1-1a fundus image or the 1-1b fundus image. The photomontage is expanded due to the application of the displacement vector. When the source image of synthesis (eg, 1-1a fundus image) and the target image of synthesis (eg, 1-1b fundus image) are designated, the displacement vector includes a source shape (eg, 1-1a blood vessel mask) and It may correspond to a parameter that minimizes the sum of distances between points of the target shape (eg, the 1-1b blood vessel mask).

The displacement vector may be calculated through a search on a distance transform (DT) of the edge shape of the target image with respect to the edge shape of the source image. For example, a parameter that minimizes the sum of distances may be calculated by calculating the sum of distances at each point while moving the target shape of the target image in pixel units on the distance map with respect to the edge shape of the source image.

The synthesized image generated in step S1400 may be provided to the user.

In an embodiment, the step ( S1400 ) may include: minimizing a difference in image specification between the 1-1a fundus image and the 1-1b .

The image specification includes intensity, color, and/or contrast of an appearance presented in an image frame. Each image frame has individual image specifications due to changes in imaging factors affecting the fundus image, including color intensity, scattering, and eyelid opening of external light. For this reason, image specifications may be different between an image to be additionally synthesized for montage (ie, 1-1b) and an existing image (ie, 1-1a fundus image). When fundus images having different image specifications are matched, overlapping regions have different pixel values. To solve this, the difference in image specifications is minimized (S1400).

In an embodiment, the step of minimizing the difference in image specification between the 1-1a fundus image and the 1-1b includes: In order to remove color distortion around the overlapping region by registration, the first The method may include transforming the pixels of the 1-2 th fundus image corresponding to the pixels of the -1a fundus image to have the same intensity or color.

The corresponding pixel relationship between the respective images is determined based on the pixel coordinates of the corresponding pixel of the registered image. In step S1400, the corresponding pixel is determined based on the pixel coordinates of the re-registered image.

Then, a composite image including an overlapping region from which color distortion is removed, which is made of transformed pixels, may be generated as a single image.

In an embodiment, the difference in image specification between the first fundus image and 1-1b can be minimized by using a multi-resolution spline technique in the overlap region and boundary of the 1-1a fundus image and the 1-1b. may be

The multi-resolution spline technique may minimize the difference in image intensity between each layer of an image frame by applying weights around the center of the 1-1a fundus image and the 1-1b pair, and then applying a Gaussian filter and a Laplacian filter. . Then, as shown in FIG. 11 , a wide-angle fundus image from which a difference in image intensity is removed is obtained.

When the colors or intensities of 1-1 and 1-1b are different from each other, as shown in FIG. 8B , a registered image in which pixels around the matching portion are corrected to a single color or intensity may be obtained.

As a result of removing the difference in image specifications, a wide-angle fundus image having uniformity in the matching portion as a single image may be provided ( S1400 ). Then, as shown in FIG. 11 , a registered image in which the 1-1a fundus image and a part of 1-1b overlap each other may be obtained. The superimposed registration image visualizes the registration result by fine-tuning the image features of the 1-1a fundus image and the 1-1b fundus image.

A more accurate registered image may be obtained through two-stage registration consisting of such feature point-based registration and deformation registration.

Steps (S1300) and (S1400) are repeatedly performed on the fundus image having the next matching order and the already generated synthetic image. As long as there is no fundus image that has not passed the validation, steps S1300 and S1400 are repeatedly performed according to the registration order for all of the plurality of fundus images acquired in step S1100. Through this, the first wide-angle fundus image and/or the second wide-angle fundus image may be generated.

For example, when the initial fundus image is selected, the remaining fundus images are synthesized according to the registration order within the group 1-4 from the registration order within the 1-1 group, and a final wide-angle fundus image is generated. a registration order of the 1-1 group consisting of the 1-1a to 1-1n fundus images with respect to the first fundus image set consisting of the entire 1A to 1N fundus images; the registration order of the 1-2 group consisting of 1-2a-th to 1-2n-th fundus images; the registration sequence of the 1-3 group consisting of the 1-3a to 1-3n fundus images; The first wide-angle fundus image may be generated by synthesizing the 1-4th group consisting of the 1-4a to 1-4n fundus images in the matching order.

In alternative embodiments, another fundus image to be added to the already synthesized wide-angle fundus image may be a sub-wide-angle fundus image synthesized by matching fundus images in other groups. For example, when the 1-1 sub wide-angle fundus image is generated by registering the fundus images in the 1-1 group, and the 1-2 sub wide-angle fundus image is generated by registering the fundus images in the 1-2 group, An expanded wide-angle fundus image may be generated by registering the 1-1 sub wide-angle fundus image and the 1-2 sub wide-angle fundus image.

In this case, the registration and synthesis of the 1-1 sub wide-angle fundus image and the 1-2 sub wide-angle fundus image are performed through the entire sub wide-angle fundus image, or performed through one fundus image among each sub wide-angle fundus image it might be For example, when registering and synthesizing the 1-1 sub wide-angle fundus image and the 1-2 sub wide-angle fundus image, the first ?? The fundus image and the last fundus image having the lowest matching rank in the 1-1 group may be used.

The apparatus 10 of the second aspect may detect a wide-angle fundus change occurring in an eyeball region corresponding to each wide-angle fundus image by using the first wide-angle fundus image and the second wide-angle fundus image. Here, the eyeball region is an eyeball region corresponding to the generated wide-angle fundus image.

Since the process of detecting the wide-angle fundus change is changed to use the first wide-angle fundus image and the second wide-angle fundus image instead of the first and second fundus images described above with reference to FIGS. 1 to 8, detailed description is omitted.

According to the embodiments described above The operation by the apparatus and the fundus change detection method may be at least partially implemented as a computer program and recorded in a computer-readable recording medium. For example, embodied with a program product consisting of a computer-readable medium containing program code, which may be executed by a processor for performing any or all steps, operations, or processes described.

The computer may be any device that may be incorporated into or may be a computing device such as a desktop computer, laptop computer, notebook, smart phone, or the like. A computer is a device having one or more alternative and special purpose processors, memory, storage, and networking components (either wireless or wired). The computer may run, for example, an operating system compatible with Microsoft's Windows, an operating system such as Apple OS X or iOS, a Linux distribution, or Google's Android OS.

The computer-readable recording medium includes all types of recording identification devices in which computer-readable data is stored. Examples of the computer-readable recording medium include ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage identification device, and the like. In addition, the computer-readable recording medium may be distributed in a network-connected computer system, and the computer-readable code may be stored and executed in a distributed manner. In addition, functional programs, codes, and code segments for implementing the present embodiment may be easily understood by those skilled in the art to which the present embodiment belongs.

Although the present application reviewed above has been described with reference to the embodiments shown in the drawings, this is merely exemplary and those skilled in the art will understand that various modifications and variations of the embodiments are possible therefrom. However, such modifications should be considered to be within the technical protection scope of the present application. Accordingly, the true technical protection scope of the present application should be determined by the technical spirit of the appended claims.

The fundus change detection method according to an aspect of the present application may use machine learning, which is one of the quaternary industrial technologies, to a) extract a blood vessel region, b) determine a region of interest in a fundus image, c) It is also possible to detect fundus changes.

In detecting the fundus change, a fundus change may be detected only with a fundus image that can be easily obtained using a fundus camera through the above-described process without requiring a separate laser scanning device. The fundus camera does not require invasive treatment, does not use radiation, and has a short imaging time of less than a few minutes. Fundus examination using such an fundus camera is very inexpensive (currently, 7,990 won for insurance, 20,770 won for general). In addition, fundus cameras are not only distributed to almost all ophthalmology clinics and ophthalmology hospitals, but also have a number of examination centers, so accessibility and dissemination are excellent.

If fundus changes can be detected using fundus photography having these advantages, it is possible to support reading of various eye diseases related to fundus changes, and it is expected to achieve a breakthrough.

Claims (15)

A matching method performed by a computing device including a processor, comprising:
acquiring first and second fundus images obtained by photographing the subject's eyes at first and second time points, respectively;
registering the first and second fundus images based on intraocular blood vessels;
detecting the region of interest in the first fundus image and the region of interest in the second fundus image from the matched image; and
Fundus change detection method comprising the step of detecting a fundus change from a first time point to a second time point based on the region of interest image in the first fundus image and the region of interest image in the second fundus image.
According to claim 1, wherein the step of matching the first fundus image and the second fundus image,
matching the first fundus image to the second fundus image based on the feature points of the first and second fundus images;
generating a first blood vessel mask representing a blood vessel region in the first fundus image;
generating a second blood vessel mask representing a blood vessel region in the second fundus image; and
and re-registering the first fundus image and the second fundus image by deformable registration of the first blood vessel mask and the second blood vessel mask.
The method of claim 2, wherein the generating of the first blood vessel mask from the first fundus image comprises:
extracting a blood vessel shape of at least a portion of the first fundus image using a blood vessel extraction model based on region characteristics in the first fundus image; and
creating a vascular mask having a shape corresponding to the vascular region;
The blood vessel extraction model is
Fundus change detection method, characterized in that it is learned to output the blood vessels in the input image as a vessel probability map (Vessel Probaibility Map).
The method of claim 2, wherein the generating of the first blood vessel mask from the first fundus image comprises:
generating a binarized blood vessel mask by binarizing an image including the extracted blood vessel shape as a blood vessel region;
detecting a valley between a blood vessel and an external boundary by inverting the image including the binarized blood vessel mask; and
Fundus change detection method, characterized in that it further comprises the step of filling the inside of the valley so that the inside of the valley represents the blood vessel.
The method of claim 2, wherein the step of matching the first fundus image to the second fundus image based on the feature points of the first and second fundus images comprises:
extracting at least one feature point from the first and second fundus images through a feature descriptor for extracting features that are invariant to the size and rotation of the image;
sampling at least one of the feature points extracted from the first and second fundus images; and
Fundus change detection method comprising the step of rigid body registration of the first and second fundus images based on the sampled feature points.
According to claim 1,
The region of interest includes at least one of an optic nerve papilla portion and a macular portion,
The step of detecting one or more regions of interest in the first fundus image,
applying the matched first and second fundus images to a first ROI estimation model to detect the positions of the ROIs in the matched first and second fundus images, respectively;
The first region of interest estimation model,
Fundus change detection method, characterized in that the model is modeled to output at least one of the x-coordinate and the y-coordinate of the optic disc portion, or at least one of the x-coordinate and the y-coordinate of the macula portion.
According to claim 1,
The region of interest includes an abnormal finding region,
Detecting one or more regions of interest in the first fundus image comprises:
The registered first and second fundus images are applied to a second ROI estimation model to detect the positions of the ROIs in the matched first and second fundus images, respectively;
The second region of interest estimation model,
A fundus change detection method, characterized in that the input fundus image is modeled to classify whether or not it belongs to a disease image group.
According to claim 1,
The region of interest includes a region corresponding to a blood vessel,
The position of the blood vessel region is a fundus change detection method, characterized in that the position of the blood vessel mask generated in the first fundus image.
The method of claim 1, wherein calculating the fundus change during the first time point and the second time point comprises:
and detecting a fundus change by applying the first and second fundus images to a change detection model,
The change detection model is
extracting a set of first features and a set of second features from the first and second fundus images, respectively;
compute a correlation map between features of the first and second sets of features;
Based on the correlation map, the pair of the first and second fundus images has a fundus change
Fundus change detection method, characterized in that it is modeled to classify whether it belongs to a group.
The method of claim 1, wherein calculating the fundus change during the first time point and the second time point comprises:
and detecting a change by comparing the color of the detected region of interest in the first fundus image and the detected region of interest in the second fundus image in units of pixels.
According to claim 1,
The first fundus image is a first wide-angle fundus image obtained by synthesizing a plurality of image frames obtained by photographing each sub-region of the entire eye region at a first time point,
The second fundus image is a second wide-angle fundus image obtained by synthesizing a plurality of image frames obtained by photographing the corresponding sub-regions of the front eye region at a second time point.
According to claim 1,
Prior to detecting the fundus change, the method further comprises generating the first or second wide-angle fundus image, wherein the generating of the first or second wide-angle fundus image comprises:
obtaining the plurality of image frames obtained by photographing the subject's eyeballs at the same time point, wherein at least one image frame among the plurality of image frames partially overlaps with at least one other image frame;
matching a first image frame of the plurality of image frames with a second image frame; and
Fundus change, comprising the step of generating a composite image in which at least a partial region of the first image frame and at least a portion of the second image frame are superimposed on the basis of the re-registration result to generate a wide-angle fundus image detection method.
A computer readable medium readable by a computing device and storing program instructions operable by the computing device, wherein the processor is configured to cause the program instructions to be executed by a processor of the computing device when the program instructions are executed by the processor of the computing device. A computer-readable recording medium for performing the method for detecting fundus changes according to any one of claims.
an image acquisition unit for acquiring a fundus image taken by a fundus camera, wherein the fundus image includes a first fundus image of the subject photographed at a first time point and a second fundus image of the subject photographed at a second time point;
a matching unit for generating a blood vessel mask representing a blood vessel region in the fundus image, and matching first and second fundus images based on the blood vessel masks of the first and second fundus images; and
A device comprising: a fundus change detector configured to detect a fundus change from a first time point to a second time point based on a matching result.
An image acquisition unit for acquiring a first fundus image set comprising a plurality of fundus images taken at a first time point, and a second fundus image set comprising a plurality of fundus images taken at a second time point - the first fundus image set or at least one fundus image in the second set of fundus images partially overlaps with at least one other fundus image;
With respect to the first fundus image set or the second fundus image set, at least one of an optic disc and a fovea is detected in a plurality of fundus images of each set, and depending on whether the optic disc and fovea are detected a classification unit that classifies the plurality of fundus images for each set into a plurality of groups, and assigns, to at least one group, a matching rank within the corresponding group to the fundus images for each group;
For the fundus images included in each group, a matching unit that matches the fundus image having the current registration order with the synthesized image generated by matching at least one fundus image up to the previous registration order based on the registration order within the group ;
In order to generate a first or second wide-angle fundus image from the first or second fundus image set, a fundus image having a current registration order is generated by registering at least one fundus image up to a previous registration order based on the registration result. a synthesizer for generating a new synthesized image from the synthesized image; and
and a fundus change detector configured to detect a fundus change in an eye area corresponding to a wide-angle fundus image from a first time point to a second time point.

Images