KR102575371B1 - Method for registrating fundus images to generate wide angle fundus image and device performing the same - Google Patents

Abstract

Embodiments include obtaining a plurality of fundus images of the subject's eye - at least one fundus image of the plurality of fundus images partially overlaps with at least one other fundus image; Matching a first fundus image of the plurality of fundus images with a second fundus image; and generating a composite image in which at least a portion of the first fundus image overlaps with at least a portion of the second fundus image based on a re-registration result to generate a wide-angle fundus image. It relates to a method of matching multiple fundus images to generate a wide-angle fundus image and a device for performing the same.

Description

Method for registering multiple fundus images to generate a wide-angle fundus image and device for performing the same {METHOD FOR REGISTRATING FUNDUS IMAGES TO GENERATE WIDE ANGLE FUNDUS IMAGE AND DEVICE PERFORMING THE SAME}

The present invention relates to a technology for registering fundus images, and more specifically, to a method for registering a plurality of fundus images to generate a wide-angle fundus image with a wider area than the shooting angle of a fundus camera, and a device for performing the same. .

Wide-angle fundus imaging is an image that comprehensively visualizes diseases occurring in various areas of the eye at a glance, and is an image required for a comprehensive diagnosis of the subject's eye.

Wide-angle fundus images are acquired using a traditional montage technique, or using a wide-angle fundus imaging device (eg, a laser scan device from OPTOS).

The traditional montage technique creates a wide fundus image by matching common areas in multiple partial images of the retina taken piecemeal. However, there is a problem of being affected by the presence of obstacles other than the subject of photography.

OPTOS' laser scanning technology, which extracts black-and-white data and visualizes it by overlaying it with an appropriate color image, can acquire a very wide area of 200 degrees, covering 80% of the entire retina. However, errors such as motion artifacts may occur while the laser is scanning. In addition, there is a limitation that OPTOS's laser scanning device is required separately from general fundus photography devices.

Korean Patent No. 10-1761510 (2017.07.26.)

According to one aspect of the present invention, an apparatus for matching a plurality of fundus images of a plurality of eye regions can be provided to generate a wide-angle fundus image having an area wider than the shooting angle of the fundus camera.

In addition, it is possible to provide a fundus image registration method for generating a wide-angle fundus image and a computer-readable recording medium on which instructions for performing the method are recorded.

A method of matching a plurality of fundus images to generate a wide-angle fundus image according to one aspect of the present application is performed by a computing device including a processor. The method includes: acquiring a plurality of fundus images of the eyeball of a subject - at least one fundus image of the plurality of fundus images partially overlaps with at least one other fundus image; Matching a first fundus image of the plurality of fundus images with a second fundus image; And in order to generate a wide-angle fundus image, it may include generating a composite image in which at least a portion of the first fundus image overlaps with at least a portion of the second fundus image based on a re-registration result.

In one embodiment, the step of matching the first fundus image and the second fundus image based on the feature points of the first and second fundus images includes at least using a feature descriptor that extracts features that are invariant to the size and rotation of the image. Extracting one feature point from the first and second fundus images; sampling one or more feature points among the feature points extracted from the first and second fundus images; And it may include rigid body registration of the first and second fundus images based on the sampled feature points.

In one embodiment, the step of generating a first blood vessel mask from the first fundus image includes extracting the shape of at least some blood vessels of the first fundus image using a segmentation model based on region characteristics within the first fundus image. step; and generating a blood vessel mask having a shape corresponding to the blood vessel region. The segmentation model is a model learned based on the characteristics of the blood vessel region, and is learned to output the blood vessels in the input image as a vessel probability map.

In one embodiment, generating a first blood vessel mask from the first fundus image includes generating a binarized blood vessel mask by binarizing an image including the extracted blood vessel shape as a blood vessel area; detecting a valley between a blood vessel and an external boundary by inverting an image including a binarized blood vessel mask; and filling the inside of the valley so that the inside of the valley represents the blood vessels.

In one embodiment, the step of re-registering the first fundus image and the second fundus image based on the first and second blood vessel masks includes deformable registration of the first blood vessel mask and the second blood vessel mask. ) may include the step of re-registering the first fundus image and the second fundus image.

In one embodiment, the step of generating the wide-angle fundus image includes pixels of the second fundus image corresponding to pixels of the first fundus image based on the re-registered image having the same intensity as the pixels of the first fundus image ( Transforming to have intensity or color; And it may also include generating a wide-angle fundus image including the area made up of the deformed pixels.

In one embodiment, the step of generating a first blood vessel mask representing at least a portion of the blood vessels in the first region may include, when a plurality of first fundus images are acquired, based on feature points of each of the first fundus images. Registering the first fundus image of; extracting the shape of at least a portion of blood vessels in a first region from the first fundus image using a segmentation model based on region characteristics within the fundus image; and generating a first blood vessel mask based on the extracted blood vessel shapes of the plurality of first fundus images.

In one embodiment, generating a first blood vessel mask based on the extracted blood vessel shapes of the plurality of first fundus images includes non-rigidly registering the extracted blood vessel shapes of the plurality of first fundus images; and generating the first blood vessel mask based on the plurality of registered blood vessel shapes.

In one embodiment, the non-rigid registration step may be performed in a direction that increases the pixel-wise similarity in the blood vessel probability maps of the two different first fundus images.

In one embodiment, the method includes: prior to registering the plurality of fundus images, detecting at least one of the optic disc and the fovea in the plurality of fundus images, and detecting at least one of the optic disc and the fovea. calculating a location; Classifying the plurality of fundus images into a plurality of groups according to whether the optic nerve head and fovea are detected; And, for at least one group, assigning a matching rank within the group to the fundus image for each group. The step of matching the first and second fundus images and generating a composite image is repeated until all fundus images included in each group are matched, and registering the first and second fundus images and generating a composite image In this step, the first fundus image is an initial fundus image or a synthesized image that has already been created, and the second fundus image is a fundus image with the current registration rank.

In one embodiment, the step of calculating the position by detecting at least one of the optic nerve head and the fovea may include detecting at least one of the optic nerve head and the fovea in an image frame of an input fundus image using a deep learning model. The deep learning model includes: a shared convolutional network that includes multiple convolutional layers and outputs a convolutional feature map; A region designation network that generates an intermediate feature map by applying a sliding window method to the input convolutional feature map and predicts a region of interest where a candidate object is likely to be located based on the intermediate feature map; and an object detection network that includes a pooling layer and a fully connected layer and detects the optic nerve head or fovea in the region of interest based on the feature vector for the region of interest in the image frame initially input to the detection model.

In one embodiment, classifying the plurality of fundus images into a plurality of groups includes: classifying the fundus image in which both the optic nerve head and the fovea are detected into a first group; Classifying the fundus image in which only the optic nerve head is detected into a second group; Classifying the fundus image in which only the fovea is detected into a third group; and classifying the fundus image in which neither the optic nerve head nor the fovea are detected into a fourth group.

In one embodiment, the step of assigning a matching rank within the group to the fundus image for each group includes: for at least one group among the first to third groups, each reference fundus image for each group. Steps to choose; And it may include assigning a matching rank within the group to the fundus image of each group using the reference fundus image. The registration rank within the first group for the fundus images of the first group is assigned based on the distance between the optic nerve head of the reference fundus image of the first group and other fundus images within the first group, and the fundus images of the second group The registration rank within the second group is assigned based on the distance between the optic nerve head of the reference fundus image of the second group and other fundus images within the second group, and the within-third group for the fundus images of the third group. The registration rank may be assigned based on the distance between the optic nerve head of the reference fundus image of the third group and other fundus images within the third group.

In one embodiment, the reference fundus image of the first or second group may be one in which the position of the detected optic nerve head of each fundus image is closest to the position of the center of the image frame.

In one embodiment, the reference fundus image of the third group may be one in which the detected foveal position of each fundus image in the third group is closest to the position of the image center.

In one embodiment, the step of allocating the within-group registration rank to the fundus image for each group may include: allocating a fourth within-group registration rank to the fundus image for each group. The matching rank within the fourth group is assigned based on the degree to which the feature points of each fundus image within the fourth group match the feature points of the first fundus image.

In one embodiment, the method may further include: setting a matching rank between groups. The matching rank between the groups is set in the order of the first group, the second group, the third group, and the fourth group, or the order of the first group, the third group, the second group, and the fourth group.

A computer-readable recording medium readable by a computing device according to another aspect of the present application stores program instructions operable by the computing device. When executed by a processor of the computing device, the program instruction causes the processor to perform the fundus image registration method according to the above-described embodiments.

A device according to another aspect of the present application includes: an image acquisition unit for acquiring a plurality of fundus images captured by a fundus camera - at least one fundus image of the plurality of fundus images is partially different from at least one other fundus image nested with; Based on the feature points of the first and second fundus images, the first fundus image and the second fundus image are matched, a first blood vessel mask representing at least a portion of the blood vessels in the first area is generated, and the blood vessels in the second area are generated. a registration unit that generates a second blood vessel mask showing at least a portion of the blood vessels, and re-registers the first fundus image and the second fundus image based on the first and second blood vessel masks; And in order to generate a wide-angle fundus image, it may include a synthesis unit that generates a composite image in which at least a portion of the first fundus image overlaps with at least a portion of the second fundus image based on a re-registration result.

A device according to another aspect of the present application includes: an image acquisition unit for acquiring a plurality of fundus images captured by a fundus camera - at least one fundus image of the plurality of fundus images is partially different from at least one other fundus image nested with; Before matching the plurality of fundus images, at least one of an optic disc and a fovea is detected in the plurality of fundus images, and the plurality of fundus images are selected according to whether the optic disc and the fovea are detected. a classification unit that classifies into a plurality of groups and, for at least one group, assigns a matching rank within the group to the fundus image for each group; For the fundus images included in each group, a registration unit that matches the fundus image with the current registration rank with a synthetic image generated by matching at least one fundus image up to the previous registration rank based on the registration rank within the group. ; and a synthesis unit that generates a new composite image from the composite image generated by matching the fundus image with the current registration rank with at least one fundus image up to the previous registration rank based on the registration result, in order to generate a wide-angle fundus image. You may.

The device according to one aspect of the present invention can generate a wide-angle fundus image by matching a plurality of fundus images with different shooting ranges. As a result, the device can also support cross-sectional analysis.

The device does not require the use of a separate device, such as a laser scanning device from OPTOS, to generate visualization images such as registered images for cross-sectional analysis. Additionally, the device may generate such visualization images without being affected by the fundus imaging environment and/or imaging device.

The effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description of the claims.

In order to more clearly explain the technical solutions of the embodiments of the present invention or the prior art, drawings necessary in the description of the embodiments are briefly introduced below. It should be understood that the drawings below are for illustrative purposes only and not for limiting purposes of the embodiments of the present specification. Additionally, for clarity of explanation, some elements may be shown in the drawings below with various modifications, such as exaggeration or omission.
1 is a conceptual diagram of a device that performs a fundus image registration method for generating a wide-angle fundus image, according to an embodiment of the present invention.
Figure 2 shows a schematic network architecture of the optic disc/macula detection model, according to one embodiment of the present invention.
Figure 3 is a flowchart of a fundus image registration method for generating a wide-angle fundus image, according to an embodiment of the present invention.
FIG. 4 is a schematic diagram of a fundus domain matching method for generating the wide-angle fundus image of FIG. 3.
Figures 5A to 5D are diagrams showing the results of post-processing the extracted blood vessel mask according to an embodiment of the present invention.
Figures 6a and 6b are diagrams showing feature point-based matching results according to an embodiment of the present invention.
FIGS. 7A and 7B are diagrams showing blood vessel mask-based registration results according to an embodiment of the present invention.
FIGS. 8A and 8B are diagrams illustrating a registered image with color distortion removed, according to an embodiment of the present invention.
Figures 9 and 10 show the results of generating the final wide-angle fundus image according to the method of Figure 3.

The terminology used herein is only intended to refer to specific embodiments and is not intended to limit the invention. As used herein, singular forms include plural forms unless phrases clearly indicate the contrary. As used in the specification, the meaning of "comprising" refers to specifying a particular characteristic, area, integer, step, operation, element and/or ingredient, and the presence or presence of another characteristic, area, integer, step, operation, element and/or ingredient. This does not exclude addition.

Although not defined differently, all terms including technical and scientific terms used herein have the same meaning as those generally understood by those skilled in the art in the technical field to which the present invention pertains. Terms defined in commonly used dictionaries are further interpreted as having meanings consistent with related technical literature and currently disclosed content, and are not interpreted in ideal or very formal meanings unless defined.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

1 is a conceptual diagram of a device that performs a fundus image registration method according to an embodiment of the present invention.

Referring to FIG. 1, the device 10 is configured to acquire a fundus image from a fundus camera (not shown). The device 10 includes an image acquisition unit 100; Feature point matching unit 300; and a blood vessel matching unit 500.

The device 10 according to embodiments may be entirely hardware, entirely software, or have aspects that are partly hardware and partly software. For example, a device may collectively refer to hardware equipped with data processing capabilities and operating software for running it. In this specification, terms such as “unit,” “module,” “device,” or “system” are intended to refer to a combination of hardware and software driven by the hardware. For example, the hardware may be a data processing device that includes a Central Processing Unit (CPU), Graphics Processing Unit (GPU), or other processor. Additionally, software may refer to a running process, object, executable, thread of execution, program, etc.

The image acquisition unit 100 is configured to acquire a fundus image (or fundus photograph) of a subject by photographing the subject's eye.

In certain embodiments, the fundus camera transmits captured data of the subject's eye to the device 10 through wired/wireless electrical communication. The image acquisition unit 100 may be an interface that receives captured data wired/wireless from an external fundus camera. In some embodiments, the image acquisition unit 100 may be a wired interface port. In some other embodiments, the image acquisition unit 100 may be a communication module that receives data wirelessly.

In certain other embodiments, a fundus camera may be integrated into the device 10. In this case, the image acquisition unit 100 may be implemented as a fundus camera.

The fundus camera includes various fundus cameras capable of acquiring fundus images. For example, the fundus camera may include a mydriatic fundus camera, a non-mydriatic fundus camera, an OCT type fundus camera, etc. The fundus image includes a retinal fundus image.

The image acquisition unit 100 may acquire multiple fundus images. The device 10 generates a single composite image having a size wider than the size of a single fundus image by partially matching and synthesizing at least some of the fundus images among the plurality of images acquired by the image acquisition unit 100.

The plurality of fundus images include a first group of fundus images including both the optic nerve head and the fovea in an image frame, a second group of fundus images including only the optic nerve head in an image frame, a third group of fundus images including only the fovea in an image frame, and Includes the fourth group of fundus images that do not include both the optic nerve head and the fovea within the image frame.

Additionally, the image acquisition unit 10 may acquire data related to the fundus image. For example, data related to a fundus image may include subject identification information that can identify the subject of the fundus image (e.g., name, identity information, identifier, etc.), time of shooting, and/or information related to the imaging device. . Additionally, label data indicating whether the acquired fundus image is the fundus image of the left eye or the fundus image of the right eye may be further obtained. Then, the device 10 generates a wide-angle fundus image by matching fundus images of the same subject, or by matching fundus images of the eye on the same side (for example, fundus images of the left eye or fundus images of the right eye). Wide-angle fundus images can also be created. When using a pair of a first binocular fundus photograph and a second binocular fundus photograph taken at a second time point, the device 10 matches the first fundus photograph of the left eye and the second fundus photograph of the left eye, or Like registering the first fundus photo of and the second fundus photo of the right eye, the device 10 may perform a registration operation for each corresponding eye.

The image acquisition unit 100 supplies a plurality of fundus images to the other components 200, 400, or 500. Additionally, the image acquisition unit 100 may provide a fundus image and data related to the image.

The classification unit 200 detects the optic nerve head or fovea in each image frame of the plurality of fundus images and classifies the plurality of fundus images into first to fourth groups.

In one embodiment, the classifier 200 may detect the optic nerve head or fovea in an image frame using a pre-learned optic nerve head/macula detection model. Additionally, the classification unit 200 may calculate the two-dimensional position of the detected object on the image frame. The classification unit 200 calculates the two-dimensional positions of the optic nerve head and fovea as pixel coordinates on the image frame.

Figure 2 shows a schematic network architecture of the optic disc/macula detection model, according to one embodiment of the present invention.

Referring to Figure 2, the optic disc/macula detection model (hereinafter, “detection model”) is a shared convolutional network; zoning network; Includes an object detection network.

The shared convolutional network includes multiple convolutional layers, the output of which is shared by the localization network and the object detection network.

The convolutional feature map output by the last shared convolutional layer of the shared convolutional network is input to the region-specific network.

The region designation network generates an intermediate feature map by applying a sliding window method to the input convolutional feature map, and predicts a region of interest where a candidate object is likely to be located based on the intermediate feature map.

The sliding method is a method of searching for a desired object in the entire range of input data by sliding a window in the input data. This window slides along the input convolutional feature map. The feature window at each sliding position is mapped to a lower-dimensional feature. Then, the region-specific network generates an intermediate feature map of lower dimension than the input convolutional feature map. The region specification network predicts the region of interest based on the intermediate feature map.

In one embodiment, the localization network may predict the region of interest such that the candidate object (i.e., optic disc or fovea) is located at the center of a bounding box. Then, the location of the bounding box corresponds to the location of the center, that is, the location of the candidate object.

In some embodiments, the region-specific network may predict a region of interest using a bounding box of a fixed size in advance.

The sizes of the optic disc and fovea generally tend to be consistent. The size of the bounding box may be preset to a value obtained based on this tendency.

As shown in Figure 2, the region-specific network predicts a region of interest that is likely to include the optic disc and fovea.

The object detection network includes a pooling layer and a fully connected layer.

The convolutional feature map output by the last shared convolutional layer of the shared convolutional network is also input to the object detection network. The input convolutional feature map is input to the pooling layer.

The pooling layer pools the input data based on the region-of-interest prediction result by the region-specific network. The pooling layer is configured to output a feature vector when a feature map is input. Then, the feature vector for the region of interest from the image frame initially input to the detection model in Figure 3 is input to the fully connected layer.

In some of the above embodiments, when the size of the alert box is fixed, the size of the intermediate feature map of the searched region of interest is also fixed. Then, the object detection network obtains a feature vector of a fixed size and inputs it into a fully connected layer.

The fully connected layer is configured to predict the positions of the optic nerve head and fovea by calculating the probability that the optic nerve head and fovea exist in the region of interest.

In one embodiment, the object detection network may output results including the two-dimensional location and size of the bounding box on the input image frame, the class of each object (i.e., optic disc or fovea), and a confidence score for each class. there is. The confidence score may be a probability value or another converted value based thereon.

The detection model detects the optic nerve head and fovea based on this result. The detection model selects the region of interest with the highest reliability value for each class and detects the optic nerve head and fovea.

In one embodiment, the detection model selects the region of interest with the highest value for the corresponding reliability for each class, and if the reliability of the class of the selected region of interest is higher than a preset reliability threshold, the class of the selected region of interest Objects can also be detected as the optic nerve head or fovea.

If the reliability is a probability value, the reliability threshold may be, for example, 0.9, but is not limited thereto.

The detection model is learned using a training data set including some or all of the first to fourth groups. The detection model is learned to have parameter values to minimize the difference between the actual and predicted values of the training fundus image.

The classification unit 200 automatically classifies a plurality of fundus images into first to fourth groups depending on whether the optic nerve head or fovea is detected. The operation of the classification unit 200 will be described in more detail below with reference to FIG. 3 and the like.

The registration unit 400 may extract at least one feature point from the fundus image. The feature points are points related to geometric features expressed in the fundus image. The geometric features include edges, curves, etc. Extracted feature points may also be used for feature point-based matching operations.

The matching unit 400 extracts blood vessel areas for each of the plurality of fundus images in order to match different fundus images. Additionally, the matching unit 400 may generate a blood vessel mask based on the extraction result of the blood vessel area.

The registration unit 400 may generate a blood vessel mask representing at least a portion of blood vessels in the fundus image.

The blood vessel mask has blood vessel areas in the shape of a mask. The blood vessel mask may include outlines of at least some blood vessels in the fundus image. Additionally, the blood vessel mask may be configured to visually distinguish between the outside of the blood vessel and the blood vessel area.

In one embodiment, the matching unit 400 may extract a blood vessel region from a fundus image using a pre-learned segmentation model. The segmentation model is configured to extract a blood vessel region based on region characteristics within the input fundus image. The segmentation model may calculate a vessel probability map consisting of a probability value indicating whether a pixel forming an image frame of a fundus image is a blood vessel, and determine the blood vessel area based on this.

The segmentation model is a machine learning model with a deep learning structure, including a feature extraction layer; and a classification layer.

The feature extraction layer is configured to extract features used to classify parts of the input image into blood vessel classes. When the segmentation model is a CNN (Convolutional Neural Network)-based model, the feature extraction layer may include a convolutional layer and/or a pooling layer.

The classification layer includes a classification layer configured to calculate the probability that a portion (eg, a pixel or a set of pixels) in the image will be classified into a blood vessel class based on the extracted features. Then, a blood vessel probability map is calculated for each pixel in the fundus image, including the probability that the pixel is a blood vessel.

The blood vessel shape is determined based on the blood vessel probability map calculated by the segmentation model. In certain embodiments, a mask corresponding to the blood vessel shape based on the calculated blood vessel probability map may be created as an ocular blood vessel mask.

In some embodiments, the classification layer may be further configured to classify into vascular classes and non-vascular classes based on the calculated probability. In this case, the matching unit 400 may generate an image of a blood vessel area composed of pixels classified into blood vessel classes.

The segmentation model may have various structures based on CNN (Convolutional Neural Network). The segmentation model includes a layer that extracts features from the input image; and a classification layer configured to calculate a probability that a portion (eg, pixel) in the image will be classified into a blood vessel class based on the extracted features. The segmentation model may include, for example, but is not limited to SSANet, or Rential SSANet based architecture.

The segmentation model is learned to calculate the probability that a portion (eg, pixel) in the image belongs to a blood vessel group using training samples including a plurality of training fundus images. The segmentation model may be a model learned through various learning processes that have class classification as the learning purpose or calculation of a probability value for class classification of an input image portion (eg, pixel).

For example, the parameters of a segmentation model with a fundus image as input may be updated and learned in a direction that minimizes the cost function of the model. Here, the cost function represents the difference between the result value calculated by the model and the actual result value. Learning of these parameters is commonly referred to as optimization. Optimization of parameters may include, for example, Adaptive Moment Estimation (ADAM), Momentum, Nesterov Accelerated Gradient (NAG), Adaptive Gradient (Adagrad), RMSProp, and various gradient descent methods.

Through this learning process, the segmentation model is trained to enhance its ability to calculate higher probability values for actual blood vessels and lower probability values for non-vessels in fundus images. Additionally, by strengthening the probability value calculation ability, the ability to identify blood vessel areas may also be strengthened.

The matching unit 400 matches images (e.g., a single image already synthesized from the beginning to the previous matching rank and a fundus image to be synthesized next) with each other based on the extracted features and/or blood vessels (i.e., blood vessel mask). You may.

The synthesis unit 500 may generate a single composite image from different fundus images that are registered.

The device 10 may generate a wide-angle fundus image using the matching unit 400 and the combining unit 500. The wide-angle fundus image is implemented as a montage in which individual images are partially overlapped and synthesized. The creation of this montage starts from the central frame of the montage and proceeds by combining the fundus image with the already created montage according to the registration ranking. Then, the size of the montage is gradually expanded to generate a final wide-angle fundus image with a comprehensive view of multiple fundus images.

The matching operation of the matching unit 400 and the wide-angle fundus image generating operation of the combining unit 500 will be described in more detail below with reference to FIG. 3 and the like.

It will be apparent to those skilled in the art that the device 10 may include other components not described herein. For example, the device 10 may include other hardware elements necessary for the operations described herein, including a network interface, an input device for data entry, and an output device for display, printing, or other data presentation. It may be possible.

The fundus image registration method for generating a wide-angle fundus image according to another aspect of the present invention may be performed by one or more processors, such as the device 10 described above. Hereinafter, for clarity of explanation, the steps of the fundus image registration method will be described in more detail based on embodiments performed by the device 10.

Figure 3 is a flowchart of a fundus image registration method for generating a wide-angle fundus image, according to an embodiment of the present invention, and Figure 4 is a schematic diagram of a fundus effect registration method for generating a wide-angle fundus image of Figure 3. FIG. 4B is a schematic partial enlarged view of step S400 of FIG. 4A.

The fundus domain registration method (hereinafter, “registration method”) for generating the wide-angle fundus image includes: acquiring a plurality of fundus images (e.g., by the image acquisition unit 100) (S100). Each of the plurality of fundus images may include the optic nerve head and/or the macular fovea, or may not include both.

The matching method includes: detecting the optic nerve head and/or fovea for each of the plurality of fundus images (e.g., by the classification unit 200) and classifying them into a plurality of groups (S200).

In one embodiment, the optic disc and/or fovea may be detected in an image frame of a fundus image using a pre-learned optic disc/macula detection model (S200). The detection model in step S200 and the detection operation of the optic nerve head and fovea using the same have been described above with reference to FIG. 2, so detailed description will be omitted.

In certain embodiments, the plurality of groups include first through fourth groups. The step (S200) includes classifying the fundus image in which both the optic nerve head and the fovea are detected into a first group; Classifying the fundus image in which only the optic nerve head is detected into a second group; Classifying the fundus image in which only the fovea is detected into a third group; And/or it may include classifying the fundus image in which neither the optic nerve head nor the fovea are detected into a fourth group. For example, in step S200, the plurality of images in step S100 may be classified into first to fourth groups, respectively.

Additionally, the matching method includes: assigning a matching rank between fundus images within each group for each group (e.g., by the classification unit 200) (S300). The step (S300) includes: assigning matching ranks to fundus images included in some or all of the first to fourth groups; And/or assigning a matching rank within the fourth group to the fundus images of the fourth group.

Once a registration rank is assigned, the order of fundus images to be registered depends on the registration rank. For example, fundus images belonging to the first group are sequentially matched according to the matching rank between the fundus images in the first group. Alternatively, the fundus images belonging to the fourth group may be sequentially matched according to the matching rank between the fundus images within the fourth group.

In one embodiment, the step (S300) includes: selecting a reference fundus image for each group for at least one of the first to third groups; And it may include assigning a matching rank within the group to the fundus image of each group using the reference fundus image. For example, a reference fundus image for each of the first to third groups is selected, and an intra-group matching rank is assigned using the reference fundus image for each group (S300).

The fovea and optic nerve head have anatomical characteristics that are close to each other. For this reason, since the fundus image in which the fovea is located in the center of the image frame generally includes the optic nerve head, it can be treated as belonging to the first group.

In one embodiment, the reference fundus image of the first group may be selected as a fundus image having the minimum distance between the position of the optic nerve head detected for each fundus image and the center position of the image frame among the fundus images belonging to the first group. There is (S300). When the first group of reference fundus images, which can be used as the initial image frame of the wide-angle fundus image, is selected in this way, as described above, it is advantageous to start from the center of the eye to generate the wide-angle fundus image, and at the same time, it is visually This is because you can have all the convenience of distinction.

Additionally, the registration rank within the first group for the fundus images of the first group may be assigned based on the distance between the optic nerve head of the reference fundus image of the first group and other fundus images within the first group. As mentioned above, this is because the optic nerve head is more easily visually distinguished. The closer the distance between the optic nerve head of another fundus image and the optic nerve head of the reference fundus image, the higher the registration rank is assigned (S300).

The distance is, for example, ||p ^o _i -p ^o _ref || It can also be calculated using the equation ₂ . Here, i is the index within the group, p ^o _i represents the pixel coordinate on the image frame of the optic nerve head of the ith fundus image of the first group, and p ^o _ref is the pixel coordinate of the optic nerve head on the image frame of the reference fundus image. represents.

Since the fundus images of the first and second groups include the optic nerve head, the reference fundus images of the first group may be used as the reference fundus images of the second group. That is, when the reference fundus image of the first group is selected, the reference fundus image of the second group is also automatically selected.

The registration rank within the second group for the fundus images of the second group is assigned based on the distance between the optic nerve head of the reference fundus image of the second group and other fundus images within the second group (S300).

The reference fundus image of the third group is similar to the process of selecting the reference fundus image of the first group described above, but is selected based on the location of the fovea instead of using the optic nerve head.

In one embodiment, the reference fundus image of the third group may be selected as a fundus image having the minimum distance between the position of the fovea detected for each fundus image and the center position of the image frame among the fundus images belonging to the third group. There is (S300).

Additionally, the registration rank within the first group for the fundus images of the third group may be assigned based on the distance between the optic nerve head of the reference fundus image of the first group and other fundus images within the first group. As mentioned above, this is because the optic nerve head is more easily visually distinguished. The closer the distance between the fovea of another fundus image and the fovea of the reference fundus image, the higher the registration rank is assigned (S300).

Meanwhile, the matching rank between fundus images in the fourth group is assigned based on the degree of matching of feature points.

In one embodiment, the registration ranking between the fundus images in the fourth group is determined so that the feature points of each of the fundus images belonging to the fourth group have already reached the initial fundus image (i.e., the reference fundus image of the first group) or the previous registration ranking. It may be assigned based on the degree of matching with the feature points of the synthesized image. The higher the degree of matching, the higher the matching rank is assigned.

In alternative embodiments, the matching method may further include: setting a matching rank between groups (e.g., by the classification unit 200) (not shown).

Assuming that a wide-angle fundus image representing the area inside the eye is being created, it is advantageous to register and synthesize based on the center of the eye. The fovea has anatomical characteristics that are close to the center of the retina. Therefore, the more the fovea is included, the higher the registration priority should be. However, the fovea has characteristics that make it visually indistinguishable, such as the appearance of a slightly darker spot within the retina. Therefore, it is not easy to use only the fovea as a reference in image alignment. On the other hand, the optic nerve head has characteristics that make it relatively easy to visually distinguish compared to the fovea.

For this reason, the highest matching rank between the groups is assigned to the first group, which includes both the optic nerve head as well as the fovea. The lowest inter-group matching rank is assigned to the fourth group. A first group and a third group are allocated between them. For example, it may be set in the order of the first group, the second group, the third group, and the fourth group, or the order of the first group, the third group, the second group, and the fourth group. Then, in order to generate a wide-angle fundus image, an initial fundus image is first selected from the first group, and the fundus images in the first group are sequentially registered to the initial fundus image according to the registration order of the fundus images in the first group. and generate a composite image.

The step of setting the matching rank between the groups may be performed before or after the step of acquiring a plurality of fundus images for generating a wide-angle fundus image (S100). For example, the step of setting the matching ranking between groups may be set before the step of synthesizing the matched fundus image (S500), which will be described below.

Then, an overall registration rank may be assigned to all the plurality of fundus images of the first to fourth groups. For example, if a registration rank is assigned to the fundus images in the first group, the first one with the highest registration rank in the second group?? Fundus images may be sorted by having the next registration rank after the last fundus image with the lowest registration rank in the first group.

The registration method includes: registering the starting fundus image and the fundus image of the next registration order based on the registration order (eg, by the registration unit 400) (S400).

In certain embodiments, when an overall registration rank is assigned to all the plurality of fundus images of the first to fourth groups, the fundus image added to the already synthesized image may be a single fundus image.

The step (S400) will be described in more detail with reference to the reference fundus image of the first group with the highest registration rank and the fundus image of the next registration rank within the first group, which can be used as an initial montage. The source image of the montage, that is, the reference fundus image of the first group, is referred to below as the first fundus image, and the fundus image in the next registration order is referred to below as the second fundus image.

Referring to FIGS. 3 and 4B, the step (S400) includes: generating a blood vessel mask of the first fundus image and the second fundus image (S410). The blood vessel mask is created by extracting blood vessel areas from the fundus image using a pre-learned segmentation model. Then, a first blood vessel mask showing at least part of the blood vessels in the first region and a second blood vessel mask showing at least a part of the blood vessels in the second area are generated (S410).

The blood vessel shape is determined based on the blood vessel probability map extracted in step S410. In certain embodiments, a mask corresponding to blood vessel shapes based on the extracted blood vessel probability map may be created as an ocular blood vessel mask.

The operation of generating a blood vessel mask by extracting the segmentation model and blood vessel region in step S410 has been described above with reference to the operation of the matching unit 400, and detailed description will be omitted.

In alternative embodiments, step S410 may include post-processing the blood vessel shape based on the extracted blood vessel probability map. Then, a more accurate and improved blood vessel mask can be generated than a blood vessel mask based only on the blood vessel probability map calculated by the blood vessel model. Post-processing includes binarization and/or redefining the boundaries of the vessel shape. In certain embodiments, the post-processing step may be performed after feature point-based registration.

Figures 5A to 5D are diagrams showing the results of post-processing the extracted blood vessel mask according to an embodiment of the present invention.

In one embodiment, the post-processing step may include: applying hysteresis thresholding to the input image to binarize it to generate a binarized fundus mask.

As shown in FIG. 5A, an image of the blood vessel area may be generated based on the blood vessel probability map calculated in step S410. The double zero-field technique is applied to the image in Figure 5a.

The double threshold technique is a technique that determines target data using an upper and lower threshold. To prevent errors in which thin blood vessels (filamentray vessels) appear disconnected in the image, the blood vessel area of the input image is binarized based on a preset first threshold (τl) and a second threshold (τh). The second threshold value (τh) may be a higher threshold value than the first threshold value (τl). Pixels above a second threshold (τ h ) are used as seeds for regions in which pixels with a higher probability than the lower threshold (τ _l ) grow. The first and second thresholds may be empirical values, such as τ ₁ = 0.1, τ _h = 0.75, etc.

By this double threshold technique, the boundary of the blood vessel shape is grown into an area of pixels with a probability higher than the first threshold. A binarized blood vessel mask may be generated based on the growth results.

Additionally, the post-processing step may include: fine-tuning the blood vessel mask to align blood vessel boundaries with the image gradient of the fundus image.

To this end, the post-processing steps include: detecting a valley between a blood vessel and its external border in the binarized image; And it may also include filling the inside of the valley. When the binarized image is inverted, the area between the blood vessel and the external border is more clearly distinguished. For example, in Figure 5b, a number of regions (e.g., the area between the blood vessel and the outer border showing the blood vessel interior, outer border, and/or blood vessel thickness, etc.) that were identically expressed as blood vessel areas are shown in Figure 5c. are more clearly distinguished. In this inverted binarized image, the valley area between the blood vessel and the external border is detected, and image processing is performed to fill the inside of the valley area. Then, as shown in FIG. 5D, a mask that improves the initial blood vessel mask based on the blood vessel probability map may be generated as the final blood vessel mask. The process of filling the inside of the valley in step S410 may be performed, for example, through image erosion processing.

When these post-processing processes are applied to the blood vessel shape calculated by the segmentation model in the first fundus image and/or the second fundus image, the blood vessel mask of the more elaborately post-processed first fundus image and/or the second fundus image is created. is created (S410).

In addition, the step (S400) includes: matching the first fundus image and the second fundus image based on the feature points of the first fundus image and the second fundus image (S430); and a step of matching the first fundus image and the second fundus image based on the generated blood vessel mask (S450). Since the matching operation of step S450 is performed after the feature point matching of step S430, it is treated as a re-matching operation.

Step S430 includes extracting feature points in the fundus image; And a step of matching the first fundus image and the second fundus image based on the extracted feature points.

The feature point includes a point that is distinguishable from the surrounding background and easy to identify when matching the image. As described above, geometric feature points may be used as feature points. In one embodiment, the feature points may include feature points related to the geometric shape of the blood vessel. For example, it may include feature points included in the shape of a blood vessel or an edge.

In step S430, feature points may be extracted from the input image (eg, the first fundus image) through various feature extraction algorithms based on feature descriptors. The feature extraction algorithm may include, for example, SIFT (Scale Invariant Feature Trnasform), which extracts points with maximum cornerability on the image axis and/or scale axis based on DoG (Difference of Gaussian), but is limited to this. It won't work. In step S430, a feature point set including a plurality of feature points may be extracted.

In step S430, feature point-based matching may be performed in the pixel domain.

In the step (S430), the first set of feature points extracted from the first fundus image and the set of second feature points extracted from the first fundus image are each rigidly registered, thereby creating the first fundus image and the second fundus image. can also be matched.

In one embodiment, step S430 may include matching the first set of feature points and the second set of feature points using a perspective transformation matrix. In some embodiments, step S300 may further use the RANSAC algorithm to match the first set of feature points and the second set of feature points. For example, in the process of matching the first set of feature points and the second set of feature points using a perspective transformation matrix, feature point-based matching can be performed by calculating a value that maximizes the matching degree between feature points of each set through the RANSAC algorithm. It may be possible.

Figures 6a and 6b are diagrams showing feature point-based matching results according to an embodiment of the present invention.

As shown in FIG. 6A, the first fundus image and the second fundus image are matched based on the feature points of the blood vessels (S430).

Referring to FIG. 6B, a registered image that visualizes the registration result of FIG. 6A, in which the first and second fundus images are registered by matching feature points of blood vessels, may be obtained.

After matching in step S430, matching in step S450 is additionally performed.

In one embodiment, the matching method may further include: after feature point-based matching, verifying the validity of the feature point-based matching result (S440). Additional matching in step S450 is performed if validity is verified in step S440.

In one embodiment, for validation purposes, a fundus image (eg, a second fundus image) with the next matching rank may be subjected to homography conversion (S440). When the second fundus image is subjected to homography conversion, feature points of the second fundus image are projected onto another fundus image (ie, the first fundus image). For the second fundus image, if the change before and after homography transformation does not meet the validity condition, the second fundus image is not registered and is not used to generate a wide-angle fundus image.

In one embodiment, the validity condition may be set to be satisfied when the area difference between the image frames before and after homography conversion is less than a preset threshold. The threshold is the difference in pixel area of the image frame before and after homography conversion, and may be the area change amount itself or the area change rate. When the area change rate is used as the threshold, the threshold may be set to a value of approximately 15% to 5%. For example, the threshold may be 10%. Then, if the difference between before and after homography conversion of the second fundus image is greater than 10%, the second fundus image is excluded from further registration and synthesis.

As a result, it causes registration inaccuracies, including the inadequacy of applying two-dimensional homography to represent the view transformation of a three-dimensional spherical object and the failure to match a limited number of feature points due to small overlap or insufficient texture. The problem is at least partially solved.

Referring again to FIGS. 3 and 4, in step S450, the first fundus image and the second fundus image are matched based on the blood vessel mask (or post-processed blood vessel mask).

In one embodiment, for re-registration after feature-based registration in step S430, the first and second fundus images are re-registered by deformable registration of the first and second blood vessel masks. Matching is also possible (S450).

Deformable registration is a non-linear form of registration in which the registration target can be freely modified. For example, a straight line matching object may be transformed into a curved, square, or circular shape.

The deformation matching algorithm used in step S450 may include, for example, the B-Spline algorithm. The B-Spline algorithm is a matching algorithm that iteratively optimizes in the direction of increasing pixel similarity between two images. The parameter that gives the highest similarity is determined to produce an optimized matching result.

As described above in step S410, when the first blood vessel mask and the second blood vessel mask are created by a segmentation model, the input of deformation matching is performed based on the more precisely extracted blood vessel region.

For example, if the B-Spline algorithm is applied using the first blood vessel mask as the source shape and the second blood vessel mask as the target shape, one or more control points of the first blood vessel mask are extracted and the first blood vessel mask is used as the target shape. The shape of may be modified to have an optimization curve that maximizes pixel similarity with the second blood vessel mask. Then, the first blood vessel mask and the second blood vessel mask may be re-registered so that errors existing in the feature point-based registration result are finely adjusted.

FIGS. 7A and 7B are diagrams showing blood vessel mask-based registration results according to an embodiment of the present invention.

Referring to FIG. 7A, in step S450, a more precise registration result may be obtained by modifying and registering the blood vessel mask to fine-tune errors existing in the feature point-based registration result.

If only rigid body registration is performed, some parts of the blood vessel may not be matched accurately (S430). For example, as shown in FIG. 6A, if only feature-based registration was performed, the registration may not have become relatively more precise as the distance from the optic nerve head increases. However, as shown in FIG. 7A, when deformation registration is further applied, the first fundus image and the second fundus image are non-rigidly registered, and an elaborate registration result may be obtained (S450).

Then, as shown in FIG. 4, a registered image in which parts of the first fundus image and the second fundus image overlap each other may be obtained. Additionally, as shown in FIG. 7B, a registered image that visualizes the result of registering the first and second fundus images so that they can be finely adjusted can be obtained.

A more accurate registered image can be obtained through two-stage registration consisting of feature point-based registration and transformation registration.

Additionally, the registration method includes a step (S500) of synthesizing the registered fundus images to generate a wide-angle fundus image (eg, by the synthesis unit 500). Based on the registration result of step S450, the first fundus image and the second fundus image used for registration in step S400 are synthesized into a single partially overlapping image (S500). The synthesized single image has an image frame that is expanded than the image frame of the synthesized first or second fundus image. As a result, a composite image having a wider view than the shooting area of a single fundus image is generated (S500).

As shown in FIG. 4, the composite image of step S500 consists of the overlapping portion of the first fundus image and the second fundus image and other portions.

In one embodiment, the step of combining the first fundus image and the second fundus image to generate the wide-angle fundus image includes: calculating a displacement vector from the registration of the blood vessel mask after B-spline optimization. ; And it may include applying the calculated displacement vector to the first fundus image or the second fundus image. The photomontage is expanded due to the application of displacement vectors. When a synthetic source image (e.g., a first fundus image) and a synthetic target image (e.g., a second fundus image) are designated, the displacement vector is generated by specifying a source shape (e.g., a first blood vessel mask) and a target shape (e.g., a first fundus image). 2 may correspond to a parameter that minimizes the sum of the distances between each point of the blood vessel mask.

The displacement vector may be calculated through a search on the distance transform (DT) of the edge shape of the target image with respect to the edge shape of the source image. For example, by calculating the sum of the distances at each point while moving the target shape of the target image in pixel units on the distance map with respect to the edge shape of the source image, a parameter that minimizes the sum of distances may be calculated.

The composite image generated in step S500 may be provided to the user.

In one embodiment, the step (S500) may include: minimizing the difference in image specifications between the first fundus image and the second fundus image.

The image specifications include the intensity, color, and/or contrast of the appearance in the image frame. Each image frame has individual image specifications due to changes in imaging factors that affect the fundus image, including color intensity of external light, scattering, eyelid opening, etc. Therefore, the image specifications may be different between the image to be additionally synthesized for the montage (i.e., the second fundus image) and the existing image (i.e., the first fundus image). When fundus images with different image specifications are matched, the overlapping areas have different pixel values. To solve this, the difference in image specifications is minimized (S500).

In one embodiment, the step of minimizing the difference in image specifications between the first fundus image and the second fundus image includes: removing the color distortion around the overlapped area by registration, The step of modifying the pixel of the second fundus image corresponding to the pixel to have the same intensity or color may be included.

The corresponding pixel relationship between each image is determined based on the pixel coordinates of the registered image. In step S500, the corresponding pixel is determined based on the pixel coordinates of the re-registered image.

Then, a composite image consisting of deformed pixels and including an overlapping area from which color distortion has been removed may be generated as a single image.

In one embodiment, the difference in image specification between the first fundus image and the second fundus image may be minimized by using a multi-resolution spline technique in the overlapping area and boundary between the first fundus image and the second fundus image. .

The multi-resolution spline technique may minimize the difference in image intensity between each layer of the image frame by assigning weights around the center of the pair of the first and second fundus images and then applying a Gaussian filter and a Laplacian filter. Then, as shown in FIG. 4, a wide-angle fundus image from which differences in image intensity are removed is obtained.

FIGS. 8A and 8B are diagrams illustrating a registered image with color distortion removed, according to an embodiment of the present invention.

The registration part of a feature point-based registration image is the feature point part. When blood vessel feature points are used, the registration portion includes the blood vessel portion. When the colors or intensities of the first and second fundus images are different, as shown in FIG. 8A, a registered image in which pixels around the registration area are corrected to a single color or intensity may be obtained.

In a blood vessel-based registered image, the registered part is the extracted blood vessel area. When the colors or intensities of the first and second fundus images are different, as shown in FIG. 8B, a registered image in which pixels around the registration part are corrected to a single color or intensity may be obtained.

As a result of eliminating these differences in image specifications, it is possible to provide a wide-angle fundus image in which the registration part has unity as a single image (S500).

The steps S400 and S500 are repeatedly performed on the fundus image with the next matching rank and the already generated composite image. Unless there is a fundus image that fails the validation, steps S400 and S500 are repeatedly performed according to the registration order for all of the plurality of fundus images acquired in step S100.

For example, when the initial fundus image is selected, the remaining fundus images are synthesized according to the registration ranking within the first group to the registration ranking within the fourth group, and the final wide-angle fundus image is generated.

In alternative embodiments, another fundus image to be added to the already synthesized wide-angle fundus image may be a sub-wide-angle fundus image synthesized by matching fundus images in different groups. For example, if the first sub-wide-angle fundus image is generated by matching the fundus images in the first group, and the second sub-wide-angle fundus image is generated by matching the fundus images in the second group, the first sub-wide-angle fundus image and A more expanded wide-angle fundus image may be generated by matching the second sub-wide-angle fundus image.

In this case, registration and synthesis of the first sub-wide-angle fundus image and the second sub-wide-angle fundus image may be performed through all of the sub-wide-angle fundus images, or may be performed through one fundus image among each sub-wide-angle fundus image. For example, when registering and combining the first sub-wide-angle fundus image and the second sub-wide-angle fundus image, the first one with the highest registration rank in the second group?? The fundus image and the last fundus image with the lowest matching rank within the first group may be used.

Figures 9 and 10 show the results of generating the final wide-angle fundus image according to the method of Figure 3.

9 and 10 are wide-angle fundus images generated from sets of different fundus images. As shown in FIGS. 9 and 10, according to the method of FIG. 3, a wide-angle fundus image accurately registered in the entire area is generated.

According to the embodiments described above The operation of the device and the fundus image registration method may be at least partially implemented as a computer program and recorded on a computer-readable recording medium. For example, implemented with a program product comprised of a computer-readable medium containing program code, which can be executed by a processor to perform any or all steps, operations, or processes described.

The computer may be a computing device, such as a desktop computer, laptop computer, notebook, smart phone, or the like, or any device that may be integrated. A computer is a device that has one or more alternative, special-purpose processors, memory, storage, and networking components (either wireless or wired). The computer may run an operating system such as, for example, Microsoft's Windows-compatible operating system, Apple's OS X or iOS, a Linux distribution, or Google's Android OS.

The computer-readable recording medium includes all types of recording and identification devices that store data that can be read by a computer. Examples of computer-readable recording media include ROM, RAM, CD-ROM, magnetic tape, floppy disk, and optical data storage and identification devices. Additionally, computer-readable recording media may be distributed across computer systems connected to a network, and computer-readable codes may be stored and executed in a distributed manner. Additionally, functional programs, codes, and code segments for implementing this embodiment can be easily understood by those skilled in the art to which this embodiment belongs.

The present invention discussed above has been described with reference to the embodiments shown in the drawings, but these are merely illustrative examples, and those skilled in the art will understand that various modifications and modifications of the embodiments are possible therefrom. However, such modifications should be considered within the technical protection scope of the present invention. Therefore, the true technical protection scope of the present invention should be determined by the technical spirit of the attached patent claims.

The fundus image registration method according to one aspect of the present invention may use machine learning, one of the 4th industrial technologies, to extract a) blood vessel area.

In detecting changes in the fundus, a wide-angle fundus image can be easily obtained using a fundus camera through the above-described process without requiring a separate laser scanning device. The fundus camera does not require invasive procedures, does not use radiation, and has a short imaging time of less than a few minutes. The cost of fundus examination using this fundus camera is very cheap (current insurance fee is 7,990 won, general fee is 20,770 won). In addition, fundus cameras are distributed to almost all ophthalmological clinics and ophthalmological hospitals, and many examination centers also have them, making them highly accessible and popular.

If wide-angle fundus images can be created using fundus photography, which has these advantages, it is expected to achieve groundbreaking developments by supporting the interpretation of various eye diseases related to fundus changes.

Claims (20)

A method of registering a plurality of fundus images to generate a wide-angle fundus image, performed by a computing device including a processor, comprising:
Obtaining the plurality of fundus images of the subject's eye - at least one fundus image of the plurality of fundus images partially overlaps with at least one other fundus image;
Detecting at least one of an optic disc and a fovea in the plurality of fundus images and calculating the position of at least one of the optic disc and the fovea;
classifying the plurality of fundus images into a plurality of groups including first to fourth groups depending on whether the optic nerve head and the fovea are detected;
assigning inter-group matching ranks to the plurality of groups;
Assigning an intra-group matching rank to the fundus images for each group;
Matching a first fundus image of the plurality of fundus images with a second fundus image; and
In order to generate a wide-angle fundus image, generating a composite image in which at least a portion of the first fundus image overlaps with at least a portion of the second fundus image based on a re-registration result - the first fundus image is an initial fundus image It is an image or a composite image that has already been created, and the second fundus image is a registration rank within the group, and is a fundus image with a current registration rank -;
The step of matching the first and second fundus images and generating a composite image is repeated until all fundus images included in each group are matched based on the inter-group registration ranking,
The step of assigning inter-group matching ranks to the plurality of groups is:
For each group, it includes assigning an inter-group matching rank in the order of the first group, the second group, the third group, and the fourth group, or the order of the first group, the third group, the second group, and the fourth group. do,
The step of assigning an intra-group matching rank to the fundus images for each group is:
For at least one group among the first to third groups, selecting a reference fundus image for each group; and
A step of assigning an intra-group matching rank to each fundus image of each group based on the distance between the optic nerve head detected in the reference fundus image and the optic nerve head detected in the fundus image included in each group,
As the inter-group matching rank, the highest intra-group matching rank in a group with n+1 rank (where n is a natural number between 1 and 3) is the inter-group matching rank, and the lowest in the group with n rank. A method characterized in that the matching ranking within the group is assigned to the following.
The method of claim 1, wherein the step of matching the first fundus image and the second fundus image based on feature points of the first and second fundus images comprises:
Extracting at least one feature point from the first and second fundus images through a feature descriptor that extracts features that are invariant to the size and rotation of the image;
sampling one or more feature points among the feature points extracted from the first and second fundus images; and
A method comprising rigid body registration of the first and second fundus images based on sampled feature points.
The method of claim 1, wherein generating the first blood vessel mask from the first fundus image comprises:
extracting the shape of at least some blood vessels of the first fundus image using a segmentation model based on region characteristics within the first fundus image; and
Generating a blood vessel mask having a shape corresponding to the blood vessel region,
The split model is,
A model learned based on the characteristics of the blood vessel region, and a method characterized in that it is learned to output the blood vessels in the input image as a vessel probability map.
The method of claim 3, wherein generating the first blood vessel mask from the first fundus image comprises:
Generating a binarized blood vessel mask by binarizing an image including the extracted blood vessel shape as a blood vessel area;
detecting a valley between a blood vessel and an external boundary by inverting an image including a binarized blood vessel mask; and
The method further comprising filling the interior of the valley so that the interior of the valley represents the blood vessel.
The method of claim 1, wherein re-registering the first fundus image and the second fundus image based on the first and second blood vessel masks comprises:
A method comprising the step of re-registering the first fundus image and the second fundus image by performing deformable registration of the first blood vessel mask and the second blood vessel mask.
The method of claim 1, wherein generating the wide-angle fundus image comprises:
Transforming pixels of a second fundus image corresponding to pixels of the first fundus image based on the re-registered image to have the same intensity or color as the pixels of the first fundus image; and
A method comprising generating a wide-angle fundus image including an area comprised of the deformed pixels.
The method of claim 1, wherein generating the first blood vessel mask from the first fundus image comprises:
When a plurality of first fundus images are acquired, matching the plurality of first fundus images based on feature points of each first fundus image;
extracting the shape of at least a portion of blood vessels in a first region from the first fundus image using a segmentation model based on region characteristics within the fundus image; and
A method comprising generating a first blood vessel mask based on the extracted blood vessel shapes of the plurality of first fundus images.
The method of claim 7, wherein the step of generating a first blood vessel mask based on the extracted blood vessel shapes of the plurality of first fundus images comprises:
Non-rigidly registering the extracted blood vessel shapes of the plurality of first fundus images; and
A method comprising generating the first blood vessel mask based on a plurality of registered blood vessel shapes.
The method of claim 8, wherein the step of non-rigid body registration,
A method characterized by matching in the direction of increasing pixel-wise similarity in the blood vessel probability maps of two different first fundus images.
delete The method of claim 1, wherein the step of detecting at least one of the optic nerve head and the fovea and calculating the location comprises:
Detect at least one of the optic nerve head and fovea in the image frame of the input fundus image using a deep learning model,
The deep learning model is:
A shared convolutional network that includes multiple convolutional layers and outputs a convolutional feature map;
A region designation network that generates an intermediate feature map by applying a sliding window method to the input convolutional feature map and predicts a region of interest where a candidate object is likely to be located based on the intermediate feature map; and
A method comprising an object detection network that includes a pooling layer and a fully connected layer and detects the optic nerve head or fovea in the region of interest based on the feature vector for the region of interest in the image frame initially input to the detection model.
The method of claim 1, wherein classifying the plurality of fundus images into a plurality of groups includes:
Classifying the fundus image in which both the optic nerve head and the fovea are detected into a first group;
Classifying the fundus image in which only the optic nerve head is detected into a second group;
Classifying the fundus image in which only the fovea is detected into a third group; and
A method comprising at least one step of classifying the fundus image in which neither the optic nerve head nor the fovea is detected into a fourth group.
delete According to paragraph 1,
The method of the reference fundus image of the first or second group is characterized in that the position of the detected optic nerve head of each fundus image is closest to the position of the center of the image frame.
According to paragraph 1,
The reference fundus image of the third group is characterized in that the detected foveal position of each fundus image in the third group is closest to the position of the image center.
According to clause 12,
The step of assigning the within-group registration rank to the fundus images for each group is:
Comprising the step of assigning a matching rank within the fourth group to the fundus image for each fourth group,
A method characterized in that the matching rank within the fourth group is assigned based on the degree to which the feature points of each fundus image within the fourth group match the feature points of the first fundus image.
delete A computer-readable recording medium that is readable by a computing device and stores program instructions operable by the computing device, wherein when the program instructions are executed by a processor of the computing device, the processor operates according to claims 1 to 9. A computer-readable recording medium for performing a method of registering a plurality of fundus images to generate a wide-angle fundus image according to any one of claims 11, 12, and 14 to 16.
delete An image acquisition unit that acquires a plurality of fundus images captured by a fundus camera - at least one fundus image among the plurality of fundus images partially overlaps with at least one other fundus image;
Before matching the plurality of fundus images, at least one of the optic disc and the fovea is detected in the plurality of fundus images to calculate the position of at least one of the optic disc and the fovea, and the optic nerve Depending on whether the papilla and fovea are detected, the plurality of fundus images are classified into a plurality of groups including first to fourth groups, an inter-group matching rank is assigned to the plurality of groups, and the fundus images for each group are assigned. a classification unit that assigns a matching rank within a group;
For the fundus images included in each group, a registration unit that matches the fundus image with the current registration rank with a synthetic image generated by matching at least one fundus image up to the previous registration rank based on the registration rank within the group. ; and
In order to generate a wide-angle fundus image, it includes a synthesis unit that generates a new composite image from a composite image generated by matching the fundus image with the current matching rank with at least one fundus image up to the previous matching rank based on the matching result; ,
The classification unit assigns the inter-group matching rank to each group in the order of the first group, the second group, the third group, and the fourth group, or the order of the first group, the third group, the second group, and the fourth group. do,
For at least one group among the first to third groups, a reference fundus image for each group is selected, and the distance between the optic nerve head detected in the reference fundus image and the optic nerve head detected in the fundus image included in each group is adjusted. Based on this, an intra-group matching rank is assigned to each fundus image in each group,
As the inter-group matching rank, the fundus image having the highest intra-group matching rank in the group with a rank of n+1 (where n is a natural number between 1 and 3), and as the inter-group matching rank, having a n rank. A device characterized in that it assigns the next within-group registration rank of the fundus image having the lowest within-group registration rank in the group.

Images