KR20230041867A - Apparatus and method for analysis of retinal disease based on deep learning - Google Patents

Abstract

The present invention relates to retinal disease diagnosis technology. More specifically, provided are a device and method for diagnosing retinal diseases based on deep learning. According to an embodiment of the present invention, retinal pigments can be detected with high accuracy through deep learning-based retinal pigment detection. Retinal diseases can be diagnosed only with retinal fundus imaging without separate ophthalmic diagnosis equipment.

Description

Apparatus and method for diagnosing retinal diseases based on deep learning {APPARATUS AND METHOD FOR ANALYSIS OF RETINAL DISEASE BASED ON DEEP LEARNING}

The present invention relates to retinal disease diagnosis technology, and more particularly, to a deep learning-based retinal disease diagnosis device and method.

Hereditary retinal disease (IRD) is a genetic disease that causes gradual deterioration of photoreceptors, and is a progressive disease in which visual acuity is permanently lost as retinal pigment epithelial cells are degenerated. Among them, retinitis pigmentosa (RP) can cause night blindness or loss of peripheral vision in the early stages due to continuous retinal pigment degradation, and can show symptoms of limited visual fields (VF) and reduced vision leading to complete vision loss, leading to early detection. And it is a disease in which early treatment is essential.

For the diagnosis of these retinal diseases, a retinal pigmentation detection method based on computer aided diagnosis (CAD) is generally used. However, most retinal pigmentation detection technologies use expensive optical coherence tomography (OCT), which can be costly.

The background art of the present invention is published in Korean Patent Registration No. 10-1887375.

The present invention provides a deep learning-based retinal disease diagnosis apparatus and method capable of detecting retinal pigments with high accuracy through deep learning-based retinal pigment detection.

The present invention provides an apparatus and method for diagnosing retinal diseases based on deep learning that can diagnose retinal diseases only with retinal fundus images without separate ophthalmic diagnosis equipment.

The technical problem to be achieved by the present invention is not limited to the above-mentioned technical problem, and other technical problems not mentioned can be clearly understood by those skilled in the art from the description below. There will be.

According to one aspect of the present invention, a device for diagnosing retinal diseases based on deep learning is provided.

An apparatus for diagnosing retinal diseases based on deep learning according to an embodiment of the present invention detects retinal pigments by performing pixel-based object classification on retinal fundus images based on an input unit that receives retinal fundus images and deep learning. It may include a retinal pigment detection unit and a diagnostic unit for diagnosing whether or not there is a retinal disease based on the detected retinal pigment.

According to another aspect of the present invention, a method for diagnosing retinal diseases based on deep learning is provided.

A method for diagnosing retinal diseases based on deep learning according to an embodiment of the present invention includes receiving a retinal fundus image and detecting retinal pigments by performing pixel-based object classification on the retinal fundus image based on deep learning. and diagnosing a retinal disease based on the detected retinal pigment.

According to an embodiment of the present invention, retinal pigments can be detected with high accuracy through deep learning-based retinal pigment detection.

In addition, according to an embodiment of the present invention, retinal diseases can be diagnosed only with retinal fundus images without separate ophthalmic diagnosis equipment.

The effects of the present invention are not limited to the above effects, and should be understood to include all effects that can be inferred from the description of the present invention or the configuration of the invention described in the claims.

1 to 6 are diagrams for explaining an apparatus for diagnosing retinal diseases based on deep learning according to an embodiment of the present invention.
7 is a view for explaining a method for diagnosing retinal diseases based on deep learning according to an embodiment of the present invention.
8 to 10 are diagrams illustrating retinal pigment detection performance of the deep learning-based retinal disease diagnosis apparatus according to an embodiment of the present invention.

Hereinafter, the present invention will be described with reference to the accompanying drawings. However, the present invention may be embodied in many different forms and, therefore, is not limited to the embodiments described herein. And in order to clearly explain the present invention in the drawings, parts irrelevant to the description are omitted, and similar reference numerals are attached to similar parts throughout the specification.

Throughout the specification, when a part is said to be "connected (connected, contacted, combined)" with another part, this is not only "directly connected", but also "indirectly connected" with another member in between. "Including cases where In addition, when a part "includes" a certain component, it means that it may further include other components without excluding other components unless otherwise stated.

Terms used in this specification are only used to describe specific embodiments, and are not intended to limit the present invention. Singular expressions include plural expressions unless the context clearly dictates otherwise. In this specification, terms such as "include" or "have" are intended to indicate that there is a feature, number, step, operation, component, part, or combination thereof described in the specification, but one or more other features It should be understood that the presence or addition of numbers, steps, operations, components, parts, or combinations thereof is not precluded.

1 to 6 are diagrams for explaining an apparatus for diagnosing retinal diseases based on deep learning according to an embodiment of the present invention.

Referring to FIG. 1 , the deep learning-based retinal disease diagnosis apparatus 100 includes an input unit 110, a retinal pigment detection unit 130, and a diagnosis unit 150.

The input unit 110 receives retinal fundus images. Here, the retinal fundus image may include at least one of the retina, retinal blood vessels, the optic disc, and the choroid.

The retinal pigment detection unit 130 detects retinal pigments by performing pixel-unit object classification on the retinal fundus image based on deep learning. Specifically, the retinal pigment detector 130 performs object classification using at least one model of a pre-learned neural network such as a single spatial fusion network (SSF-Net) or a triplet spatial fusion network (TSF-Net). Here, the SSF-Net and TSF-Net models are models specialized for semantic segmentation, an object classification technique.

,

및

와 스킵 연결 특징인

을 융합하여 아래의 수학식 1과 같이

를 생성한다.Referring to FIGS. 2 and 3, SSF-Net has three feature map scales, Scale-1, Scale-2, and Scale-4. In , object classification is performed using Dilated Convolution. Specifically, SSF-Net generates a feature map half the size of the input image by applying a dilation factor and stride to 1 in Scale-1, and scale-1. By applying a dilation factor and a stride of 2 at 2 (Scale-2), a feature map is created with a size of 1/4 of the input image, and a dilation factor at Scale-4 (dilation factor) and stride of 4 to create a feature map with a size of 1/8 of the input image. A total of three feature map scales are fused through skip connection. At this time, SSF-Net represents three scales.

,

and

Characterized by a skip connection with

By fusion, as shown in Equation 1 below

generate

[Equation 1]

=

+

,

=

+

,

의 마지막 단계에서 픽셀 분류(pixel classification)를 통해 망막색소 검출을 수행한다. 최종적으로, 망막색소검출부(130)는 SSF-Net를 이용하여 검출된 망막색소를 기반으로 입력 영상에서 색소(pigment) 및 배경(background)이 구분된 마스크를 생성한다. 도 4 내지 도 6을 참조하면, TSF-Net는 SSF-Net와 동일하게 3개의 특징맵 스케일(feature map scales)에서 확장된 컨볼루션(Dilated Convolution)을 사용하여 객체 분류를 수행한다. 구체적으로, TSF-Net는 스케일-1(Scale-1)에서 확장 팩터(dilation factor) 및 스트라이드(stride)를 1로 적용하여 특징맵(feature map)을 입력 영상의 절반 크기로 생성하고, 스케일-2(Scale-2)에서 확장 팩터(dilation factor) 및 스트라이드(stride)를 2로 적용하여 특징맵을 입력 영상의 1/4 크기로 생성하고, 스케일-4(Scale-4)에서 확장 팩터(dilation factor) 및 스트라이드(stride)를 4로 적용하여 특징맵을 입력 영상의 1/8 크기로 생성한다. SSF-Net generated

In the last step of , retinal pigment detection is performed through pixel classification. Finally, the retinal pigment detection unit 130 generates a mask in which pigment and background are separated from the input image based on the detected retinal pigment using the SSF-Net. Referring to FIGS. 4 to 6 , TSF-Net performs object classification using dilated convolution on three feature map scales like SSF-Net. Specifically, TSF-Net generates a feature map half the size of the input image by applying a dilation factor and a stride of 1 in Scale-1, and scale-1. By applying a dilation factor and a stride of 2 at 2 (Scale-2), a feature map is created with a size of 1/4 of the input image, and a dilation factor at Scale-4 (dilation factor) and stride of 4 to create a feature map with a size of 1/8 of the input image.

를 생성한다. Here, in the TSF-Net, a total of three feature map scales are fused through skip connection. At this time, unlike the SSF-Net, the TSF-Net achieves EF ( early fusion), IF (intermediate fusion), and LF (late fusion) are finally generated as shown in Equation 2 below

generate

[Equation 2]

=

=

Through this, TSF-Net can perform retinal pigment detection with high accuracy by enhancing spatial information of each scale.

의 마지막 단계에서 픽셀 분류(pixel classification)를 통해 망막색소 검출을 수행한다.TSF-Net is generated

In the last step of , retinal pigment detection is performed through pixel classification.

Finally, the retinal pigment detection unit 130 creates a mask in which pigment and background are separated from the input image based on the detected retinal pigment using the TSF-Net.

)을 계산하여 망막 질환의 진단을 수행한다. The diagnosis unit 150 diagnoses retinal disease based on the detected retinal pigment. Specifically, the diagnosis unit 150 divides the number of pigment pixels by the number of background pixels (Number of non-pigment pixels) as shown in Equation 3 below.

) is calculated to perform diagnosis of retinal disease.

[Equation 3]

=

=

At this time, the diagnosis unit 150 may diagnose a retinal disease if the calculated retinal pigment ratio is greater than a preset threshold value.

7 is a diagram for explaining a method for diagnosing retinal diseases based on deep learning according to an embodiment of the present invention.

Referring to FIG. 7 , the apparatus 100 for diagnosing retinal diseases based on deep learning in S710 receives retinal fundus images. Here, the retinal fundus image may include at least one of the retina, retinal blood vessels, the optic disc, and the choroid.

In S730, the apparatus for diagnosing retinal diseases based on deep learning 100 detects retinal pigments by performing pixel-based object classification on retinal fundus images based on deep learning. Specifically, the deep learning-based retinal disease diagnosis apparatus 100 uses at least one model of a pre-learned neural network such as a single spatial fusion network (SSF-Net) or a triplet spatial fusion network (TSF-Net) to classify objects carry out Here, the SSF-Net and TSF-Net models are models specialized for semantic segmentation, an object classification technique.

,

및

와 스킵 연결 특징인

을 융합한다. SSF-Net는 생성된

의 마지막 단계에서 픽셀 분류(pixel classification)를 통해 망막색소 검출을 수행한다. 최종적으로, 딥러닝 기반의 망막 질환 진단 장치(100)는 SSF-Net를 이용하여 검출된 망막색소를 기반으로 입력 영상에서 색소(pigment) 및 배경(background)이 구분된 마스크를 생성한다. SSF-Net is a Dilated Convolution on three feature map scales, Scale-1, Scale-2 and Scale-4. to perform object classification. Specifically, SSF-Net generates a feature map half the size of the input image by applying a dilation factor and stride to 1 in Scale-1, and scale-1. By applying a dilation factor and a stride of 2 at 2 (Scale-2), a feature map is created with a size of 1/4 of the input image, and a dilation factor at Scale-4 (dilation factor) and stride of 4 to create a feature map with a size of 1/8 of the input image. A total of three feature map scales are fused through skip connection. At this time, SSF-Net represents three scales.

,

and

Characterized by a skip connection with

to fuse SSF-Net generated

In the last step of , retinal pigment detection is performed through pixel classification. Finally, the deep learning-based retinal disease diagnosis apparatus 100 generates a mask in which a pigment and a background are separated from an input image based on the detected retinal pigment using SSF-Net.

Like SSF-Net, TSF-Net performs object classification using Dilated Convolution on three feature map scales. Specifically, TSF-Net generates a feature map half the size of the input image by applying a dilation factor and a stride of 1 in Scale-1, and scale-1. By applying a dilation factor and a stride of 2 at 2 (Scale-2), a feature map is created with a size of 1/4 of the input image, and a dilation factor at Scale-4 (dilation factor) and stride of 4 to create a feature map with a size of 1/8 of the input image.

를 생성한다. 이를 통해, TSF-Net는 각 스케일의 공간 정보(spatial information)를 강화하여 정확도 높은 망막색소 검출을 수행할 수 있다. Here, in the TSF-Net, a total of three feature map scales are fused through skip connection. At this time, unlike the SSF-Net, the TSF-Net achieves EF ( early fusion), IF (intermediate fusion) and LF (late fusion), and finally

generate Through this, TSF-Net can perform retinal pigment detection with high accuracy by enhancing spatial information of each scale.

의 마지막 단계에서 픽셀 분류(pixel classification)를 통해 망막색소 검출을 수행한다.TSF-Net is generated

In the last step of , retinal pigment detection is performed through pixel classification.

Finally, the deep learning-based retinal disease diagnosis apparatus 100 creates a mask in which pigment and background are separated from the input image based on the detected retinal pigment using TSF-Net. .

)을 계산하여 망막 질환의 진단을 수행한다. In S750, the deep learning-based retinal disease diagnosis apparatus 100 diagnoses retinal disease based on the detected retinal pigment. Specifically, the deep learning-based retinal disease diagnosis apparatus 100 divides the number of pigment pixels by the number of background pixels (Number of non-pigment pixels), and the retinal pigment ratio (

) is calculated to perform diagnosis of retinal disease.

At this time, the deep learning-based retinal disease diagnosis apparatus 100 may diagnose a retinal disease when the calculated retinal pigment ratio is greater than a preset threshold value.

8 to 10 are diagrams illustrating retinal pigment detection performance of the deep learning-based retinal disease diagnosis apparatus according to an embodiment of the present invention.

Here, the retinal disease diagnosis apparatus 100 based on deep learning proceeded with retinal pigment detection performance evaluation using an open database, RIPS (Retinal Images for Pigment Signs). The RIPS database includes images generated by a fundus camera with a resolution of 1440 × 2160 pixels, and includes 30 retinal fundus images of 4 people, totaling 120 images.

Referring to FIG. 8 , as a result of performing retinal pigment detection using an open database, the deep learning-based retinal disease diagnosis apparatus 100 has a mask (b) and a mask (b) manually detected by a professional ophthalmologist in a retinal fundus image (a). You can see that it creates matching masks (c,d).

9 and 10, sensitivity (Se), specificity (Sp), precision (Pr), F-score ( As a result of measuring F-Score) and accuracy (Acc), it was confirmed that the measured values generated through SSF-Net and TSF-Net were higher than those of existing image prediction methods. Through this, it can be confirmed that the retinal pigment detection performance of the deep learning-based retinal disease diagnosis apparatus 100 according to an embodiment of the present invention is excellent.

The above description of the present invention is for illustrative purposes, and those skilled in the art can understand that it can be easily modified into other specific forms without changing the technical spirit or essential features of the present invention. will be. Therefore, the embodiments described above should be understood as illustrative in all respects and not limiting. For example, each component described as a single type may be implemented in a distributed manner, and similarly, components described as distributed may be implemented in a combined form.

The scope of the present invention is indicated by the following claims, and all changes or modifications derived from the meaning and scope of the claims and equivalent concepts should be interpreted as being included in the scope of the present invention.

Claims (9)

In the deep learning-based retinal disease diagnosis device,
an input unit for receiving a retina fundus image;
a retinal pigment detection unit that detects retinal pigments by performing pixel-based object classification on the retinal fundus image based on deep learning; and
A deep learning-based retinal disease diagnosis device comprising a diagnostic unit for diagnosing whether or not there is a retinal disease based on the detected retinal pigment.
According to claim 1,
The retinal pigment detection unit
A deep learning-based retinal disease diagnosis device for detecting retinal pigments using at least one model of a pre-learned neural network, SSF-Net (single spatial fusion network) or TSF-Net (triplet spatial fusion network).
According to claim 2,
The retinal pigment detection unit
A deep learning-based retinal disease diagnosis device that generates a mask in which pigment and background are separated from an input image based on the detected retinal pigment.
According to claim 1,
The diagnosis department
A deep learning-based retinal disease diagnosis device that calculates a retinal pigment ratio by dividing the number of pixels of retinal pigment by the number of background pixels and diagnoses a retinal disease when the retinal pigment ratio is greater than a preset threshold value.
In the method for diagnosing retinal diseases based on deep learning,
Receiving a retinal fundus image as an input;
detecting retinal pigments by performing pixel-based object classification on the retinal fundus image based on deep learning; and
Deep learning-based retinal disease diagnosis method comprising the step of diagnosing whether or not there is a retinal disease based on the detected retinal pigment.
According to claim 5,
The step of detecting the retinal pigment
A deep learning-based retinal disease diagnosis method for detecting retinal pigments using at least one model of a pre-learned neural network, SSF-Net (single spatial fusion network) or TSF-Net (triplet spatial fusion network).
According to claim 6,
The step of detecting the retinal pigment
A deep learning-based retinal disease diagnosis method for generating a mask in which pigment and background are separated from an input image based on the detected retinal pigment.
According to claim 5,
The step of diagnosing whether or not there is a retinal disease based on the detected retinal pigment
A method for diagnosing retinal diseases based on deep learning, which calculates a retinal pigment ratio by dividing the number of pixels of retinal pigment by the number of background pixels, and diagnoses a retinal disease when the retinal pigment ratio is greater than a preset threshold value.
A computer program that executes the deep learning-based retinal disease diagnosis method of claim 5 and is recorded on a computer-readable recording medium.

Images