KR20190128292A - Method and System for Early Diagnosis of Glaucoma and Displaying suspicious Area - Google Patents

Abstract

The present invention relates to a method and a system which use deep learning of a convolutional neural network (CNN) method to diagnose a progress degree of glaucoma from the thickness of a retinal nerve fiber layer (p-RNFL) around an optic disk of optical coherence tomography (OCT) of an eyeball. An image of the retinal nerve fiber layer (p-RNFL) around the optic disk acquired by optical coherence tomography (OCT) is diagnosed as normal, early glaucoma, and severe glaucoma by applying the convolutional neural network (CNN) method in machine learning, and an area of an image becoming a reason for making the diagnosis is displayed by using a gradient class activation map (Grad-CAM) on a model. Therefore, the present invention can play a role of accurate and quick auxiliary diagnosis.

Description

Method for early diagnosis and indication of suspected glaucoma and system for it {Method and System for Early Diagnosis of Glaucoma and Displaying suspicious Area}

The present invention relates to a method and system for diagnosing glaucoma from the thickness of the optic nerve papillary retinal nerve fiber layer of the ocular optical coherence tomography using deep learning, and more particularly, to the convolutional neural network (CNN) method. The present invention relates to a method and system for diagnosing the progression of glaucoma from the thickness of the retinal nerve fiber layer around the optic nerve papilla using deep learning.

Glaucoma is a disease that causes glaucoma changes in the optic nerve and thus visual field disorders, and progresses gradually over time, and blindness occurs without proper treatment. Unlike other eye diseases such as cataracts, once the optic nerve is damaged, it is impossible to restore the optic nerve even by surgery. Therefore, it is necessary to diagnose the glaucoma early and maintain treatment and management to maintain vision.

Glaucoma (glaucoma) is a progressive disease that causes optic nerve damage and visual field impairment, resulting in functional changes such as narrowing the field of vision with progressive loss of axons for some reason in the retinal nerve fiber layer (RNFL). In order to measure these functional changes through visual field examination, nerve fibers must be damaged by 30 to 50%, and prior to this field change, a change in the area called the optic nerve head (ONH) is known. Therefore, it is important to observe the change of optic nerve papilla in order to predict the progress and progression of glaucoma early.

The optic nerve papilla is a part of the bundle of nerve fibers that connect the retina with the brain just before exiting the eye, and is a bright full moon shaped circle that distinguishes it from the surrounding retina. The central part of the optic nerve papilla is dented in the form of a puddle when it is normal (depressed), and the area is light and pale with less blood vessel distribution. Around the depression, orange or pink nerve tissue, made up of nerve fibers, is surrounded by a rim (neural frame). With glaucoma, the optic nerve fibers die off and the nerves shrink and the depressions become wider and deeper.

The optic nerve gathers in the optic nerve papilla like a thin thread of nerve fibers, the wires that connect the brain from the retina in the eye. If a part of the fiber is damaged, the nerve fiber in the area is not observed, and thus, early diagnosis of glaucoma can be performed through retinal nerve fiber layer imaging before visual field disorder occurs. Laser coherent optical coherence tomography (OCT) equipment can measure the thickness of the retinal nerve fiber layer, thus obtaining an image of the peripapillary retinal nerve fiber layer (p-RNFL) around the optic nerve papilla. However, the prior art has a limitation in that it is impossible to provide an objective early glaucoma diagnostic data with the image information.

US Pat. No. 9,554,755 relates to “Methods, Systems and Computer-readable Media for Predicting Early Expression Glaucoma”, which is obtained by Time- and Spectral Domain Optical Coherence Tomography (ON-H). A diagnostic technique for early glaucoma combining the retinal nerve fiber layer (p-RNFL) and macular ganglion cell-inner plexiform layer (GCIPL) parameters around the optic nerve papilla is disclosed. However, the patent puts a value derived from SD-OCT equipment, not an image, and infers a glaucoma discrimination score according to a specific calculation formula, which is complicated and not intuitive.

The present invention was devised to solve the above problems, and classifies an image by applying deep learning to a retinal nerve fiber layer (p-RNFL) image around an optic nerve papilla among optical coherence tomography (OCT) images. To provide a method and system for predicting the progression of glaucoma.

The present invention provides a method for early diagnosis of glaucoma and displaying a suspected area: the method may include imaging of a peripapillary retinal nerve fiber layer (p-RNFL) around the optic nerve head (ONH) by optical coherence tomography (OCT). Obtaining; The obtained retinal nerve fiber layer image around the optic nerve papilla is input to a convolutional neural network (CNN) structure during deep learning, classified into glaucoma, early glaucoma, or normal class, and then a feature map. ) And extracting a classification score for each grade; Acquiring a gradient class activation map (Grad-CAM) as a gradient of the classification value and matrix multiplication of the feature map; Displaying the progressive grade activity map as an image; And displaying a place of high activity in the progressive grade activity map as a suspicious region, wherein the CNN structure uses a back-propagation method to adjust a weight to estimate a grade of an image. To provide early diagnosis of glaucoma and indication of suspected areas.

In another aspect, the CNN structure extracts an image feature while moving a convolution filter on a plane, passes the extracted image through an active function Relu, and repeats the process of reducing the size by pooling. Provide a method for early diagnosis of glaucoma and indication of suspected areas using the characteristics obtained.

을 알고리즘으로 구현하는, 녹내장 조기진단과 의심영역 표시 방법을 제공한다.In another aspect, the present invention, the step of obtaining the Gradient Class Activation Map (Grad-CAM),

A method for early diagnosis of glaucoma and the display of suspicious areas.

The present invention also provides a system for early diagnosis of glaucoma and a suspected area display system, the system including a management server having a database and a computer device including a display connected to the management server via a wired or wireless network, wherein the management server includes: And an image acquisition unit configured to store a peripapillary retinal nerve fiber layer (p-RNFL) image around an optic nerve papilla (ONH) obtained by an optical coherence tomography (OCT) device according to the instructions of the computer device. The retinal nerve fiber layer image around the optic nerve papilla stored in the image acquisition unit is input to a convolutional neural network (CNN) structure during deep learning under the instruction of the computer device, and is classified into glaucoma, early glaucoma, or normal person. A classification and display unit for classifying and storing in the database and displaying on the display; A feature map and a classification value extraction unit for extracting a feature map by performing a composite product operation on the retinal nerve fiber layer images around the optic nerve papilla of the classification and display unit, and extracting the classification scores for each grade; A grade activity map acquisition unit for obtaining a gradual class activation map (Grad-CAM) by a gradient of the classification value and a matrix multiplication of the feature map under the instruction of the computer device; ; An active map image display unit for displaying the progressive grade activity map as an image on the display in response to an instruction of the computer device; And a suspicious area display unit which displays a place of high activity among the gradual grading activity maps as a suspicious area under the instruction of the computer device, wherein the CNN structure can adjust a weight to estimate a grading of an image. To provide a system for early diagnosis of glaucoma and indication of suspected areas, using back-propagation methods.

In another aspect, the CNN structure extracts an image feature while moving a convolution filter on a plane, passes the extracted image through an active function Relu, and repeats the process of reducing the size by pooling. It provides a system for early diagnosis of glaucoma and indication of suspected areas, utilizing the features obtained.

을 알고리즘으로 구현하는, 녹내장 조기진단과 의심영역 표시를 위한 시스템을 제공한다.In another aspect, the present invention, the step of obtaining the Gradient Class Activation Map (Grad-CAM),

A system for early diagnosis of glaucoma and indication of suspicious areas is implemented.

The present invention is applied to the retinal nerve fiber layer (p-RNFL) image around the optic nerve papilla acquired by optical coherence tomography (OCT) by applying the CNN (Convolutional Neural Network) method during deep learning normal (early Glaucoma) In addition to diagnosing with) and Severe Glaucoma, the gradual class activation map (Grad-CAM) is used to display the area of the image that is the reason for diagnosis on the model. Can act as an assistant diagnostic

FIG. 1 is a flow chart of a diagnostic interrogation region display algorithm using Grad-CAM and glaucoma diagnosis using CNN during deep learning.
Figure 2 is an optical coherence tomography (OCT) image of the retinal nerve fiber layer around the optic nerve papilla.
FIG. 3 shows normal, early glaucoma and severe glaucoma state images obtained by optical coherence tomography (OCT) of the retinal nerve fiber layer around the optic nerve papilla. FIG.
4 is a convolutional neural network (CNN) structure of deep learning applied to an embodiment of the present invention.
FIG. 5 shows an example of applying a Gradient Class Activation Map (Grad-CAM) to a white pelican image.
FIG. 6 shows a gradual class activation map (Grad-CAM) implemented for optical coherence tomography (OCT) imaging of the retinal nerve fiber layer around the optic nerve papilla.
7 illustrates a conceptual diagram of a system for early diagnosis of glaucoma and indication of a suspected area according to an embodiment of the present invention.

Various embodiments are disclosed with reference to the drawings. In the following description, numerous specific details are set forth in order to provide a thorough understanding in one or more embodiments. However, it should be appreciated that each implementation may be practiced without these specific details. The following description and the annexed drawings set forth in detail certain illustrative examples of one or more embodiments. However, these examples are exemplary and some of the various methods may be used in the principles of various embodiments and the description set forth is intended to include both such embodiments and equivalents.

Various embodiments and features will be presented by an apparatus that can include a number of parts and components. It should also be understood and appreciated that various devices may include additional components, components, and / or may not include all of the components, components discussed in connection with the drawings.

As used herein, "embodiment", "embodiment", "example" and the like should not be construed as any embodiment or design described above being better or advantageous than other embodiments or designs.

In addition, the term “or” is intended to mean an inclusive “or” rather than an exclusive “or”. In other words, unless specified otherwise or unambiguously in context, "X uses A or B" is intended to mean one of the natural implicit substitutions. That is, X uses A; X uses B; Or where X uses both A and B, "X uses A or B" can be applied in either case. It is also to be understood that the term "and / or" as used herein refers to and includes all possible combinations of one or more of the listed related items.

Also, the terms "comprises" and / or "comprising" mean that there is a corresponding feature, step, action, component, and / or component, but one or more other features, steps, actions, components, configurations It should be understood that it does not exclude the presence or addition of parts and / or groups thereof. Also, unless otherwise specified or in the context of indicating a singular form, the singular in the specification and claims should generally be interpreted as meaning "one or more."

1 is a flow chart of a diagnostic interrogation region display algorithm using glad-diagnosing glaucoma and Cd-CAM during deep learning according to an embodiment of the present invention. Glaucoma early diagnosis and suspected area display method according to an embodiment of the present invention, by using optical coherence tomography (OCT) image of the retinal nerve fiber layer (peripapillary retinal nerve fiber layer: p-RNFL) around the optic nerve papilla (ONH) Start at the stage of acquisition. In one embodiment of the present invention, the obtained retinal nerve fiber layer image around the optic nerve papilla is input into a convolutional neural network (CNN) structure during deep learning and classified into glaucoma, early glaucoma, or normal class. A feature map and a classification score for each grade are extracted. A gradient class activation map (Grad-CAM) is obtained from the extracted gradient of the classification value and matrix multiplication of the feature map. In one embodiment of the present invention, the progressive grade activity map is displayed as an image, and the place of high activity among the progressive grade activity maps is displayed as a suspect area. In addition, the CNN structure may use a back-propagation method so that a weight may be adjusted to estimate a grade of an image. In one embodiment of the invention, the CNN is trained, the learning is to put the image and if the correct answer of the image is, for example, glaucoma, the CNN structure automatically extracts the feature points in the image, matching the correct answer given by the label Proceed with varying weights.

2 is an optical coherence tomography (OCT) image of the retinal nerve fiber layer around the optic nerve papilla, and FIG. 3 is a normal (Omal) obtained by optical coherence tomography (OCT) of the retinal nerve fiber layer around the optic nerve papilla. Early Glaucoma and Severe Glaucoma Status Imaging. In one embodiment of the present invention, the image of the peripapillary retinal nerve fiber layer (p-RNFL) around the optic nerve papilla (ONH) obtained by optical coherence tomography (OCT) is the retinal nerve fiber layer around the optic nerve papilla The thickness is displayed as a color-map. In one embodiment of the present invention, the ground truth of glaucoma diagnosis was based on the opinions of three ophthalmologists and glaucoma specialists in Seoul National University Hospital.

4 is a convolutional neural network (CNN) structure of deep learning applied to an embodiment of the present invention. In one embodiment of the present invention, the CNN structure is the process of extracting the image feature while moving the convolution filter (planar convolution filter) on the plane, passing the extracted image through the active function Relu, the process of reducing the size by pooling Use the features obtained by repeating In one embodiment of the present invention, the CNN structure is a structure applied to classification, detection, segmentation, etc. of an image, and a convolution product is a right and a lower convolution filter. While moving, the image feature is extracted using the inner product between the image and the filter. In one embodiment of the present invention, the extracted image passes through an activation function called Relu. The feature maps that pass through the active function are reduced in size through a process called pooling, and are lined up with the features obtained after multiple times of the product, active function, and pooling process. Classification is performed by reducing the number of nodes as large as the number of classes through the operation of a fully connected layer. In one embodiment of the present invention, in the learning process, a weight value is adjusted to estimate an image of a class to be predicted through a method called back-propagation. In the present invention, p-RNFL image of OCT image is cropped for classification and used as input of CNN, and classified into three types of glaucoma, early glaucoma, and normal using CNN structure.

FIG. 5 shows an example of applying a Gradient Class Activation Map (Grad-CAM) to a white pelican image. In one embodiment of the present invention, the step of obtaining the Gradient Class Activation Map (Grad-CAM) is a mathematical formula.

Is implemented by an algorithm. In one embodiment of the present invention, after the classification is completed, the activity for each class is displayed as an image using an internal weight and a feature map, wherein the feature map is displayed on the image. It means the features created after the multiplication operation. In one embodiment of the present invention, the method for obtaining the progressive grade activity map (Grad-CAM) may include a feature map passed through a convolution and a score to be classified into a specific class for each grade. This is obtained using the product of the gradient of) and the feature map passed through the convolution. In one embodiment of the present invention, the progressive grade activity map (Grad-CAM) can be used in almost all CNN structures, overcoming the disadvantage that the existing structure of the grade activity map (CAM) cannot be used universally. . In this way, if the grading grade activity map (Grad-CAM) drawn in this way is overlapped with the original image size, it can be identified which part of the original image is classified as a specific class.

Figure 6 shows a Gradient Class Activation Map (Grad-CAM) implemented on optical coherence tomography (OCT) images of the retinal nerve fiber layer around the optic nerve papilla. 6 (a) is the original image, 6 (b) is an image showing a box with a lot of activation, 6 (c) is a progressive grade active map, 6 (d) is a progressive grade active map transparent This is an overlapping video. In the original image, red means a thick layer of retinal nerve fiber layer, and blue means a thin portion of retinal nerve fiber layer. Glaucoma is evaluated by comprehensively considering the location, extent, and pattern of the thinner retinal nerve fiber layer, that is, the blue area. Looking at the active region of the implemented Grad-CAM, the left side of Supero-Temporal (the upper left of FIG. 6 (c), the progressive grade activity map) and Infero-Temporal (the progressive grade activity map, FIG. You can see that a lot of activity occurs at the bottom). Through the image data, it is possible to confirm that the evidence for diagnosis of glaucoma on the deep learning and the area that the clinician considers important for the diagnosis of glaucoma are substantially coincident with each other. Pulling out the Grad-CAM area can mark the thinner part of the optic nerve as a box area, which can help the clinician see the suspected area and make a fast and accurate diagnosis of glaucoma. In the CNN structure trained in one embodiment of the present invention, a feature map and a classification score for each grade are extracted in a test step, and the formula shown in the above formula is implemented on an algorithm to incrementally grade. Activity map (Grad-CAM) was implemented.

7 illustrates a conceptual diagram of a system for early diagnosis of glaucoma and indication of a suspected area according to an embodiment of the present invention. In one embodiment of the invention, the system comprises a management server 1 having a database and a computer device 300 including a display, connected to the management server and a wired or wireless network 100, wherein the management The server 1, under the direction of the computer device, peripapillary retinal nerve around the optic nerve papilla (ONH) obtained by the optical coherence tomography (OCT) device 200, which may not belong to the components of the present invention. an image acquisition unit 10 receiving and storing a fiber layer (p-RNFL) image through the wired / wireless network; The retinal nerve fiber layer image around the optic nerve papilla stored in the image acquisition unit is input to a convolutional neural network (CNN) structure during deep learning under the instruction of the computer device, and is classified into glaucoma, early glaucoma, or normal person. A classification and display unit 20 for storing the information in the database and classifying the information into a display; A feature map and a classification value extraction unit for extracting a feature map and extracting the classification scores by performing a composite product operation on the retinal nerve fiber layer images around the optic nerve papilla of the grading and display unit. 30); A grade activity map acquisition unit for obtaining a gradual class activation map (Grad-CAM) by a gradient of the classification value and a matrix multiplication of the feature map under the instruction of the computer device; 40; An active map image display unit 50 displaying the progressive grade activity map as an image on the display in response to an instruction of the computer device; And a suspicious region display unit 60 for displaying a place of high activity among the gradual grade activity maps as a suspicious region under the instruction of the computer device, wherein the CNN structure has a weight to estimate the grade of the image. A back-propagation method is used to adjust, extract image features while moving a convolution filter on a plane, pass the extracted image through an active function Relu, and then adjust the size by pooling. Use the features obtained by repeating the reduction process. In addition, the step of obtaining the Gradient Class Activation Map (Grad-CAM) is,

Is implemented by an algorithm.

The above description has described the technical idea of the present invention by using one embodiment, and those skilled in the art to which the present invention pertains may make various modifications and changes without departing from the essential characteristics of the present invention. . Therefore, the embodiments described in the present invention are not intended to limit the technical spirit of the present invention but to describe the present invention, and the scope of the technical idea of the present invention is not limited by these embodiments. The protection scope of the present invention should be interpreted by the claims, and all technical ideas within the equivalent scope should be interpreted as being included in the scope of the present invention.

1. Management server with database
10. Image Acquisition Unit
20. Classification and display
30. Feature Map and Classification Value Extraction Unit by Grade
40. Rank Active Map Acquisition Unit
50. Active Map Image Display
60. Suspect Area Display
100. Wired and wireless network
200. Optical Coherence Tomography (OCT) Device
300. Computer device including display

Claims (6)

Early diagnosis and indication of suspected glaucoma:
The method includes acquiring an image of a peripapillary retinal nerve fiber layer (p-RNFL) around an optic nerve papilla (ONH) by optical coherence tomography (OCT);
The obtained retinal nerve fiber layer image around the optic nerve papilla is input into a convolutional neural network (CNN) structure during deep learning, classified into glaucoma, early glaucoma, or normal class, and then displayed as a feature map. ) And extracting a classification score for each grade;
Acquiring a gradient class activation map (Grad-CAM) as a gradient of the classification value and matrix multiplication of the feature map;
Displaying the progressive grade activity map as an image; And
And displaying a place of high activity in the progressive grade activity map as a suspicious area,
The CNN structure uses a back-propagation method to adjust a weight to estimate a grade of an image.
Early diagnosis of glaucoma and indication of suspected areas.
The method of claim 1,
The CNN structure extracts an image feature while moving a convolution filter on a plane, passes the extracted image through an active function Relu, and utilizes the feature obtained by repeatedly reducing the size by pooling. ,
Early diagnosis of glaucoma and indication of suspected areas.
The method of claim 1,
Obtaining the Gradient Class Activation Map (Grad-CAM),

Is implemented by an algorithm,
Early diagnosis of glaucoma and indication of suspected areas.
Glaucoma early diagnosis and suspected area indication system:
The system includes a management server having a database and a computer device including a display connected to the management server in a wired or wireless network,
The management server receives a peripapillary retinal nerve fiber layer (p-RNFL) image around the optic nerve papilla (ONH) obtained by an optical coherence tomography (OCT) device under the direction of the computer device. Image acquisition unit for receiving and storing;
The retinal nerve fiber layer image around the optic nerve papilla stored in the image acquisition unit is input to a convolutional neural network (CNN) structure during deep learning under the instruction of the computer device, and is classified into glaucoma, early glaucoma, or normal person. A classification and display unit for classifying and storing in the database and displaying on the display;
A feature map and a classification value extraction unit for extracting a feature map by extracting a feature map by performing a synthetic product operation on the retinal nerve fiber layer images around the optic nerve papilla of the classification and display unit;
A grade activity map acquisition unit for obtaining a gradual class activation map (Grad-CAM) by the gradient of the classification value and matrix multiplication of the feature map under the instruction of the computer device. ;
An active map image display unit for displaying the progressive grade activity map as an image on the display in response to an instruction of the computer device; And
A suspicious area display unit which displays a place of high activity among the progressive grade activity maps as a suspicious area under the instruction of the computer device,
The CNN structure uses a back-propagation method to adjust weights to estimate the grade of an image.
System for early diagnosis of glaucoma and indication of suspected areas.
The method of claim 4, wherein
The CNN structure extracts an image feature while moving a convolution filter on a plane, passes the extracted image through an active function Relu, and utilizes the feature obtained by repeatedly reducing the size by pooling. ,
System for early diagnosis of glaucoma and indication of suspected areas.
The method of claim 4, wherein
Obtaining the Gradient Class Activation Map (Grad-CAM),

Is implemented by an algorithm,
System for early diagnosis of glaucoma and indication of suspected areas.

Images