KR102313143B1 - Diabetic retinopathy detection and severity classification apparatus Based on Deep Learning and method thereof - Google Patents

Abstract

The present invention relates to an apparatus and method for detecting and classifying diabetic retinopathy based on deep learning.
According to the present invention, an image collecting unit for collecting fundus images of patients diagnosed with diabetic retinopathy, and inputting the collected fundus images to a Faster R-CNN based object recognition algorithm to detect the pathological characteristics of diabetic retinopathy When the learning unit for learning receives the fundus image of the subject who wants to be diagnosed with the disease, the inputted fundus image of the subject is inputted into the Faster R-CNN based object recognition algorithm that has completed learning, and a bounding box that sets the area to be identified as a pathological symptom A detection unit that detects by indicating, a signal processing unit that performs a pre-processing process for the pathological data, and inputs the pre-processed pathological data to a random forest classifier, and analyzes the input pathological data to detect diabetic retinopathy. and a severity classification unit for classifying the severity into a plurality of grades.
The device for detecting and classifying diabetic retinopathy according to the present invention can prevent the subjective interpretation intervention of an ophthalmic disease medical staff or examiner, make a specific diagnosis using objective data and indicators, and increase the efficiency of work in the medical image analysis field. .

Description

Diabetic retinopathy detection and severity classification apparatus Based on Deep Learning and method thereof

The present invention relates to an apparatus and method for detecting and classifying diabetic retinopathy based on deep learning, and more particularly, microvascular flow, retinal hemorrhage and To a device and method for detecting and classifying diabetic retinopathy based on deep learning that detects hard exudate and classifies the severity of the detected condition.

Diabetic retinopathy is a representative microvascular complication of diabetic patients and is one of the major causes of visual loss and blindness in patients in Korea, and about 60% of diabetic patients develop this disease. Although the prevalence decreases as the rapid treatment is performed through fundus examination, the detection and treatment tends to be delayed because there are no recognizable symptoms in the early stage of the patient.

In the case of non-proliferative diabetic retinopathy, the initial stage of diabetic retinopathy, pathological symptoms such as microaneurysm, hard exudate, soft exudate, and retinal hemorrhage can be observed through fundus examination. can Microvascular aneurysm is the first clinical finding observed in diabetic retinopathy. As diabetic retinopathy progresses, it expands abnormally, and in severe cases, it can develop into proliferative diabetic retinopathy.

In the case of retinal hemorrhage, the blood vessels in the retina rupture, resulting in blurred vision or reduced vision. At this time, in the case of hard exudate, serous fluid leaks from the retinal blood vessels and lipids remain, and the patient's blood cholesterol level can be estimated through the hard exudate.

In the Early Treatment Diabetic Retinopathy Study (ETDRS), diabetic retinopathy is classified into proliferative (PDR) and non-proliferative (NPDR), and non-proliferative is classified as No apparent retinopathy and Mild NPDR. ), moderate non-proliferative diabetic retinopathy (NPDR), severe non-proliferative diabetic retinopathy (Severe NPDR), and proliferative diabetic retinopathy (PDR).

Through this classification, different treatments are performed according to grades in ophthalmology, and since each treatment method is different, it is necessary to classify the severity of the retinal condition of the patient. However, when reading and classifying fundus images, the subjective interpretation of the examiner may be involved, and the brightness, contrast, and color tone may vary depending on the fundus imaging device.

Therefore, the need for research to improve the specificity and sensitivity for early diagnosis and classification of diabetic retinopathy is emerging.

The technology that is the background of the present invention is disclosed in Republic of Korea Patent Publication No. 10-1848321 (2018.04.20. Announcement).

The technical task of the present invention is to detect microvascular flow, retinal hemorrhage, and hard exudate, which are pathological features of diabetic retinopathy, using a photographed fundus image, and deep learning to classify the severity of the detected disease into grades. An object of the present invention is to provide an apparatus and method for detecting and classifying diabetic retinopathy based on the present invention.

According to an embodiment of the present invention for achieving this technical task, in the apparatus for detecting and classifying diabetic retinopathy based on deep learning, an image collecting unit that collects fundus images of patients diagnosed with diabetic retinopathy , a learning unit that learns the pathological characteristics of diabetic retinopathy by inputting the collected fundus images into an object recognition algorithm based on Faster R-CNN. A detection unit that inputs the fundus image to a Faster R-CNN based object recognition algorithm that has been completed learning and indicates an area determined as a pathological symptom with a set bounding box and detects it, a signal processing unit that performs a pre-processing on the pathological symptom data, and pre-processing and a severity classifier for inputting the completed pathology data into a random forest classifier, and classifying the severity of diabetic retinopathy into a plurality of grades by analyzing the input pathological data.

The detector may generate a bounding box for an area corresponding to at least one of retinal hemorrhage, minor exudate, and microvascular flow from the fundus image by using the feature map included in the object recognition algorithm.

The detection unit acquires pathology data corresponding to the bounding box, and the pathological data includes an image for the bounding box, a coordinate value for the bounding box, the number of pixels included in the bounding box, and diabetic retina. It may include at least one of a pathology type and a probability value corresponding to the diabetic retinopathy type.

The signal processing unit converts all pixel values in the bounding box to a grayscale having a plurality of levels of intensity values, inversely transforms the pixels changed to the grayscale, and applies a histogram smoothing process to the inversely transformed pixels to obtain a distribution of the contrast values. After the uniformity, the pixels in which the distribution of the light and dark values have become uniform may be binarized.

The signal processing unit derives coordinate values of the X and Y axes based on the bounding box, calculates a distance value between objects detected in the bounding box through the derived coordinate value, and performs pathology through the sum of the calculated distance values A distribution map of symptom data can be obtained.

The severity classification unit calculates a result value by randomly applying at least one of the number of pixels of the bleeding lesion included in the bounding box, the distribution of pathological symptoms, and the disease name characteristic information probability to a plurality of pre-constructed decision trees, The severity level may be classified using the result values calculated from the plurality of decision trees, respectively.

In addition, according to an embodiment of the present invention, in the method for detecting and classifying diabetic retinopathy using a device for detecting and classifying diabetic retinopathy, collecting fundus images of patients diagnosed with diabetic retinopathy; Learning how to analyze the pathological characteristics of diabetic retinopathy by inputting the collected fundus images into a Faster R-CNN based object recognition algorithm. Setting a bounding box by inputting the fundus image of the subject into an object recognition algorithm based on Faster R-CNN that has been trained, detecting pathological data for the set bounding box, and performing a pre-processing process for the pathological data and inputting the pre-processed pathological data into a random forest classifier, and analyzing the input pathological data to classify the severity of diabetic retinopathy into a plurality of grades.

As described above, according to the present invention, the classifier is trained and tested using the data signal processing process, the characteristics of the pathology, and the characteristics of machine learning in order to efficiently classify the test and the correlation between the corresponding pathology and the severity grade. By doing so, an appropriate classification result can be derived. Therefore, according to the present invention, it is possible to prevent the subjective interpretation intervention of an ophthalmic disease medical staff or examiner, to make a specific diagnosis using objective data and indicators, and to increase the efficiency of work in the field of medical image analysis.

1 is a configuration diagram for explaining an apparatus for detecting and classifying diabetic retinopathy according to an embodiment of the present invention.
2 is a flowchart illustrating a method for detecting and classifying diabetic retinopathy according to an embodiment of the present invention.
FIG. 3 is a flowchart for explaining step S220 shown in FIG. 2 .
4 is an exemplary view showing an example of the fundus image input in step S222 shown in FIG.
FIG. 5 is a diagram showing the structure of Faster R-CNN used to detect diabetic retinopathy pathological symptoms in step S223 shown in FIG. 3 .
6 is an exemplary diagram illustrating an example of an original fundus image input to Faster R-CNN in step S230 shown in FIG. 2 and a bounding box extracted from the original fundus image at a location where pathological symptoms occur.
7 is a view for explaining pathological symptom data for the bounding box extracted in FIG. 6 .
FIG. 8 is a flowchart for explaining step S240 shown in FIG. 2 .
9 is an exemplary diagram illustrating an example of steps S241 to S244 shown in FIG. 8 .

Hereinafter, preferred embodiments according to the present invention will be described in detail with reference to the accompanying drawings. In this process, the thickness of the lines or the size of the components shown in the drawings may be exaggerated for clarity and convenience of explanation.

In addition, the terms described below are terms defined in consideration of functions in the present invention, which may vary according to the intention or custom of the user or operator. Therefore, definitions of these terms should be made based on the content throughout this specification.

Hereinafter, an apparatus for detecting and classifying diabetic retinopathy according to an embodiment of the present invention will be described in more detail with reference to FIG. 1 .

1 is a configuration diagram for explaining an apparatus for detecting and classifying diabetic retinopathy according to an embodiment of the present invention.

As shown in FIG. 1 , the diabetic retinopathy detection and severity classification apparatus 100 includes an image collection unit 110 , a learning unit 120 , a detection unit 130 , a signal processing unit 140 , and a severity classification unit 150 . ) is included.

First, the image collection unit 110 collects the fundus image generated through a special fundus camera for photographing the retina located behind the eyeball. In this case, the collected fundus image refers to an image obtained by photographing the eyes of a patient diagnosed with diabetic retinopathy.

Then, the learning unit 120 inputs the fundus image collected by the image collection unit 110 to a previously built neural network model to learn the pathological characteristics of diabetic retinopathy. In other words, the learning unit 120 constructs an object recognition algorithm based on Faster R-CNN. Then, the learning unit 120 receives the fundus image collected in the constructed Faster R-CNN based object recognition algorithm, and learns the characteristics of microvascular flow, retinal hemorrhage, and hard exudate through the input fundus image.

Then, the detection unit 130 receives the fundus image of the subject to be diagnosed with the condition, and inputs the received fundus image of the subject to the R-CNN-based object recognition algorithm that has been trained. And, the detection unit 130 acquires the pathology data according to the Faster R-CNN based object recognition algorithm. Next, the detection unit 130 detects any one pathological symptom among microvascular flow, retinal hemorrhage, and hard exudate by using the acquired pathological symptom data.

At this time, the pathology data includes an image of a bounding box that makes a pathological symptom suspect, a coordinate value for the boundary area, the number of pixels included in the boundary area, the type of pathology, and a probability value corresponding to the condition. include

The signal processing unit 140 performs a pre-processing process on the acquired pathology data. In other words, the signal processing unit 140 purifies the data and performs preprocessing in order to transmit the pathological symptom data output from the detection unit 130 to the severity classification unit 150 . Accordingly, the signal processing unit 140 clearly distinguishes the background region and the symptom-appearing part for all pixels within the bounding box included in the pathological symptom data.

Finally, the severity classification unit 150 classifies the severity by inputting the pathology data preprocessed by the signal processing unit 140 into the random forest classifier. That is, the severity classification unit 150 classifies the diabetic disease into three grades by using the average value of the amount of information obtained from each tree in the set of decision trees constituting the random forest.

Hereinafter, a method for detecting and classifying diabetic retinopathy according to an embodiment of the present invention will be described in detail with reference to FIGS. 2 to 9 .

2 is a flowchart illustrating a method for detecting and classifying diabetic retinopathy according to an embodiment of the present invention.

As shown in FIG. 2 , first, the image collection unit 110 collects fundus images of patients diagnosed with diabetic retinopathy ( S210 ).

The fundus image represents an image of the retina located at the back of the eyeball. In the retina of a patient diagnosed with diabetic retinopathy, structural changes in retinal blood vessels occur due to high blood sugar. Therefore, hemorrhage, edema, and accumulation of exudate appear on fundus images of patients diagnosed with the disease.

Next, the learning unit 120 performs learning by inputting the plurality of fundus images collected by the image collecting unit 110 to the Faster R-CNN (S220).

Hereinafter, a learning process of Faster R-CNN will be described in more detail with reference to FIGS. 3 to 5 .

3 is a flowchart for explaining step S220 shown in FIG. 2 , FIG. 4 is an exemplary view showing an example of the fundus image input in step S222 shown in FIG. 3 , and FIG. 5 is S223 shown in FIG. 3 . It is a diagram showing the structure of Faster R-CNN used to detect diabetic retinopathy pathological symptoms in the step.

First, the learning unit 120 performs a labeling operation for diabetic retinopathy (S221).

For example, microvascular flow is labeled 0, retinal hemorrhage is labeled 1, and hard exudate is labeled 2.

Then, the learning unit 120 extracts learning data from the collected fundus image (S222).

As shown in FIG. 4 , the fundus image of a patient diagnosed with diabetic retinopathy shows symptoms of microvascular flow, retinal hemorrhage, and hard exudate. Therefore, the learning unit 120 extracts learning data from the received fundus image. Here, the learning data includes the overall size of the fundus image, the name of the object to be learned, that is, diabetic retinopathy, and the coordinates (Xmax, Ymax, Xmin, Ymin) of the object.

Then, the learning unit 120 inputs the extracted learning data to the Faster R-CNN to learn to set a bounding box indicating pathological symptoms (S223).

Faster R-CNN shares calculations for feature extraction for a region of interest (RoI) and detects objects by introducing a region proposal network (RPN) based on deep learning. In this case, the RPN is an algorithm that detects an object by selectively searching an area that is likely to include an object.

Therefore, as shown in FIG. 5 , the Faster R-CNN first configures a plurality of layers in the convolution process. And, Faster R-CNN extracts a feature map by repeatedly passing the input learning data through a plurality of pre-configured layers. Then, the Faster R-CNN delivers a feature map to the RPN and the region of interest pooling layer (RoI Pooling Layer), respectively. Then, the RPN acquires a large amount of windows formed in positions that are likely to include pathological symptoms, and detects the bounding box by searching the feature maps included in the acquired plurality of windows.

In addition, the region of interest pooling layer (RoI Pooling Layer) acquires a probability value corresponding to the diabetic retinopathy type for the detected object and a coordinate value for the bounding box.

After step S220 is completed, the detection unit 130 receives the fundus image of the subject to be diagnosed with the condition. And the detection unit 130 inputs the fundus image of the subject to be learned into the Faster R-CNN based object recognition algorithm to set a bounding box in the area determined as a pathological symptom, and the pathological symptom data for the set bounding box Extract (S230).

6 is an exemplary view showing an example of an original fundus image input to Faster R-CNN in step S230 shown in FIG. 2 and a bounding box extracted from the original fundus image at a location where pathological symptoms occur, It is a diagram for explaining pathological symptom data for the bounding box extracted in FIG. 6 .

First, (a) of FIG. 6 shows an original fundus image of a subject to be diagnosed with diabetic retinopathy, and (b) shows a state in which a plurality of bounding boxes are set in the original fundus image of the subject.

In more detail, the detector 130 inputs the original fundus image of the subject to Faster R-CNN. Then, Faster R-CNN sets a plurality of bounding boxes in the original fundus image of the subject according to the object recognition algorithm. And the detection unit 130 detects pathology data for each bounding box.

Here, the pathology data includes at least one of an image for the bounding box, a coordinate value for the bounding box, the number of pixels included in the bounding box, a diabetic retinopathy type, and a probability value corresponding to the diabetic retinopathy type. .

In other words, as shown in FIG. 7 , the coordinate value of the bounding box represents the position value of the bounding box and is expressed by four variables including (x, y, width, and height). And, the number of pixels varies depending on the size of the bounding box, and represents the number of pixels distributed throughout the bounding box. In addition, the diabetic retinopathy type is classified by a preset label value to classify the bounding box as a pathological symptom. Therefore, if the pathology included in the bounding box is microvascular flow, it is indicated as 0, if it is retinal hemorrhage, it is indicated as 1, and if it is hard exudate, it is indicated as 2.

Finally, the probability value corresponding to the diabetic retinopathy type represents the probability corresponding to the previously classified pathology.

When the pathology data detection is completed in step S230, the signal processing unit 140 purifies the detected pathology data and performs a pre-processing process (S240).

Hereinafter, a method of purifying pathological data and performing a pre-processing process will be described in more detail with reference to FIGS. 8 and 9 .

FIG. 8 is a flowchart for explaining step S240 shown in FIG. 2 , and FIG. 9 is an exemplary diagram illustrating an example of steps S241 to S244 shown in FIG. 8 .

8 and 9, the signal processing unit 140 converts the detected pathology data into gray scale (S241).

The signal processing unit 140 expresses a pixel value of a corresponding object as a 256-level contrast value (grayscale) in order to distinguish it from a background area for all pixels included in a rectangular area formed of a bounding box.

Next, the signal processing unit 140 inversely transforms the grayscale-converted pixel (S242).

That is, the signal processing unit 140 inversely transforms the pixel values expressed in gray scales of 256 steps (0 to 255) by applying Equation 1 below.

Here, x is a pixel value of any one selected pixel, and x' denotes an inversely transformed pixel value.

When step S242 is completed, the signal processing unit 140 applies a histogram smoothing process to the inversely transformed pixel (S243).

Here, histogram smoothing represents the process of redistribution of light and dark distributions and setting new values. Accordingly, the signal processing unit 140 performs histogram smoothing on the inversely transformed pixels to make the distribution of light and dark values uniform and to clearly distinguish boundaries. Accordingly, the signal processing unit 140 may separate the background region included in the bounding box and the microvascular and retinal hemorrhage regions.

Next, the signal processing unit 140 binarizes the pixels in which the distribution of light and dark values is uniform ( S244 ).

The signal processing unit 140 converts binary data represented by 0 and 1 using a basic threshold processing method. Then, the signal processing unit 140 obtains pixel data by separating only the region where the bleeding has occurred.

Finally, the signal processing unit 140 obtains the distribution of pathology data by using the coordinate values of the bounding box (S245).

In other words, the signal processing unit 140 derives the coordinate values of the X and Y axes based on the box of the extracted object in order to analyze the distribution of retinal hemorrhage and microvascular flow in the fundus image. And the signal processing unit 140 calculates the distance value between the objects by applying the derived coordinate value to Equation (2).

Here, X represents the X-axis value of the object, and Y represents the Y-axis value of the object.

And, the signal processing unit 140 acquires the distribution of pathology data through the sum of the calculated distance values. In this case, the pathological symptom data includes characteristic information transmitted to the classifier through a pre-processing process, and is used to classify the severity of diabetic retinopathy.

That is, the pathology data includes the number of pixels of microvascular flow and retinal hemorrhage, the maximum distance of each object, a diabetic retinopathy type, and a probability value corresponding to the diabetic retinopathy type.

When step S245 is completed, the severity classification unit 150 inputs the pre-processed pathological symptom data to the random forest classifier. Then, the random forest classifier analyzes the input pathology data, and the severity classifier 150 classifies the severity of diabetic retinopathy into a plurality of grades according to the result (S250).

In other words, the severity classification unit 150 constructs approximately nine decision trees. At this time, although the number of decision trees is set to 9 in the embodiment of the present invention, the number of decision trees is changed and applied according to the rules constituting the input variable and the decision tree without being limited thereto.

Then, the severity classification unit 150 randomly inputs the pre-processed pathological symptom data into the nine constructed decision trees. Here, the pathology data includes the number of pixels of bleeding lesions included in the bounding box, the distribution of pathological symptoms, and the probability of disease name characteristic information.

Then, the nine decision trees calculate each result value using the input pathology data. Then, the severity classifying unit 150 determines the severity of the disease according to the logic of the majority vote based on the calculated respective result values. In addition, the severity classifying unit 150 may represent the determined result in three grades: high, medium, and low.

As described above, the device for detecting and classifying diabetic retinopathy according to the present invention is a data signal processing process, pathological symptom characteristics, and machine learning in order to efficiently classify the test and grade for the correlation between the corresponding pathology and the severity grade. Appropriate classification results can be derived by training and testing the classifier using the characteristics of . Therefore, the apparatus for detecting and classifying diabetic retinopathy according to the present invention can prevent the intervention of an ophthalmic disease medical staff or examiner's subjective interpretation, make a specific diagnosis using objective data and indicators, and increase the efficiency of the medical image analysis field. can

Although the present invention has been described with reference to the embodiment shown in the drawings, which is merely exemplary, those skilled in the art will understand that various modifications and equivalent other embodiments are possible therefrom. will be. Therefore, the true technical protection scope of the present invention should be determined by the technical spirit of the following claims.

100: diabetic retinopathy detection and severity classification device
110: image collecting unit
120: study department
130: detection unit
140: signal processing unit
150: severity classification unit

Claims (13)

In the device for detecting and classifying diabetic retinopathy based on deep learning,
When a fundus image of a subject to be diagnosed is received, the fundus image of the subject is input into an object recognition algorithm based on a Faster R-CNN that has been learned, and the pathology of retinal hemorrhage, minor exudate and microvascular flow from the fundus image A detection unit that detects an area identified as a symptom with a set bounding box,
A signal processing unit that performs pre-processing on the pathological data, and
It includes a severity classification unit that inputs the pre-processed pathological data into a random forest classifier, analyzes the input pathological data, and classifies the severity of diabetic retinopathy into a plurality of grades,
The detection unit,
Acquire pathological symptom data in response to the bounding box,
The pathology data is
an image for the bounding box, a coordinate value for the bounding box, the number of pixels included in the bounding box, a diabetic retinopathy type, and a probability value corresponding to the diabetic retinopathy type,
The severity classification unit,
Calculate the result by randomly applying at least one from the number of pixels of bleeding lesions included in the bounding box, the distribution of pathological symptoms, and the disease name characteristic information probability to a plurality of pre-constructed decision trees,
A diabetic retinopathy detection and severity classification device for classifying a grade of severity using the result values calculated from the plurality of decision trees. According to claim 1,
An image collection unit for collecting fundus images of patients diagnosed with diabetic retinopathy, and
Diabetic retinopathy detection and severity classification apparatus further comprising a learning unit for learning the pathological characteristics of diabetic retinopathy by inputting the collected fundus image to a Faster R-CNN based object recognition algorithm. delete delete According to claim 1,
The signal processing unit,
All pixel values in the bounding box are converted to grayscale having multiple levels of intensity values, the pixels converted to the grayscale are inversely transformed, and a histogram smoothing process is applied to the inversely transformed pixels to make the distribution of the intensity values uniform, A diabetic retinopathy detection and severity classification device for binarizing pixels with a uniform distribution of light and dark values. 6. The method of claim 5,
The signal processing unit,
The coordinate values of the X and Y axes are derived based on the bounding box, the distance values between objects detected in the bounding box are calculated through the derived coordinate values, and the distribution of pathological symptom data is obtained through the sum of the calculated distance values. Acquiring diabetic retinopathy detection and severity classification device. delete In the diabetic retinopathy detection and severity classification method using a diabetic retinopathy detection and severity classification device,
Collecting fundus images of patients diagnosed with diabetic retinopathy,
learning a method of analyzing the pathological characteristics of diabetic retinopathy by inputting the collected fundus images into a Faster R-CNN based object recognition algorithm;
When a fundus image of a subject to be diagnosed is received, the fundus image of the subject is input into an object recognition algorithm based on a Faster R-CNN that has been learned, and the pathology of retinal hemorrhage, minor exudate and microvascular flow from the fundus image setting a bounding box in an area determined as a symptom, and detecting pathological symptom data corresponding to the set bounding box;
performing a pre-processing process on the pathology data, and
inputting the pre-processed pathological data into a random forest classifier, and analyzing the input pathological data to classify the severity of diabetic retinopathy into a plurality of grades,
The pathology data is
It includes an image for the bounding box, a coordinate value for the bounding box, the number of pixels included in the bounding box, a diabetic retinopathy type, and a probability value corresponding to the diabetic retinopathy type,
The step of classifying the severity is
Calculating a result value by randomly applying at least one of the pixel count of the bleeding lesion included in the bounding box, the distribution of pathological symptoms, and the disease name characteristic information probability to a plurality of pre-constructed decision trees, and
A method for detecting and classifying diabetic retinopathy, comprising the step of classifying a grade of severity using the result values calculated from the plurality of decision trees. delete delete 9. The method of claim 8,
The step of performing the pre-processing is,
converting all pixel values in the bounding box into grayscale having multiple levels of intensity values;
Inversely transforming the pixel changed to the grayscale;
Method for detecting and classifying the severity of diabetic retinopathy, comprising applying a histogram smoothing process to the inversely transformed pixels to make the distribution of light and dark values uniform, and then binarizing and outputting the pixels with the uniform distribution of light and dark values . 12. The method of claim 11,
The step of performing the pre-processing is,
The coordinate values of the X and Y axes are derived based on the bounding box, the distance values between objects detected in the bounding box are calculated through the derived coordinate values, and the distribution of pathological symptom data is obtained through the sum of the calculated distance values. Diabetic retinopathy detection and severity classification method further comprising the step of obtaining. delete

Images