WO2021241830A1

WO2021241830A1 - Deep neural network-based retinal image analysis method and apparatus for detection of abnormal kidney function

Info

Publication number: WO2021241830A1
Application number: PCT/KR2020/019362
Authority: WO
Inventors: 이용훈; 이기원; 남상민; 한현욱
Original assignee: 주식회사 스파이더코어
Priority date: 2020-05-27
Filing date: 2020-12-30
Publication date: 2021-12-02
Also published as: KR20210146690A; KR102410292B1

Abstract

Proposed is a deep neural network-based retinal image analysis method and apparatus for detection of abnormal kidney function. A deep neural network-based retinal image analysis method, according to one embodiment, comprises the steps of: pre-processing a retinal image so as to detect abnormalities in kidney function by using the retinal image; learning whether there are abnormalities in kidney function through a deep neural network by using the pre-processed retinal image; and reporting whether there are abnormalities in kidney function by using the learned deep neural network. The method reports whether there are abnormalities in kidney function and provides report grounds, thereby being able to help a user make a decision.

Description

Deep neural network-based retinal image analysis method and device to detect renal dysfunction

The following embodiments relate to a deep neural network-based retinal image analysis method and apparatus for detecting renal function abnormalities, and more specifically, by analyzing retinal images with artificial intelligence, a doctor for renal function abnormalities that could not be obtained with the naked eye. It relates to a method and apparatus for analyzing a retinal image based on a deep neural network that provides information.

Deep neural networks are being used for various medical image analysis. Regarding retinal imaging, there is a study to diagnose diabetic retinopathy. After learning with Inception-v3 (Non-Patent Document 1) using 128,175 photos obtained from 69,573 patients, it showed a diagnostic accuracy similar to that of an ophthalmologist. Recently, a study of predicting not only retinal disease but also cardiovascular disease with retinal imaging using a deep neural network has been reported.

Korean Patent No. 10-2058883 describes a method of analyzing iris images and retinal images with artificial intelligence to diagnose diabetes and prognostic symptoms.

The prior art has proposed an artificial intelligence technology for diagnosing diabetic retinopathy through retinal images. This is a technique that a doctor can do with the naked eye, and the algorithm shows performance results similar to that of a doctor. Accordingly, there is a need for a technology to diagnose kidney abnormalities through retinal images through specialized technology of artificial intelligence that cannot be judged with the naked eye of a doctor.

Korean Patent No. 10-2058883

<Non-patent literature>

(Non-Patent Document 1) Christian Szegedy, et. al. "Rethinking the inception architecture for computer vision." In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR). pp. 2818-2826, 2016.

(Non-Patent Document 2) Kaiming He, et. al. "Deep residual learning for image recognition." In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR). 2016.

(Non-Patent Document 3) Sergey Zagoruyko, and Nikos Komodakis. "Wide residual networks." In Proceedings of the British Machine Vision Conference (BMVC), 2016.

The embodiments describe a method and apparatus for analyzing a retinal image based on a deep neural network for detecting abnormalities in renal function, and more specifically, by analyzing retinal images with artificial intelligence, information on abnormalities in renal function that a doctor could not obtain with the naked eye. provide the technology provided.

The embodiments provide a deep neural network structure and learning method that can non-invasively diagnose kidney disease by using retinal images and patient medical information, so that simple image shooting without specialized personnel, samples, analysis equipment, etc. required for blood collection and blood analysis, etc. An object of the present invention is to provide a method and apparatus for analyzing a retinal image based on a deep neural network for detecting a renal function abnormality, which can detect a renal function abnormality only with only one.

A deep neural network-based retinal image analysis method using a computer-implemented deep neural network-based retinal image analysis apparatus includes the steps of: preprocessing a retinal image to detect abnormalities in renal function using a retinal image; learning whether there is an abnormality in renal function through a deep neural network using the pre-processed retinal image; and reporting the presence or absence of abnormality in renal function using the learned deep neural network, reporting the presence or absence of abnormality in renal function, and providing a reporting basis to help the user's judgment.

A deep neural network-based retinal image analysis apparatus according to another embodiment includes: a preprocessor for preprocessing a retinal image in order to detect abnormalities in renal function using a retinal image; a learning unit for learning whether there is an abnormality in renal function through a deep neural network using the pre-processed retinal image; and a determination unit for reporting abnormalities in renal function using the learned deep neural network, reporting the presence or absence of abnormalities in renal function, and providing a report basis to help the user in judgment.

According to embodiments, by providing a deep neural network structure and learning method that can non-invasively diagnose kidney disease using retinal images and patient medical information, simple It is possible to provide a method and apparatus for analyzing a retinal image based on a deep neural network that detects renal function abnormality, which can detect renal function abnormality only by imaging.

In addition, according to embodiments, it is possible to simultaneously detect a decrease in renal function due to diabetic nephropathy through fundus imaging, which is regularly performed in diabetic patients, so that it is possible to more effectively manage diabetic complications. A deep neural network-based retinal image analysis method and apparatus may be provided.

1 is a flowchart illustrating a method for analyzing a retinal image based on a deep neural network according to an embodiment.

2 is a diagram illustrating a block diagram of an apparatus for analyzing a retinal image based on a deep neural network according to an embodiment.

3 is a diagram illustrating an example of a structure of a deep neural network according to an embodiment.

4 is a diagram illustrating a method of embedding medical information according to an exemplary embodiment.

5 is a diagram illustrating a pre-processing of photo size adjustment according to an exemplary embodiment.

6A and 6B are diagrams illustrating an eGFR 60 reference ROC curve according to an embodiment.

7A and 7B are diagrams illustrating an eGFR 30 reference ROC curve according to an embodiment.

8A and 8B are diagrams illustrating an attention map of a deep neural network according to an embodiment.

9A and 9B are diagrams illustrating a future renal function prognosis/prediction ROC curve according to an embodiment.

10 is a diagram illustrating an example of a photo that is difficult to read by a deep neural network according to an embodiment.

Hereinafter, embodiments will be described with reference to the accompanying drawings. However, the described embodiments may be modified in various other forms, and the scope of the present invention is not limited by the embodiments described below. In addition, various embodiments are provided in order to more completely explain the present invention to those of ordinary skill in the art. The shapes and sizes of elements in the drawings may be exaggerated for clearer description.

The following examples are intelligent reading systems and digital diagnostic devices that utilize deep neural network (DNN)-based artificial intelligence. It is a health technology that makes

The embodiments relate to a method and apparatus for analyzing a retinal image based on a deep neural network for detecting abnormalities in renal function, and proposed an artificial intelligence technology for diagnosing kidney abnormalities through retinal images, which is an artificial intelligence that cannot be determined with the naked eye of a doctor. It is its own specialized technology. According to embodiments, not only a retinal image but also medical information of a patient may be used.

Referring to Figure 1, the deep neural network-based retinal image analysis method according to an embodiment, in order to detect abnormalities in renal function using the retinal image, the step of preprocessing the retinal image (S110), using the preprocessed retinal image Including the step (S120) of learning the presence or absence of abnormality in kidney function through the deep neural network, and the step (S130) of reporting the presence or absence of abnormality in kidney function using the learned deep neural network, reporting the presence or absence of abnormality in kidney function, , it can help users make decisions by providing reporting grounds.

According to embodiments, by providing a deep neural network structure and learning method that can non-invasively diagnose kidney disease using retinal images and patient medical information, simple Imaging alone can detect kidney dysfunction.

Below, each step of the deep neural network-based retinal image analysis method according to an embodiment will be described.

The deep neural network-based retinal image analysis method according to an embodiment may be performed by a computer system, for example, by a component that a processor of the computer system may include. As an example of the processor of the computer system, the deep neural network-based retinal image analysis method according to the embodiment may be described in more detail by taking the deep neural network-based retinal image analysis apparatus according to the embodiment.

Referring to FIG. 2 , the apparatus 200 for analyzing a retinal image based on a deep neural network according to an embodiment may include a preprocessor 210 , a learner 220 , and a determiner 230 .

In step S110 , the preprocessor 210 may preprocess the retinal image to detect abnormalities in renal function using the retinal image.

More specifically, the preprocessor 210 unifies the resolution of retinal images to use the retinal image in the deep neural network, measures the brightness, and removes the retinal image having a low brightness below a preset standard, thereby providing a high-quality retinal image. can be selected. Also, the preprocessor 210 may adjust the shape of the edge of the photo to be the same. And, the preprocessor 210 may search for the retina part, adjust the size so that the diameter of the searched retina becomes the same pixel, and then cut it out and use it.

In step S120 , the learning unit 220 may learn whether there is an abnormality in renal function through a deep neural network using the pre-processed retinal image.

In order to improve the performance of the deep neural network, the learning unit 220 may learn together with the deep neural network by combining medical information data of at least one patient among age, sex, diabetes, and hypertension. The learning unit 220 may learn a convolutional neural network (CNN) to predict the patient's medical information data along with the presence or absence of renal function abnormality. For example, the learning unit 220 may learn by combining the patient's age information with a feature vector of a convolutional neural network (CNN). Also, the learner 220 may embed the patient's medical information data in the same size as the feature vector and mask the feature vector. Also, the learning unit 220 may use the patient's medical information data as a label or use it as a channel of an input image. At this time, since age information has a continuous value, it can always be used by combining it with a feature vector of a convolutional neural network (CNN).

The learning unit 220 calculates the value and label obtained by passing the finally obtained feature vector through a fully-connected layer as cross-entropy to define and minimize the loss function to learn the neural network. have.

Also, the learning unit 220 may use an estimated Glomerular Filtration Rate (eGFR) value calculated by a CKD (Chronic Kidney Disease) calculation method from a serum creatinine concentration as a criterion for determining whether there is an abnormality in renal function. At this time, when the eGFR value is lower than 60, it can be reported as early abnormality, and when the eGFR value is lower than 30, it can be reported as intermediate stage abnormality.

In step S130 , the determination unit 230 may report the presence or absence of abnormality in renal function using the learned deep neural network. In this case, when determining whether the renal function is abnormal, the determination unit 230 may collect the results of the left and right eyes and report whether the renal function of the patient is abnormal. The determination unit 230 may report the presence or absence of abnormality in renal function, and may assist the user in determining by providing a report basis.

In the pre-processing process, pictures that are difficult to read due to low brightness were removed, but there are cases in which it is difficult to read due to the low quality of the pictures. Accordingly, the determination unit 230 may threshold the output value (predicted probability value) of the deep neural network indicating the kidney abnormality probability to evaluate the reliability of the fundus photo reading or report it as unreadable.

The determination unit 230 may analyze the retinal image through a deep neural network to report a prognosis and prediction for a future renal function as well as a current state of renal function. Here, the determination unit 230 may report three pieces of information simultaneously through one neural network, rather than independently reporting the prognosis and prediction of early abnormality, intermediate stage abnormality, and future renal function of the current renal function.

In addition, the determination unit 230 may calculate the gradient of the retina image used when learning the deep neural network, and display the portion carefully observed by the deep neural network on the retina image.

The embodiments propose a deep neural network structure and a learning method capable of non-invasively diagnosing a kidney disease by using a retinal image and medical information of a patient. The deep neural network structure used is based on the Wide Residual Network (WRN) (Non-Patent Document 3) that has improved the ResNet (Non-Patent Document 2) structure. The WRN structure may be composed of a residual block composed of several convolutional layers.

The deep neural network structure according to an embodiment may be configured based on WRN, and an example of the WRN structure may be shown as shown in FIG. 3 .

In order to better train the deep neural network learning, the patient's medical information data such as age, gender, presence of diabetes, and presence of hypertension can additionally be used. Additional information of the patient's medical information data may be utilized in various ways.

4A to 4D , examples of use of patient's medical information data, ie, a medical information embedding method, are shown according to an embodiment. As shown in FIG. 4A , for example, the patient's medical information data may be combined (Type 1) with a feature vector of a convolutional neural network (CNN). Here, as described above, the patient's medical information data may be age, gender, presence or absence of diabetes, and presence or absence of hypertension. In addition, the convolutional neural network (CNN) may be replaced with a deep neural network structure such as a recurrent neural network (RNN). As shown in FIG. 4B , the patient's medical information data may be embedded in the same size as the feature vector and masked (Type 2) in the feature vector. In addition, as shown in FIG. 4C , the patient's medical information data may be used as a label (Type 3), and as shown in FIG. 4D , the patient's medical information data is used as a channel of the input image (Type 4) ) can be Among them, age information has a continuous value, so it can always be used in combination with a feature vector of a convolutional neural network (CNN).

A neural network can be trained by defining and minimizing the loss function by calculating the value and label obtained by passing the finally obtained vector through a fully-connected layer as cross-entropy.

Additionally, in order to help the final judgment of the specialist, the deep neural network may include a function of displaying on the retina image the carefully looked at when determining whether there is a kidney abnormality. By calculating the gradient of the input picture with respect to the final output of the deep neural network, a pixel having a gradient value greater than or equal to a specific threshold may be displayed. The threshold may be adaptively applied according to the brightness of a given retinal image.

Example

A deep neural network for diagnosing kidney disease was learned using retinal images and patient medical information, and the performance of the learned neural network was confirmed. The data consisted of 266,579 retinal photographs of 75,663 patients. Additionally, there is medical information for each patient, and the information it contains is as follows.

[Table 1]

The entire data was randomly divided into a development set and a test set at a ratio of 9:1 based on the patient for the experiment. In this case, the development set contains 68,096 patients and the test set contains 7,567 patients. Then, the development set was again divided into a training set and a validation set at a ratio of 8:2, and learning was performed in a 5-cross validation method.

The collected retinal images were pre-processed to be used in deep neural networks. Since the resolution of retinal photos differs slightly depending on the device in which they were taken, the work of unifying the resolution was performed. First, the retina part was searched for through Hough transform, and the size was adjusted so that the diameter of the searched retina was 400 pixels, and then cut out and used. As a result, all photos were scaled to 400x400 size.

As shown in FIG. 5 , a pre-processing process for photo size adjustment can be summarized. Additionally, pictures that could not be obtained information due to low brightness were removed. The distribution was obtained by measuring the brightness of all pictures, and pictures having brightness lower than (mean) - 2 (standard deviation) were removed based on the brightness.

Renal function is calculated by calculating an estimated Glomerular Filtration Rate (eGFR) value from serum creatinine concentration.

[Equation 1]

Here, Scr means serum creatinine concentration, and A, B, and C are given in Table 2 below.

Table 2 shows parameter values in the CKD scheme.

[Table 2]

After setting eGFR < 60 for early detection of kidney function abnormalities and eGFR < 30 for diagnosing mid-stage abnormalities, the learned neural network structure was applied to the test patient group to measure AUC (Area Under the Curve) performance.

6A and 6B are diagrams illustrating an eGFR 60 reference ROC curve according to an embodiment. More specifically, Figure 6a shows the eGFR 60 reference ROC curve, Figure 6b shows the eGFR 60 reference ROC curve for the diabetic or hypertension patient group.

As shown in Fig. 6a, based on eGFR 60, a result of 93.09% (95% confidence interval: 92.59%-93.60%) was obtained.

In addition, as shown in FIG. 6B , a result of 86.66% (95% confidence interval: 85.53%-87.78%) was obtained for the diabetic or hypertension patient group.

7A and 7B are diagrams illustrating an eGFR 30 reference ROC curve according to an embodiment. More specifically, FIG. 7A shows an eGFR 30 reference ROC curve, and FIG. 7B shows an eGFR 30 reference ROC curve for a diabetic or hypertensive patient group.

As shown in Fig. 7a, a result of 96.67% (95% confidence interval: 96.10%-97.24%) was obtained based on eGFR 30.

In addition, as shown in FIG. 7B , a result of 93.87% (95% confidence interval: 92.67%-95.07%) was obtained for the diabetic or hypertension patient group.

Referring to FIGS. 8A and 8B , an example of an attention map is shown in which the learned deep neural network reads the presence or absence of kidney abnormality, which is carefully examined.

Referring to FIGS. 9A and 9B , data on patients who visited the hospital twice or more were additionally collected to predict/predict future renal function as well as the current status of renal function. Renal function at visit 1 was compared with renal function at visit 2, and data were labeled as maintained versus worsened.

As a result of self-test, the prognostic/predictive function of future renal function obtained an AUC result of 92.81% (95% confidence interval: 90.93% to 94.68%), and 87.05% (95% confidence interval: 83.29% to 90.81%) for the diabetic group. ) of AUC results were obtained.

Referring to FIG. 10 , in the pre-processing process, a photo that is difficult to read due to a low brightness of the photo is removed. In order to remove these pictures, the neural network output value representing the probability of kidney abnormality was thresholded (0.45≤y≤0.55).

Embodiments are health care technology that enables detection of abnormal kidney function belonging to nephrology in the field of health care through retinal images belonging to sensory science. According to embodiments, it is possible to detect kidney function abnormality only by simple imaging without professional personnel, samples, analysis equipment, etc. required for blood collection and blood analysis. In addition, it is possible to simultaneously detect a decrease in renal function due to diabetic nephropathy through fundus imaging, which is regularly performed in diabetic patients, so that diabetic complications can be managed more effectively.

The device described above may be implemented as a hardware component, a software component, and/or a combination of the hardware component and the software component. For example, devices and components described in the embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable array (FPA), It may be implemented using one or more general purpose or special purpose computers, such as a programmable logic unit (PLU), microprocessor, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to execution of the software. For convenience of understanding, although one processing device is sometimes described as being used, one of ordinary skill in the art will recognize that the processing device includes a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that can include For example, the processing device may include a plurality of processors or one processor and one controller. Other processing configurations are also possible, such as parallel processors.

The software may comprise a computer program, code, instructions, or a combination of one or more thereof, which configures a processing device to operate as desired or is independently or collectively processed You can command the device. The software and/or data may be any kind of machine, component, physical device, virtual equipment, computer storage medium or apparatus, to be interpreted by or to provide instructions or data to the processing device. may be embodied in The software may be distributed over networked computer systems, and stored or executed in a distributed manner. Software and data may be stored in one or more computer-readable recording media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the embodiment, or may be known and available to those skilled in the art of computer software. Examples of the computer readable recording medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floppy disks. - includes magneto-optical media, and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like.

As described above, although the embodiments have been described with reference to the limited embodiments and drawings, various modifications and variations are possible from the above description by those skilled in the art. For example, the described techniques are performed in a different order than the described method, and/or the described components of the system, structure, apparatus, circuit, etc. are combined or combined in a different form than the described method, or other components Or substituted or substituted by equivalents may achieve an appropriate result.

Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

Claims

In the deep neural network-based retinal image analysis method using a computer-implemented deep neural network-based retinal image analysis device,

Pre-processing the retinal image to detect abnormalities in renal function using the retinal image;

learning whether there is an abnormality in renal function through a deep neural network using the pre-processed retinal image; and

Reporting the presence or absence of abnormality in renal function using the learned deep neural network

including,

Reporting the presence or absence of abnormalities in the renal function and providing a basis for reporting to help the user's judgment

Characterized in, a deep neural network-based retinal image analysis method.
According to claim 1,

The step of pre-processing the retina image,

In order to use the retinal image in the deep neural network, unifying the resolution of the retinal images, measuring the brightness, and removing the retinal image having a low brightness below a preset standard

Characterized in, a deep neural network-based retinal image analysis method.
According to claim 1,

The step of learning whether there is an abnormality in kidney function through the deep neural network,

In order to improve the performance of the deep neural network, combining medical information data of at least one patient among age, sex, diabetes and hypertension with the deep neural network and learning together

Characterized in, a deep neural network-based retinal image analysis method.
4. The method of claim 3,

The step of learning whether there is an abnormality in kidney function through the deep neural network,

Learning the convolutional neural network (CNN) to predict the patient's medical information data along with the presence or absence of renal function abnormality

Characterized in, a deep neural network-based retinal image analysis method.
4. The method of claim 3,

The step of learning whether there is an abnormality in kidney function through the deep neural network,

Learning by combining the patient's age information with a feature vector of a convolutional neural network (CNN)

Characterized in, a deep neural network-based retinal image analysis method.
According to claim 1,

The step of learning whether there is an abnormality in kidney function through the deep neural network,

Using the estimated Glomerular Filtration Rate (eGFR) value calculated by the CKD (Chronic Kidney Disease) calculation method from the serum creatinine concentration as a criterion for determining whether the renal function is abnormal

Characterized in, a deep neural network-based retinal image analysis method.
7. The method of claim 6,

The step of reporting the abnormality of kidney function using the deep neural network is,

When the eGFR value is lower than 60, it is reported as early abnormality, and when the eGFR value is lower than 30, it is reported as intermediate stage abnormality

Characterized in, a deep neural network-based retinal image analysis method.
According to claim 1,

The step of reporting the abnormality of kidney function using the deep neural network is,

Calculating the gradient of the retinal image used during learning of the deep neural network and displaying the part carefully looked at by the deep neural network on the retina image

Characterized in, a deep neural network-based retinal image analysis method.
9. The method of claim 8,

The step of reporting the abnormality of kidney function using the deep neural network is,

Thresholding the output value of the deep neural network indicating the probability of kidney abnormality to evaluate the reliability of the picture reading or to report it as unreadable

Characterized in, a deep neural network-based retinal image analysis method.
According to claim 1,

The step of reporting the abnormality of kidney function using the deep neural network is,

When judging whether there is an abnormality in the renal function, collecting the results of the left and right eyes and reporting the abnormality of the renal function of the patient

Characterized in, a deep neural network-based retinal image analysis method.
According to claim 1,

The step of reporting the abnormality of kidney function using the deep neural network is,

Analyzing the retinal image through the deep neural network, reporting the prognosis and prediction for the current state of renal function and future renal function

Characterized in, a deep neural network-based retinal image analysis method.
According to claim 1,

The step of reporting the abnormality of kidney function using the deep neural network is,

To simultaneously report three pieces of information on the prognosis and prediction of early abnormality, intermediate stage abnormality, and future renal function of current renal function through one deep neural network.

Characterized in, a deep neural network-based retinal image analysis method.
In order to detect abnormalities in renal function using the retinal image, a preprocessor for preprocessing the retinal image;

a learning unit for learning whether there is an abnormality in renal function through a deep neural network using the pre-processed retinal image; and

Determination unit for reporting abnormalities in renal function using the learned deep neural network

including,

Reporting the presence or absence of abnormalities in the renal function and providing a basis for reporting to help the user's judgment

Characterized in, a deep neural network-based retinal image analysis device.
14. The method of claim 13,

The learning unit,

In order to improve the performance of the deep neural network, combining medical information data of at least one patient among age, sex, diabetes and hypertension with the deep neural network and learning together

Characterized in, a deep neural network-based retinal image analysis device.
14. The method of claim 13,

The learning unit,

Using the estimated Glomerular Filtration Rate (eGFR) value calculated by the CKD (Chronic Kidney Disease) calculation method from the serum creatinine concentration as a criterion for determining whether the renal function is abnormal

Characterized in, a deep neural network-based retinal image analysis device.