KR102599464B1 - Apparatus and method for predicting fundus age using fundus image and axial length - Google Patents

Abstract

An apparatus and method for predicting fundus age using a fundus photograph and axial length are disclosed, and a method for predicting fundus age using a fundus photograph and axial length according to an embodiment of the present application includes a plurality of learning fundus photographs and corresponding Preparing a plurality of learning axial lengths and a plurality of learning age information to determine the fundus age of the subject of the fundus photograph using the plurality of learning fundus photographs, the plurality of learning axial lengths, and the plurality of learning age information. It may include generating a learning model for predicting, receiving a prediction target fundus photograph, and predicting the fundus age of the subject of the prediction target fundus photograph using the generated learning model.

Description

Device and method for predicting fundus age using fundus photograph and axial length {APPARATUS AND METHOD FOR PREDICTING FUNDUS AGE USING FUNDUS IMAGE AND AXIAL LENGTH}

This application relates to a device and method for predicting fundus age using fundus photographs and axial length. In particular, this study relates to a device and method for predicting fundus age, which is of high importance in early diagnosis of eye diseases, brain diseases, etc., using deep neural networks.

Fundus photographs are highly useful in the examination process at ophthalmology because they are non-invasive and can be easily obtained through basic examinations. In particular, fundus photographs can be taken with little effort and time even for patients who are relatively uncooperative or have reduced vision, and have the advantage of being easy to take even when the subject's pupils are not dilated (non-mydriatic). There is.

Additionally, fundus photography can be used to detect various retinal diseases. For example, in diagnosing various eye diseases such as diabetic retinopathy, retinal vein occlusion, and macular degeneration, a specialist's opinion based on fundus photographs is essential.

However, in the past, it was required to be equipped with an expensive fundus photography device to obtain fundus photographs, but recently, a technology has been developed to easily obtain fundus photographs using a lens mounted on a smartphone without an expensive device for fundus photography. It is being prepared.

The inner structures of the eye, including the retina, undergo changes depending on the patient's age, and the condition indicator of these structures, which is confirmed through the fundus, is called fundus age. Therefore, through the fundus age, it is possible to determine whether the patient's fundus changes are more severe or not compared to the patient's actual age. In addition, it has been reported that retinal blood vessels and conditions related to fundus age are related to cardiovascular diseases and brain diseases. In particular, in the case of young people where the patient's age is relatively young, if the fundus age prediction results are much larger than the actual age, suspicion of systemic vascular conditions and brain aging can be raised, and it can be used for early diagnosis of related diseases. Therefore, prediction of fundus age can be used as an indicator of health related to systemic blood vessels and brain aging.

The prior art discloses a method of diagnosing glaucoma using fundus photographs, but there is no technology for predicting fundus age that can be used to diagnose not only various retinal diseases but also systemic blood vessels and brain aging using fundus photographs. I couldn't do it.

The technology behind this application is disclosed in Korean Patent Publication No. 10-2019-0087272.

The purpose of the present application is to solve the problems of the prior art described above, and to provide a device and method for predicting fundus age more precisely using fundus photographs and axial length. Herein, the axial length may include a directly measured axial length or an axial length predicted through a separate deep neural network.

However, the technical challenges sought to be achieved by the embodiments of the present application are not limited to the technical challenges described above, and other technical challenges may exist.

As a technical means for achieving the above-mentioned technical problem, a method of predicting fundus age using a fundus photograph and an axial length according to an embodiment of the present application includes a plurality of learning fundus photographs, a plurality of learning axial lengths, and a plurality of learning axial lengths corresponding thereto. Preparing learning age information, generating a learning model that predicts the fundus age of the subject of the fundus photograph using the plurality of learning fundus photographs, the plurality of learning axial lengths, and the plurality of learning age information. , receiving the prediction target fundus photograph, and predicting the fundus age of the subject of the prediction target fundus photograph using the generated learning model.

In addition, the step of generating the learning model includes training a first deep neural network by specifying the plurality of learning fundus photographs as input values and the plurality of learning axial lengths as labels, and the learned first deep neural network Deriving a plurality of predicted axial lengths for the plurality of learning fundus photographs using , and specifying the plurality of learning fundus photographs and the plurality of predicted axial lengths as input values and using the plurality of learning age information as labels. It may include a step of specifying and training a second deep neural network.

Additionally, the learning model includes one or more deep neural networks, and each deep neural network includes an input layer, a feature extraction module, and a prediction module, and the step of generating the learning model includes inputting the plurality of learning fundus photographs. A step of training a first deep neural network by specifying a value and the plurality of learning axial lengths as labels, and deriving a plurality of predicted axial lengths for the plurality of learning fundus photographs using the learned first deep neural network. Step, specifying the plurality of learning fundus photos as input values, inserting the plurality of predicted axial lengths into any one layer of the feature extraction module of the second deep neural network, and specifying the plurality of learning age information as a label It may include training a second deep neural network.

In addition, the step of generating the learning model includes specifying the plurality of learning fundus photographs and the plurality of learning axial lengths as input values, and specifying the plurality of learning age information as labels to train a deep neural network. can do.

In addition, it further includes the step of receiving the measured axial length of the subject, and the step of predicting the fundus age includes specifying the prediction target fundus photograph and the measured axial length as input values of the deep neural network. It may be used to predict the fundus age of the subject.

In addition, it further includes the step of deriving a basis area used for prediction of the fundus age from the prediction target fundus photograph, and outputting a basis image highlighting the evidence area in the prediction target fundus photograph, The step of deriving the basis area is to calculate the contribution value representing the influence of any node of the I-th layer on any node of the I+1-th layer through Equation 1 below, and I through Equation 2 Calculate the contribution value representing the influence of any node in the lth layer on the entire I+1th layer, and calculate the contribution value representing the influence of any node in the lth layer on the entire l+2th layer through Equation 3. Calculate the contribution value. Therefore, in a deep neural network composed of multiple complex layers, Equation 3 can be used in chain to calculate the value that one arbitrary node contributes to the output node of the neural network to derive the basis area used for prediction.

[Equation 1]

[Equation 2]

[Equation 3]

In addition, receiving the corrected fundus age determined by a medical staff as feedback on the predicted fundus age, and adding a plurality of learning fundus photos and a plurality of learning axial lengths corresponding to the corrected fundus age to the dataset to model the learning model. The step of updating may further be included.

Meanwhile, an apparatus for predicting fundus age using a fundus photograph and an axial length according to an embodiment of the present application prepares a plurality of learning fundus photographs, a plurality of learning axial lengths corresponding thereto, and a plurality of learning age information, and the plurality of learning age information is provided. A learning unit that generates a learning model for predicting the fundus age of the subject of the fundus photograph using the learning fundus photograph, the plurality of learning axial lengths, and the plurality of learning age information, and receiving the prediction target fundus photograph, It may include an inference unit that predicts the fundus age of the subject of the prediction target fundus photograph using the generated learning model.

In addition, the learning unit trains a first deep neural network by specifying the plurality of learning fundus photographs as input values and the plurality of learning axial lengths as labels, and uses the learned first deep neural network to A second deep neural network is learned by deriving a plurality of predicted axial lengths for the learning fundus photographs, specifying the plurality of learning fundus photographs and the plurality of predicted axial lengths as input values, and specifying the plurality of learning age information as labels. It may be to do so.

In addition, the learning model includes one or more deep neural networks, and each deep neural network includes an input layer, a feature extraction module, and a prediction module, and the learning unit specifies the plurality of learning fundus photos as input values and A first deep neural network is trained by specifying a plurality of learning axial lengths as labels, a plurality of predicted axial lengths for the plurality of learning fundus photographs are derived using the learned first deep neural network, and the plurality of learning fundus Designating a photo as an input value, inserting the plurality of predicted axial lengths into one layer of the feature extraction module of the second deep neural network, and specifying the plurality of learning age information as a label to train the second deep neural network. It may be.

In addition, the learning unit may train a deep neural network by designating the plurality of learning fundus photographs and the plurality of learning axial lengths as input values and the plurality of learning age information as labels.

In addition, the inference unit may receive the measured axial length of the subject, and predict the fundus age of the subject by specifying the prediction target fundus photograph and the measured axial length as input values of the deep neural network. there is.

In addition, it further includes a visualization unit that derives a basis area used for predicting the fundus age from the prediction target fundus photo, and outputs a basis image highlighting the evidence area in the prediction target fundus photo, and the visualization unit , Calculate the contribution value representing the influence of any node of the I-th layer on any node of the I+1-th layer through Equation 1 below, and calculate the contribution value of any node of the I-th layer through Equation 2. Calculate the contribution value representing the influence of the entire I+1th layer, and calculate the contribution value representing the influence of any node in the lth layer on the entire l+2th layer through Equation 3. Therefore, in a deep neural network composed of multiple complex layers, Equation 3 can be used in chain to calculate the value that one arbitrary node contributes to the output node of the neural network to derive the basis area used for prediction.

[Equation 1]

[Equation 2]

[Equation 3]

In addition, the learning unit receives the corrected fundus age determined by the medical staff as feedback on the predicted fundus age, and adds a plurality of learning fundus photos and a plurality of learning axial lengths corresponding to the corrected fundus age to the dataset. This may be updating the learning model.

The above-described means of solving the problem are merely illustrative and should not be construed as intended to limit the present application. In addition to the exemplary embodiments described above, additional embodiments may be present in the drawings and detailed description of the invention.

According to the above-described means of solving the problem of the present application, it is possible to provide a fundus age prediction device and method that can predict the fundus age more precisely by using the axial length along with the fundus photograph.

According to the above-described means of solving the problem of our institute, by predicting the axial length using fundus photographs that can be easily obtained without expensive axial length measuring equipment, the axial length can be measured even in developing countries where it is difficult to have expensive ocular length measuring equipment or where the medical industry infrastructure is underdeveloped. Alternatively, it can support the simple and reliable acquisition of fundus age.

According to the above-described means of solving the problem of our hospital, it is possible to secure the reliability of the predicted fundus age by highlighting the areas of evidence that had a significant influence on the prediction of fundus age, and assist the accurate judgment of the medical staff who visually confirms this. can do.

However, the effects that can be obtained herein are not limited to the effects described above, and other effects may exist.

Figure 1 is a schematic configuration diagram of a fundus age prediction system including a device for predicting fundus age using a fundus photograph and axial length according to an embodiment of the present application.
Figure 2 is a conceptual diagram for explaining a deep neural network-based learning model according to an embodiment of the present application.
Figure 3 is a conceptual diagram for explaining a deep neural network-based learning model according to another embodiment of the present application.
Figure 4 is a conceptual diagram for explaining a deep neural network-based learning model according to another embodiment of the present application.
Figure 5 is a diagram for explaining a method of obtaining a basis area according to an embodiment of the present application.
Figure 6 is a diagram for explaining a method of obtaining a basis area for a special case according to an embodiment of the present application.
Figure 7 is a diagram illustrating an example of a ground image displayed by emphasizing a ground area in a fundus photograph according to an embodiment of the present application.
Figure 8 is a schematic configuration diagram of a device for predicting fundus age using a fundus photograph and axial length according to an embodiment of the present application.
Figure 9 is an operation flowchart of a method for predicting fundus age using a fundus photograph and axial length according to an embodiment of the present application.
Figure 10 is an operation flowchart of a method for generating a learning model for predicting fundus age according to an embodiment of the present application.
Figure 11 is an operation flowchart of a method for generating a learning model for predicting fundus age according to another embodiment of the present application.
Figure 12 is an operation flowchart of a method for generating a learning model for predicting fundus age according to another embodiment of the present application.
Figure 13 is an operation flowchart of a method for visualizing the basis area used for predicting fundus age according to one or another embodiment of the present application.
Figure 14 is an operation flowchart of a method for visualizing the basis area used for predicting fundus age according to another embodiment of the present application.

Below, with reference to the attached drawings, embodiments of the present application will be described in detail so that those skilled in the art can easily implement them. However, the present application may be implemented in various different forms and is not limited to the embodiments described herein. In order to clearly explain the present application in the drawings, parts that are not related to the description are omitted, and similar reference numerals are assigned to similar parts throughout the specification.

Throughout this specification, when a part is said to be “connected” to another part, this means not only “directly connected” but also “electrically connected” or “indirectly connected” with another element in between. "Includes cases where it is.

Throughout this specification, when a member is said to be located “on”, “above”, “at the top”, “below”, “at the bottom”, or “at the bottom” of another member, this means that a member is located on another member. This includes not only cases where they are in contact, but also cases where another member exists between two members.

Throughout the specification of the present application, when a part "includes" a certain component, this means that it may further include other components rather than excluding other components unless specifically stated to the contrary.

This application relates to a device and method for predicting fundus age using fundus photographs and axial length. In particular, our study relates to a device and method for predicting fundus age, which is of high importance in early diagnosis of eye diseases, brain diseases, etc., using big data on fundus photographs and deep neural networks.

Figure 1 is a schematic configuration diagram of a fundus age prediction system including a device for predicting fundus age using a fundus photograph and axial length according to an embodiment of the present application.

Referring to FIG. 1, the fundus age prediction system 10 is a device 100 for predicting fundus age using a fundus photograph and axial length according to an embodiment of the present application (hereinafter referred to as 'fundus age prediction device 100') '), a learning DB 200, and a fundus photography device 30. Although not shown, the fundus age prediction system 10 may further include an axial length measuring device.

The fundus age prediction device 100, the learning DB 200, and the fundus photography device 30 may communicate with each other through the network 20. The network 20 refers to a connection structure that allows information exchange between nodes such as terminals and servers. Examples of such networks 20 include the 3rd Generation Partnership Project (3GPP) network, Long Term Evolution) network, 5G network, WIMAX (World Interoperability for Microwave Access) network, Internet, LAN (Local Area Network), Wireless LAN (Wireless Local Area Network), WAN (Wide Area Network), PAN (Personal Area) Network), wifi network, Bluetooth network, satellite broadcasting network, analog broadcasting network, DMB (Digital Multimedia Broadcasting) network, etc., but are not limited thereto.

According to an embodiment of the present application, the fundus photography device 30 may include a user terminal 31. The user terminal 31 includes, for example, a smartphone, a SmartPad, a tablet PC, etc., as well as a Personal Communication System (PCS), a Global System for Mobile communication (GSM), a Personal Digital Cellular (PDC), and a PHS ( Personal Handyphone System), PDA (Personal Digital Assistant), IMT (International Mobile Telecommunication)-2000, CDMA (Code Division Multiple Access)-2000, W-CDMA (W-Code Division Multiple Access), Wibro (Wireless Broadband Internet) terminal It can be any type of wireless communication device, such as: As another example, the fundus photography device 30 may include a fundus camera 32, which is a dedicated imaging device provided at an ophthalmology clinic, university hospital, etc., to obtain a fundus photograph of a patient.

According to an embodiment of the present application, the fundus age prediction device 100 may include one or more computers, and if a plurality of computers exist, calculations may be performed in parallel or serially through a computer network.

In addition, according to an embodiment of the present application, the learning DB 200 is a database including a plurality of learning fundus photographs (1), a plurality of learning axial lengths (2) corresponding thereto, and a plurality of learning age information (3). You can. In the description of the embodiment of the present application, the learning fundus photo (1) may correspond to the learning axial length (2), which is the axial length of the subject, and the learning age information (3), which is the age of the subject, as the correct answer data (ground truth). The fundus age prediction device 100 of the hospital is constructed to derive a predicted fundus age for a fundus photo (in other words, a 'prediction target fundus photo (4)') for which information on the fundus age of the subject is unknown. This may refer to image data used in the process of implementing a learning model.

First, the fundus age prediction device 100 may prepare a plurality of learning fundus photographs (1), a plurality of learning axial lengths (2), and a plurality of learning age information (3). Specifically, the fundus age prediction device 100 may receive a plurality of learning fundus photos (1), a plurality of learning axial lengths (2), and a plurality of learning age information (3) from the learning DB 200.

In addition, in order to prevent overfitting when implementing a learning model described later, the fundus age prediction device 100 includes a plurality of received learning fundus photographs (1), a plurality of learning axial lengths (2) corresponding thereto, and a plurality of learning ages. The data distribution of information (3) can be adjusted. In other words, the fundus age prediction device 100 may select some data among the received plurality of learning fundus photographs 1 and corresponding information based on a predetermined standard, and allow the selected data to be used for learning.

For example, the fundus age prediction device 100 aligns a plurality of learning fundus photos (1) and corresponding information based on the age (age) of the person being photographed, or a plurality of learning fundus pictures (1) and corresponding information based on a plurality of learning axial lengths (2). By sorting the fundus photo (1) and the corresponding information, learning can be performed by selecting a group of learning data that falls within a range that is similar to the pre-predicted value of the fundus photo (4) to be predicted by a predetermined level or more.

The fundus age prediction device 100 uses a plurality of learning fundus photographs (1), a plurality of learning axial lengths (2), and a plurality of learning age information (3) as learning data of a deep neural network to predict the subject of the fundus photograph. A learning model that predicts fundus age can be created. The learning model creation method will be described in detail in FIGS. 2 to 4 below.

Additionally, the fundus age prediction device 100 may receive a fundus photo 2 to be predicted. According to an embodiment of the present application, the fundus age prediction device 100 may receive a fundus photograph 2 to be predicted from the fundus photographing device 30, but is not limited thereto. As another example, the fundus age prediction device 100 may be mounted (installed) on the user terminal 31 in the form of an application. At this time, the fundus age prediction device 100 is installed within the user terminal 31, which is an edge device. We have a learning model launched by our institute through our own Edge Learning, and even if the prediction target fundus photograph (2) taken through the user terminal (31) is not transmitted to a separate server or computing device through the network, the user terminal (31) ) can operate to output a predicted value for fundus age on its own.

The fundus age prediction device 100 can predict the fundus age of the subject of the prediction target fundus photo 4 using the generated learning model. For example, the fundus age prediction device 100 may predict the axial length using the received prediction target fundus image 4 and predict the fundus age using the prediction target fundus image 4 and the predicted axial length. In this way, when using the axial length predicted using a deep neural network, the axial length can be predicted and used to measure fundus age without expensive axial length measurement equipment, so even in developing countries with underdeveloped medical industry infrastructure, axial length or fundus age can be easily measured. We can help you obtain it reliably.

Additionally, the fundus age prediction device 100 may obtain the axial length of the subject measured through an axial length measuring device. The fundus age prediction device 100 can predict the fundus age of the subject using the received prediction target fundus photo 4 and the measured axial length. In this way, if the measured axial length of the subject of the prediction target fundus photograph 4 is secured, the prediction result of the fundus age can be more accurate.

In the description of the embodiments of the present application, fundus age may refer to the age of the subject predicted through a fundus photograph. This is a value that reflects the subject's ocular health status and fundus blood vessel condition as determined from the fundus photograph. As described later, this is a value that is reflected through mutual comparison between the subject's actual age and the predicted fundus age. It can be used to evaluate the health status of the person being photographed or to diagnose (determine) the presence or absence of a disease in the person being photographed.

As a specific example, the fundus age of the subject predicted based on the prediction target fundus photo 4 received by the fundus age prediction device 100 is derived to be lower than the actual age of the subject. In this case, the ocular health or systemic vascular condition of the subject is judged to be good, but if the predicted fundus age is determined to be higher than the actual age of the subject, the eye health of the subject is judged to be good. It may be inferred that the health condition or systemic vascular condition is abnormal.

The fundus age prediction device 100 can derive the basis area used to predict fundus age from the prediction target fundus photograph 4. The method of deriving the basis area will be described in detail in the description of FIGS. 5 and 6 below.

Additionally, the fundus age prediction device 100 may output (display) a ground image that emphasizes and displays the derived ground area. For example, the fundus age prediction device 100 allows subjects (medical staff, etc.) to visually check the output results by ensuring that the color, contrast, sharpness, etc. of the derived evidence area contrasts with other areas of the prediction target fundus photo 4. It is possible to intuitively recognize the basis area. An example of the basis image will be described in FIG. 7 below.

The fundus age prediction device 100 may receive the corrected fundus age determined by the medical staff as feedback on the predicted fundus age. If the corrected fundus age determined by the medical staff is different from the fundus age predicted by the learning model, the existing learning model can be updated by adding data near the corrected fundus age. This can increase the accuracy of prediction values during clinical trials. Additionally, if expansion and transformation of the learning model structure are necessary, expansion and transformation may be possible in order to increase the accuracy of the predicted value in clinical trials.

Figure 2 is a conceptual diagram for explaining a deep neural network-based learning model according to an embodiment of the present application.

Referring to FIG. 2, a learning model according to an embodiment of the present invention may include one or more deep neural networks 210 and 220. The deep neural network in the present invention can refer to a multi-layered network that can learn through multiple data and their corresponding labels, and each deep neural network largely consists of an input layer and a feature extraction module. It can be divided into three parts: extraction module, and prediction module. The number of nodes in the input layer is the same as the dimension of the fundus photo, and the fundus photo includes the original photo or a fundus photo that has been preprocessed for smooth learning of the deep neural network. The number of nodes in the output layer is the dimension of the fundus age to be predicted, and since fundus age is a scalar value, it is one. The feature extraction module is a convolution layer, which is a layer that calculates the correlation with a specific pattern in a local area of the input photo, or a fully connected layer that calculates the correlation with a specific pattern in all areas of the image. It can be made up of layers. In a convolutional layer, the depth dimension of a layer is determined by the number of filters, and the height and width dimensions can be determined by the size of the kernel, which is the range for calculating correlation with a specific pattern. Additionally, a fully connected layer may mean a layer in which all nodes of the previous layer and all nodes of the next layer are connected with weights.

In this way, the feature extraction module can be composed of multiple convolutional layers, multiple fully connected layers, or a mixture of these, and can be connected in series or parallel. In particular, a convolutional layer can be applied by adding the values before and after the input value passes through the layer. The level of abstraction of a convolutional layer composed of multiple layers is determined by the number of layers, and as the dimension of each layer increases, detailed features can be extracted. In addition, by applying pooling, non-linear filters, linear filters, or partial linear filters that extract only the necessary values between each layer, it can be less affected by unnecessary information or noise generated in the previous step. Such feature vectors can represent classification or prediction values by passing through a prediction module composed of multiple fully connected layers connected in sequence.

The learning model according to an embodiment of the present invention shown in Figure 2 is an example of predicting the axial length, which can be an important variable in predicting fundus age, from a fundus photograph and using this value to predict fundus age. The first deep neural network 210 for predicting axial length may be composed of an input layer, a feature extraction module, and a prediction module, which have the same structure as the second deep neural network 220 for predicting fundus age in FIG. 2. The number of input nodes of this deep neural network is the same as the number of dimensions of the fundus photograph, and since the axial length is a scalar, the number of output nodes is one. The axial length predicted through this can be input to the second deep neural network 220 that predicts the fundus age along with the fundus photograph. The second deep neural network 220 also consists of an input layer, a feature extraction module, and a prediction module. The number of input layer nodes is the sum of the dimension of the axial length and the dimension of the fundus photograph, and the number of output nodes outputs the fundus age value. It becomes one because it has to be done. Using this structure, it is possible to additionally predict axial length without the need for medical equipment to measure axial length, and it can help predict fundus age.

Figure 3 is a conceptual diagram for explaining a deep neural network-based learning model according to another embodiment of the present application.

Referring to FIG. 3, the learning model according to another embodiment of the present application consists of two deep neural networks like the learning model shown in FIG. 2, a first deep neural network 310 predicting the axial length, and a predicted axial length It consists of a second deep neural network 320 that extracts fundus age through length and fundus photographs. At this time, the difference from FIG. 2 is that the axial length predicted by the first deep neural network 310 is not entered as an input value of the second deep neural network 320, but is input into the feature extraction module. In other words, the feature extraction module extracts features from the fundus photograph and uses the concatenated vector of the extracted feature vector and the axial length as a new feature vector. The feature extraction module consisting of multiple layers can encompass a form that allows insertion of the axial length prediction value at any position, and the feature vector finally extracted from the feature extraction module can be input to the prediction module to predict the fundus age. Using this structure, it is possible to predict axial length without medical equipment that measures axial length, and it can help predict fundus age.

Figure 4 is a conceptual diagram for explaining a deep neural network-based learning model according to another embodiment of the present application.

Referring to Figure 4, the learning model according to another embodiment of the present application shown in Figure 4 is an example of a deep neural network that inputs the directly measured axial length along with the fundus photograph, unlike the learning model shown in Figures 2 and 3. am. This axial length refers to actual, precisely measured data using a professional measuring device, and can be used to predict fundus age along with fundus photographs. The deep neural network used at this time also consists of an input layer, a feature extraction module, and a prediction module. The number of input nodes is the sum of the dimensionality of the fundus photograph and the axial length, and the number of output nodes is also one. To utilize this structure, medical equipment that can measure axial length must be available.

The dataset used in the deep neural network of FIGS. 2 to 4 described above is a fundus image that includes all fundus anatomical landmarks such as optic disc, retinal vein, retinal artery, macular region, etc. as clearly as possible. ), and the age information (age) of the patient corresponding to each image must be collected. Additionally, if the axial length corresponding to each image is collected, this can be used as auxiliary information for predicting fundus age. This axial length and age information can be used as input or as a label when learning a deep neural network. The more uniform the data pairs are distributed according to age information, the better, and the more pairs there are, the better.

The variable update method used when learning a deep neural network can be based on gradient descent using first-order differentiation, and can include various methods applying this, such as stochastic gradient descent, momentum, and adam, but is not limited to this. Additionally, the distribution of learning data can be adjusted to increase the prediction performance of the deep neural network during training. For example, if the prediction error in a specific section is large, the error deviation can be reduced by acquiring more data for that part or by equalizing the distribution of the data overall.

Figure 5 is a diagram for explaining a method of obtaining a basis area according to an embodiment of the present application. In the present invention, the basis area refers to the basis area used to predict the fundus age in the prediction target fundus photograph.

The basis area calculation method shown in FIG. 5 is a method of measuring the importance of a node using the rate of change of the prediction result value for an arbitrary node value, or the rate of change of the node value of the next layer relative to the node value of the previous layer.

Referring to Figure 5, case 1 shows the influence of an arbitrary node on the node of the next layer through first-order differentiation. Random node in l+1th layer ( ), a random node in the lth layer for ( ) the derivative of In order to obtain a more accurate contribution rate, the differential value can be multiplied by an arbitrary weight function f. Function f is the nodes of the lth layer ( ) and the nodes of the next layer for these nodes ( )'s derivatives ( ), and the bias of the l+1th layer ( ) function for , i.e. It can be. This function can be a linear function, a non-linear function, or a piecewise-linear function.

Case 2 in Figure 5 is a case showing the influence of a random node on the entire next layer. In this case, the contribution value for the layer can be obtained by adding up all the calculation results of Case 1 for the nodes corresponding to the l+1th layer. Case 3 is to calculate the contribution value of a random node to the l+2th layer. If the node is called x, when calculating the contribution value for the l+1th layer of node x, add the contribution value corresponding to the l+2th layer of each node to the contribution value for each l+1th node. By multiplying, the result is the contribution value for the l+2th layer of node x. Therefore, even in the case of a deep neural network with multiple layers, if the calculation of Case 3 is performed serially, the contribution of the input nodes to the output value can be obtained. Examples of using this include a method using Taylor Series Approximation, which is an approximation method that sets all terms to 0 except for the first few terms in the Taylor Series, and a method using normalization for differential values. Currently applied technologies include the first method, Layerwise Relevance Propagation and guided-backpropagation, and the second method, CAM and grad-CAM, but are not limited thereto.

Figure 6 is a diagram for explaining a method of obtaining a basis area for a special case according to an embodiment of the present application. In other words, the evidence area calculation method shown in FIG. 6 is a special case of the evidence area calculation method shown in FIG. 5, which calculates the extent to which the previous node explicitly contributes to the prediction result or the node value of the subsequent layer. Indicates the method.

Referring to FIG. 6, the basis area calculation method of FIG. 6 uses the nodes of the lth layer ( ) nodes of the next layer for ( )'s derivatives ( ) to the weight of the l+1th layer ( ) It may be in a specific form.

The calculation method in Figures 5 and 6 is a method that can be applied to a convolutional layer, a fully connected layer, or any type of layer in a deep neural network, and can quantify how an arbitrary node or arbitrary layer affects the output value. Evidence can be provided. If this quantified value is expressed through color, the degree of influence can be easily confirmed. In addition to the calculation methods in FIGS. 5 and 6, feature vectors that have a significant impact on prediction can be visualized by showing the weight vectors and output values of the layers that make up the feature extraction module in the learned deep neural network.

Figure 7 is a diagram illustrating an example of a ground image displayed by emphasizing a ground area in a fundus photograph according to an embodiment of the present application. In other words, Figure 7 shows the process of extracting a part that can serve as the basis for the calculated fundus age prediction value and visualizing it together with the fundus age prediction value.

Referring to Figure 7, (a) shows an example of a fundus photograph that is a prediction target. (b) shows an example image showing the basis area that can serve as the basis for the fundus age prediction value. (c) shows an example of a basis image that visualizes the basis area along with the predicted value of fundus age.

According to an embodiment of the present application, in the highlighted evidence area, the greater the influence (contribution) value calculated for each position (pixel), the greater the elements such as color, contrast, and sharpness in contrast to areas that are not the evidence area. It can be output (displayed) to change. Referring to (c) of Figure 7, it can be seen that the part corresponding to the evidence area is displayed in red and green and is visually distinguished from the part that is not the evidence area.

In Figure 7 (c), the part corresponding to the basis area is displayed overlapping with the prediction target fundus photograph, but the display method is not limited to this, and two or more images from (a), (b), and (c) are displayed side by side. It can be displayed to help medical staff diagnose or ensure the reliability of predicted values.

Figure 8 is a schematic configuration diagram of a device for predicting fundus age using a fundus photograph and axial length according to an embodiment of the present application.

Referring to FIG. 8 , the fundus age prediction device 100 may include a learning unit 110, an inference unit 120, and a visualization unit 130.

The learning unit 110 prepares a plurality of learning fundus photographs, a plurality of learning axial lengths, and a plurality of learning age information corresponding thereto, and uses the plurality of learning fundus photographs, the plurality of learning axial length lengths, and the plurality of learning age information. Thus, a learning model that predicts the fundus age of the subject of the fundus photo can be created.

In addition, the learning unit 110 trains the first deep neural network by specifying a plurality of learning fundus photos as input values and a plurality of learning axial lengths as labels to generate a learning model according to an embodiment of the present application. , use the learned first deep neural network to derive multiple predicted axial lengths for multiple learning fundus photos, specify multiple learning fundus photos and multiple predicted axial lengths as input values, and label multiple learning age information. You can train the second deep neural network by specifying .

In addition, the learning unit 110 trains the first deep neural network by specifying a plurality of learning fundus photos as input values and a plurality of learning axial lengths as labels to generate a learning model according to another embodiment of the present application. , using the learned first deep neural network to derive multiple predicted axial lengths for multiple learning fundus photos, specifying multiple learning fundus photos as input values, and applying the multiple predicted axial lengths to the characteristics of the second deep neural network. A second deep neural network can be trained by inserting it into any one layer of the extraction module and specifying a plurality of learning age information as a label.

In addition, the learning unit 110 specifies a plurality of learning fundus photos and a plurality of learning axial lengths as input values and designates a plurality of learning age information as labels to create a learning model according to another embodiment of the present application. Neural networks can be trained.

In addition, the learning unit 110 receives the corrected fundus age determined by the medical staff as feedback on the predicted fundus age, and when the corrected fundus age and the predicted fundus age are different, the learning unit 110 corresponds to the corrected fundus age or provides information near the corrected fundus age. The learning model can be updated by adding multiple learning fundus photos, multiple learning axial lengths, and multiple learning age information to the dataset.

The inference unit 120 may receive the prediction target fundus photo and predict the fundus age of the subject of the prediction target fundus photo using the learning model generated by the learning unit 110.

In addition, the inference unit 120 receives the measured axial length of the subject, and uses the prediction target fundus photo and the measured axial length as input values of the learning model generated by the learning unit 110 to predict the fundus age. .

The visualization unit 130 may derive an evidence area from the prediction target fundus photograph whose influence on the prediction of fundus age is at least a predetermined level and output an evidence image highlighting the evidence region from the prediction target fundus photograph.

For example, the visualization unit 130 calculates a contribution value indicating the influence of any node of the I-th layer on any node of the I+1-th layer through Equation 1 below, and Equation 2: Through Equation 3, the contribution value representing the influence of any node in the I-th layer on the entire I+1-th layer is calculated, and the influence of any node in the l-th layer on the entire l+2-th layer is calculated through Equation 3. Calculate the contribution value representing . Therefore, in a deep neural network composed of multiple complex layers, Equation 3 can be used in chain to calculate the value that one arbitrary node contributes to the output node of the neural network to derive the basis area used for prediction.

[Equation 1]

[Equation 2]

[Equation 3]

According to one embodiment of the present application, the visualization unit 130 displays a ground image representing the derived ground area (e.g., a saliency map that can be understood through (c) of FIG. 7, a class activation map (CAM), a Grad -CAM (gradient-class activation map, etc.) is displayed together (e.g., displayed horizontally or vertically side by side) to distinguish it from the original prediction target fundus photo (e.g., (a) in Figure 7), or the original An image representing the basis area may be overlaid and displayed on the prediction target fundus photo, but is not limited to this.As another example, the visualization unit 130 displays a mode in which the original prediction target fundus photo is displayed as is and the basis area is highlighted. The output modes are provided as variably as possible, but each mode can be switched according to the user input applied to the fundus age prediction device 100.

Below, we will briefly look at the operation flow of the present application based on the details described above.

Figure 9 is an operation flowchart of a method for predicting fundus age using a fundus photograph and axial length according to an embodiment of the present application.

The method of predicting biometric values based on the fundus photograph shown in FIG. 9 may be performed by the fundus age prediction device 100 described above. Therefore, even if the content is omitted below, the information described regarding the fundus age prediction device 100 can be equally applied to the description of the method of predicting the fundus age using the fundus photograph and the axial length.

Referring to FIG. 9, the fundus age prediction device 100 may prepare a plurality of learning fundus photos, a plurality of learning axial lengths, and a plurality of learning age information corresponding thereto in step S910.

Next, in step S920, a learning model that predicts the fundus age of the subject of the fundus photograph can be generated using a plurality of learning fundus photographs, a plurality of learning axial lengths, and a plurality of learning age information.

Next, the prediction target fundus photo may be received in step S930.

Next, the fundus age of the subject of the prediction target fundus photograph can be predicted using the learning model generated in step S940.

In the above description, steps S910 to S940 may be further divided into additional steps or combined into fewer steps, depending on the implementation of the present disclosure. Additionally, some steps may be omitted or the order between steps may be changed as needed.

Figure 10 is an operation flowchart of a method for generating a learning model for predicting fundus age according to an embodiment of the present application.

Referring to FIG. 10, in step S1010, a first deep neural network can be trained by specifying a plurality of learning fundus photographs as input values and a plurality of learning axial lengths as labels.

Next, a plurality of predicted axial lengths for a plurality of learning fundus photographs can be derived using the first deep neural network learned in step S1020.

Next, in step S1030, a second deep neural network can be trained by specifying a plurality of learning fundus photographs and a plurality of predicted axial lengths as input values and a plurality of learning age information as labels.

Figure 11 is an operation flowchart of a method for generating a learning model for predicting fundus age according to another embodiment of the present application.

Referring to FIG. 11, in step S1110, a first deep neural network can be trained by specifying a plurality of learning fundus photographs as input values and a plurality of learning axial lengths as labels.

Next, a plurality of predicted axial lengths for a plurality of learning fundus photographs can be derived using the first deep neural network learned in step S1120.

Next, in step S1130, a plurality of learning fundus photos are specified as input values, a plurality of predicted axial lengths are inserted into any one layer of the feature extraction module of the second deep neural network, and a plurality of learning age information is designated as a label to create the first 2 Deep neural networks can be trained.

Figure 12 is an operation flowchart of a method for generating a learning model for predicting fundus age according to another embodiment of the present application.

Referring to FIG. 12, in step S1210, a deep neural network can be trained by specifying a plurality of learning fundus photographs and a plurality of learning axial lengths as input values, and specifying a plurality of learning age information as labels.

Figure 13 is an operation flowchart of a method for visualizing the basis area used for predicting fundus age according to one or another embodiment of the present application.

In step S1310, the first basis area used to predict the axial length can be derived from the prediction target fundus photograph.

Next, in step S1320, a second basis area used to predict fundus age can be derived from the prediction target fundus photograph.

Next, in step S1330, a ground image in which the first and second ground regions are highlighted in the prediction target fundus photograph may be output.

Figure 14 is an operation flowchart of a method for visualizing the basis area used for predicting fundus age according to another embodiment of the present application.

Referring to FIG. 14, in step S1410, the basis area used for predicting fundus age can be derived from the fundus photograph to be predicted.

Next, in step S1420, a ground image in which the ground region is highlighted and displayed in the prediction target fundus photograph can be output.

The method of predicting fundus age using a fundus photograph and axial length according to an embodiment of the present application may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc., singly or in combination. Program instructions recorded on the medium may be specially designed and constructed for the present invention or may be known and usable by those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. -Includes optical media (magneto-optical media) and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, flash memory, etc. Examples of program instructions include machine language code, such as that produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa.

Additionally, the method of predicting fundus age using the above-described fundus photograph and axial length may also be implemented in the form of a computer program or application executed by a computer stored in a recording medium.

The description of the present application described above is for illustrative purposes, and those skilled in the art will understand that the present application can be easily modified into other specific forms without changing its technical idea or essential features. Therefore, the embodiments described above should be understood in all respects as illustrative and not restrictive. For example, each component described as unitary may be implemented in a distributed manner, and similarly, components described as distributed may also be implemented in a combined form.

The scope of the present application is indicated by the claims described below rather than the detailed description above, and all changes or modified forms derived from the meaning and scope of the claims and their equivalent concepts should be construed as being included in the scope of the present application.

10: Fundus age prediction system
100: Fundus age prediction device
110: Learning Department
120: Inference unit
130: Visualization unit
20: Network
30: Fundus photography device
200: Learning DB
1: Study fundus photo
2: Learning optic axis length
3:Learning age information
4: Fundus photo of prediction target

Claims (14)

In a method of predicting fundus age using a fundus photograph and axial length performed by a device for predicting fundus age using a fundus photograph and axial length,
A learning unit preparing a plurality of learning fundus photos, a plurality of learning axial lengths corresponding thereto, and a plurality of learning age information;
The learning unit generates a learning model that predicts the fundus age of the subject of the fundus photograph using the plurality of learning fundus photographs, the plurality of learning axial lengths, and the plurality of learning age information;
A step of the inference unit receiving a prediction target fundus photograph; and
The inference unit predicting the fundus age of the subject of the prediction target fundus photograph using the generated learning model,
Including, fundus age prediction method.
According to paragraph 1,
The step of creating the learning model is,
Training a first deep neural network by designating the plurality of learning fundus photographs as input values and the plurality of learning axial lengths as labels;
Deriving a plurality of predicted axial lengths for the plurality of learning fundus photographs using the learned first deep neural network; and
Training a second deep neural network by specifying the plurality of learning fundus photographs and the plurality of predicted axial lengths as input values and the plurality of learning age information as labels.
A method for predicting fundus age, comprising:
According to paragraph 1,
The learning model includes one or more deep neural networks, each deep neural network including an input layer, a feature extraction module, and a prediction module,
The step of creating the learning model is,
Training a first deep neural network by designating the plurality of learning fundus photographs as input values and the plurality of learning axial lengths as labels;
Deriving a plurality of predicted axial lengths for the plurality of learning fundus photographs using the learned first deep neural network; and
The plurality of learning fundus photographs are designated as input values, the plurality of predicted axial lengths are inserted into any one layer of the feature extraction module of the second deep neural network, and the plurality of learning age information is designated as a label to obtain the second deep neural network. Steps to train a neural network
A method for predicting fundus age, comprising:
According to paragraph 1,
The step of creating the learning model is,
Designating the plurality of learning fundus photographs and the plurality of learning axial lengths as input values, and specifying the plurality of learning age information as labels to train a deep neural network.
A method for predicting fundus age, comprising:
According to paragraph 4,
Receiving the measured axial length of the person being photographed
It further includes,
The step of predicting the fundus age is to predict the fundus age of the subject of the prediction target fundus photo by specifying the prediction target fundus photo and the measured axial length as input values of the deep neural network. method.
According to paragraph 1,
Deriving a basis area used for predicting the fundus age from the prediction target fundus photograph; and
Outputting a base image highlighting the base area in the prediction target fundus photograph;
It further includes,
In the step of deriving the evidence area, a contribution value representing the influence of any node of the I-th layer on any node of the I+1-th layer is calculated through Equation 1 below, and through Equation 2: Calculate the contribution value representing the influence of any node in the I-th layer on the entire I+1-th layer, and calculate the influence of any node in the l-th layer on the entire l+2-th layer through Equation 3. Fundus age prediction, which involves calculating the contribution value indicated above and applying Equation 3 in chain to calculate the value that a random node contributes to the output node of the corresponding neural network to derive the basis area used for prediction. method.
[Equation 1]


[Equation 2]


[Equation 3]


According to paragraph 1,
Receiving the corrected fundus age determined by a medical staff as feedback on the predicted fundus age; and
Updating the learning model by adding a plurality of learning fundus photos and a plurality of learning axial lengths corresponding to the corrected fundus age to the dataset.
A method for predicting fundus age, further comprising:
In a device for predicting fundus age using fundus photographs and axial length,
Prepare a plurality of learning fundus photographs, a plurality of learning axial lengths, and a plurality of learning age information corresponding thereto, and prepare a fundus photograph using the plurality of learning fundus photographs, the plurality of learning axial lengths, and the plurality of learning age information. A learning unit that generates a learning model that predicts the fundus age of the subject; and
An inference unit that receives the prediction target fundus photo and predicts the fundus age of the subject of the prediction target fundus photo using the generated learning model;
Including, fundus age prediction device.
According to clause 8,
The learning department,
The plurality of learning fundus photographs are designated as input values, the plurality of learning axial lengths are designated as labels, a first deep neural network is trained, and the learned first deep neural network is used to perform multiple training for the plurality of learning fundus photographs. Fundus age, which is to derive the predicted axial length of, designate the plurality of learning fundus photos and the plurality of predicted axial lengths as input values, and designate the plurality of learning age information as labels to train a second deep neural network. Prediction device.
According to clause 8,
The learning model includes one or more deep neural networks, each deep neural network including an input layer, a feature extraction module, and a prediction module,
The learning department,
The plurality of learning fundus photographs are designated as input values, the plurality of learning axial lengths are designated as labels, a first deep neural network is trained, and the learned first deep neural network is used to perform multiple training for the plurality of learning fundus photographs. Derive the predicted axial length, specify the plurality of learning fundus photographs as input values, insert the plurality of predicted axial lengths into any one layer of the feature extraction module of the second deep neural network, and obtain the plurality of learning age information. A fundus age prediction device that trains the second deep neural network by designating as a label.
According to clause 8,
The learning department,
A fundus age prediction device that trains a deep neural network by specifying the plurality of learning fundus photos and the plurality of learning axial lengths as input values and the plurality of learning age information as labels.
According to clause 11,
The inference unit,
Receiving the measured axial length of the subject,
A fundus age prediction device that predicts the fundus age of the subject by specifying the prediction target fundus photograph and the measured axial length as input values of the deep neural network.
According to clause 8,
A visualization unit that derives a basis area used for prediction of the fundus age from the prediction target fundus photo, and outputs a basis image highlighting the evidence area in the prediction target fundus photo.
It further includes,
The visualization unit,
Calculate the contribution value representing the influence of any node in the I-th layer on any node in the I+1-th layer through Equation 1 below, and calculate the contribution value representing the influence of any node in the I-th layer through Equation 2. Calculate the contribution value representing the influence on the entire I+1th layer, and calculate the contribution value representing the influence of any node in the lth layer on the entire l+2th layer through Equation 3, and calculate the contribution value representing the influence on the entire l+2th layer using Equation 3. A fundus age prediction device that calculates the value that a random node contributes to the output node of the corresponding neural network by applying Equation 3 in chain to derive the basis area used for prediction.
[Equation 1]


[Equation 2]


[Equation 3]


According to clause 8,
The learning department,
Receiving the corrected fundus age determined by the medical staff as feedback on the predicted fundus age, and updating the learning model by adding a plurality of learning fundus photos and a plurality of learning axial lengths corresponding to the corrected fundus age to the dataset. In, fundus age prediction device.

Images