Classifications

• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T7/00—Image analysis
  • G06T7/0002—Inspection of images, e.g. flaw detection
  • G06T7/0012—Biomedical image inspection
• • G—PHYSICS
  • G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
  • G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
  • G16H30/00—ICT specially adapted for the handling or processing of medical images
  • G16H30/40—ICT specially adapted for the handling or processing of medical images for processing medical images, e.g. editing
• • G—PHYSICS
  • G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
  • G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
  • G16H50/00—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics
  • G16H50/20—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics for computer-aided diagnosis, e.g. based on medical expert systems
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T2207/00—Indexing scheme for image analysis or image enhancement
  • G06T2207/20—Special algorithmic details
  • G06T2207/20081—Training; Learning

Definitions

• This description is generally directed toward the prediction of geographic atrophy progression. More specifically, this description provides methods and systems for predicting geographic atrophy progression using various models and analyses (e.g., gradient activation map analysis, ablation analysis) performed for these models.
• models and analyses e.g., gradient activation map analysis, ablation analysis
• Age-related macular degeneration is a leading cause of vision loss in patients 50 years or older.
• GA Geographic atrophy
• RPE retinal pigment epithelium
• GA progression varies between patients, and currently, no widely accepted treatment for preventing or slowing down the progression of GA exists. Therefore, evaluating GA progression in individual patients may be important to researching GA and developing an effective treatment.
• Figure 4 is a flowchart of a process 400 for improving model performance in accordance with various embodiments.
• FIG. 8 is a block diagram of a computer system in accordance with various embodiments.
• the figures are not necessarily drawn to scale, nor are the objects in the figures necessarily drawn to scale in relationship to one another.
• the figures are depictions that are intended to bring clarity and understanding to various embodiments of apparatuses, systems, and methods disclosed herein. Wherever possible, the same reference numbers will be used throughout the drawings to refer to the same or like parts. Moreover, it should be appreciated that the drawings are not intended to limit the scope of the present teachings in any way.
• a GA lesion can be imaged by various imaging modalities including, but not limited to, fundus autofluorescence (FAF) and optical coherence tomography (OCT).
• fundus autofluorescence (FAF) images may be input into one or more models (e.g., one or more deep learning models) to predict baseline lesion area, lesion growth rate, or both.
• the FAF images may be baseline FAF images that are taken at a baseline point in time.
• the baseline point in time may be the beginning of the clinical trial, the time of the initial assessment, a time just prior to a first administration of treatment, a time coincident with the first administration of treatment, a same day as the first administration of treatment, or some other baseline point in time.
• the embodiments described herein provide methods and systems for evaluating the image regions or features of images (e.g., FAF images) that contribute to the prediction of GA progression parameters (e.g., lesion growth rate, baseline lesion area, etc.) by a model (e.g., a deep learning model).
• GA progression parameters e.g., lesion growth rate, baseline lesion area, etc.
• model e.g., a deep learning model
• various types of visualizations are used to understand how such models process their inputs.
• An ablation analysis may include performing segmentation of a retinal image and then ablating (e.g., removing) various combinations of the segmented regions.
• a segmentation algorithm may be used to segment out (or separately identify) a GA lesion, a rim (e.g., 500 pm-wide margin) around the GA lesion, and a background (e.g., any portion of the image not identified as the GA lesion or the rim).
• Various combinations of the GA lesion, the rim, and the background may be ablated from the retinal image to form an ablated image that is then fed as input into the deep learning model. Comparing the performance of the model based on different types of ablated image inputs allows a determination of which image regions or features are relevant (e.g., most relevant) to the one or more GA progression parameters predicted by a deep learning model.
• the information provided by the ablation analysis may be used to validate the deep learning model and confirming whether or not the deep learning model is focusing on non-spurious and relevant portions of the retinal image based on what is known or expected.
• the ablation analysis may be used to generate an output for use in improving performance of the deep learning model.
• the ablation analysis may be used to identify new biomarkers or localize known biomarkers to thereby more narrowly tailor the focus of the deep learning model.
• the ablation analysis may be used to narrow the focus of the deep learning model to reduce the time and computing resource expenditure of the deep learning model, while maintaining a desired level of predictive accuracy.
• the results of the ablation analysis may be used to generate an output that identifies modifications that can be made to the deep learning model to improve the accuracy and/or reliability of the deep learning model.
• FIG. 1 is a block diagram of a lesion evaluation system 100 in accordance with various embodiments.
• Lesion evaluation system 100 is used to evaluate geographic atrophy (GA) lesions in the retinas of subjects.
• Lesion evaluation system 100 includes computing platform 102, data storage 104, and display system 106.
• Computing platform 102 may take various forms. In one or more embodiments, computing platform 102 includes a single computer (or computer system) or multiple computers in communication with each other. In other examples, computing platform 102 takes the form of a cloud computing platform.
• Data storage 104 and display system 106 are each in communication with computing platform 102.
• data storage 104, display system 106, or both may be considered part of or otherwise integrated with computing platform 102.
• computing platform 102, data storage 104, and display system 106 may be separate components in communication with each other, but in other examples, some combination of these components may be integrated together.
• Lesion evaluation system 100 includes image processor 108, which may be implemented using hardware, software, firmware, or a combination thereof. In one or more embodiments, image processor 108 is implemented in computing platform 102.
• a baseline image is an image captured at a baseline point in time.
• the baseline point in time may be the beginning of the clinical trial, the time of an initial assessment or initial clinic or clinical trial visit, a time just prior to a first administration of treatment, a time coincident with the first administration of treatment, a same day as the first administration of treatment, or some other baseline point in time.
• set of retinal images 110 may include one or more other types of retinal images (e.g., color fundus (CF) photography images, near infrared (NIR) images, etc.).
• CF color fundus
• NIR near infrared
• image processor 108 includes model analyzer 120.
• Model analyzer 120 may be used to generate a visualization output for each of models 112.
• model analyzer 120 may generate first visualization output 122 for first model 117 and second visualization output 123 for second model 118.
• model analyzer 120 may generate an output that indicates that an adjustment(s) should be made to first model 117.
• This adjustment(s) may include, for example, without limitation, retraining first model 117, changing the layers used in first model 117, modifying the architecture of first model 117, combining first model 117 with another model, or a combination thereof.
• Second model 118 may be similarly evaluated using second visualization output 123.
• model analyzer 120 may be used to generate first visualization output 122 and second visualization output 123 after first model 117 and second model 118, respectively, have been trained and tested for the prediction of set of GA progression parameters 114 based on set of retinal images 110, which may include a plurality of training baseline FAF images.
• These two visualization outputs may be used to determine whether one model is better suited relative to the other model for a particular type of GA lesion, whether the models perform similarly for the same types of GA lesions, etc.
• the information provided by these visualization outputs may then be used to determine which model to select for use in predicting set of GA progression parameters 114 for a particular subject or group of subjects.
• an ablated lesion is formed by blacking out the pixels of the GA lesion and the rim such that the ablated image is an image of just the background (e.g., a background retained image).
• an ablated lesion is formed by blacking out the pixels of the GA lesion such that the ablated image is an image of the rim and the background (e.g., a rim and background retained image).
• Different groupings of ablated images may be used to train, for example, first model 117 to form different trained models. The performance of these different trained models may be evaluated using, for example, model analyzer 120.
• Model analyzer 120 may evaluate performance with respect to accuracy, precision, reliability, a coefficient of determination (r 2 ), one or more other metrics, or a combination thereof. Examples of how this training and evaluation may be performed are described in greater detail with respect to Figure 4 below. Evaluating the performance of these different trained models may help identify or localize biomarkers, gain insight into GA pathology, develop trust that first model 117 is not focusing on spurious or irrelevant image regions or features, and/or improve the performance of first model 117.
• model analyzer 120 may be used to generate output 132 based the evaluation of the performance of the different trained models formed using the ablation techniques described above and/or based on first visualization output 122 and second visualization output 123.
• Output 132 may, for example, identify new biomarkers and/or a localized set of biomarkers that can be used to help narrow the focus of one or more of models 112.
• Output 132 may, for example, identify information that provides insight into GA pathology.
• Output 132 may, for example, indicate whether a model of models 112 can be validated.
• Output 132 may, for example, confirm whether or not a model of models 112 is focusing on non-spurious and relevant image regions or features.
• Output 132 may, for example, identify one or more modifications that can be made to a model of models 112 to improve the performance of that model.
• Step 204 includes training each model of a plurality of models to predict a set of geographic atrophy (GA) progression parameters for a geographic atrophy (GA) lesion using the set of retinal images.
• the plurality of models may include, for example, without limitation, a plurality of deep learning models.
• the plurality of models in step 204 may be one example of an implementation for models 112 in Figure 1.
• the plurality of models may include a first deep learning model comprised of one or more convolutional neural networks and a second deep learning model comprised of one or more convolutional neural networks.
• the set of GA progression parameters may be one example of an implementation for set of GA progression parameters 114 in Figure 1.
• the set of GA progression parameters may include growth rate, baseline lesion area, or both.
• Step 206 generating a visualization output for each model of the plurality of models, wherein the visualization output for a corresponding model of the plurality of models provides information about how the corresponding model uses the set of retinal images to predict the set of GA progression parameters.
• step 206 may include generating a gradient activation map for a corresponding retinal image in the set of retinal images for the corresponding model.
• the gradient activation map indicates (e.g., visually identifies) a set of regions in the corresponding retinal image that contributed to the set of GA progression parameters predicted by the corresponding model for the GA lesion.
• the gradient activation map identifies the one or more regions of the retinal image that were relevant to (or that drove) the prediction of the set of GA progression parameters.
• the gradient activation map may visually identify the degree of relevancy of these one or more regions to the prediction of the set of GA progression parameters.
• the visualization outputs generated for the different models may be used to validate these models.
• one of the plurality of models may be a deep learning model.
• the visualization output generated for this deep learning model may be used to validate the deep learning model and confirm that the deep learning model is focusing on non-spurious, relevant portions of the relevant image to predict the set of GA progression parameters.
• a comparison of the different visualization outputs generated for different models may be performed to select a best model for predicting the set of GA progression parameters.
• one of the plurality of models may be modified to form a new model based on the visualization output generated for that model to thereby improve a performance of the model.
• Performance of the model may be measured with respect to accuracy, precision, reliability, a time spent generating the prediction, an amount of computing resources utilized to generate the prediction, a coefficient of determination, or a combination thereof.
• Figure 3 is a flowchart of a process 300 for evaluating a geographic atrophy lesion in accordance with various embodiments.
• process 300 is implemented using the lesion evaluation system 100 described in Figure 1.
• process 300 may be used to predict one or more GA progression parameters.
• Step 302 includes receiving a set of retinal images.
• the set of retinal images may be one example of an implementation for set of retinal images 110 in Figure 1.
• the set of retinal images may belong to a single subject that has been diagnosed with geographic atrophy or, in some cases, a precursor stage to geographic atrophy.
• the set of retinal images may include a set of baseline FAF images, a set of baseline OCT images, or both for the same retina of a subject. These baseline images may include images captured for the same or substantially same (e.g., within the same hour, within the same day, within the same 1-3 days, etc.) point or points in time.
• Step 304 includes predicting a set of geographic atrophy (GA) progression parameters for a geographic atrophy (GA) lesion in the retina using the set of retinal images and a deep learning model.
• the set of GA progression parameters comprises at least one of a growth rate for the GA lesion or a baseline lesion area for the GA lesion.
• Step 306 includes generating a set of gradient activation maps corresponding to the set of retinal images for the deep learning model, wherein a gradient activation map in the set of gradient activation maps for a corresponding retinal image of the set of retinal images identifies a set of regions in the corresponding retinal image that is relevant to predicting the set of GA progression parameters by the deep learning model.
• the gradient activation map may, for example, visually identify a degree of relevancy of the one or more regions in the retinal image using coloring, shading, highlighting, pattern, text and/or numerical labels, other types of indicators, or a combination thereof.
• Process 300 may optionally include step 308, which includes generating an output for use in improving a performance of the deep learning model based on the set of gradient activation maps.
• the output may, for example, identify new biomarkers and/or a localized set of biomarkers that can be used to help narrow the focus of the deep learning model, identify information that provides insight into GA pathology and that can be used to modify the deep learning model, indicate whether the deep learning can be validated, confirm whether or not the deep learning model is focusing on non-spurious and relevant image regions or features, identify one or more modifications that can be made to the deep learning model to improve the performance of that model, or a combination thereof.
• Figure 4 is a flowchart of a process 400 for improving model performance in accordance with various embodiments.
• process 400 is implemented using the lesion evaluation system 100 described in Figure 1.
• Step 402 includes receiving a plurality of retinal images for a plurality of subjects.
• This plurality of retinal images may be one example of an implementation for set of retinal images 110 in Figure 1.
• the plurality of retinal images may include, for example, without limitation, baseline FAF images for the subjects. These subjects may be persons who were diagnosed with GA or a precursor stage of GA and for who GA progression information (e.g., growth rate of GA lesion) is known.
• each retinal image of the plurality of retinal images captures a GA lesion in the retina of a corresponding subject.
• ablating a selected portion or region of an image is performed by blacking out the pixels corresponding to the ablated portion or region.
• step 404 may include segmenting the plurality of retinal images using a segmentation algorithm to identify the portion of the image that represents the GA lesion, the portion of the image that represents the rim of the GA lesion, and the background in each retinal image (portion of image not identified as GA lesion or rim).
• Ablating out a portion of the image may include assigning the pixels identified as representing the that portion to a value of black (e.g., a pixel value of zero).
• the rim of the GA lesion may be ablated by assigning those pixels segmented out as representing the GA lesion to a pixel value of zero.
• the plurality of ablated image groups formed in step 404 may be formed by ablating one or more different portions of the retinal images.
• an ablated image group may be a lesion retained image group comprised of a plurality of lesion retained images in which the rim of the GA lesion and the background are ablated such that only the GA lesion of the original retinal image is retained.
• An ablated image group may be a lesion rim image group comprised of a plurality of rim retained images in which the GA lesion and the background are ablated such that only the rim of the GA lesion of the original retinal image is retained.
• An ablated image group may be a background retained image group comprised of a plurality of background retained in which the GA lesion and the rim of the GA lesion are ablated such that only the background of the original retinal image is retained.
• Another ablated image group may be a lesion ablated image group comprised of a plurality of lesion ablated images in which the GA lesion is ablated such that the rim of the GA lesion and the background of the original retinal image are retained.
• Yet another ablated image group may be a rim ablated image group comprised of a plurality of rim ablated images in which the rim of the GA lesion is ablated such that the GA lesion and the background of the original retinal image are retained.
• Still another ablated image group may be a background ablated image group comprised of a plurality of background ablated images in which the background is ablated such that the GA lesion and the rim of the GA lesion of the original retinal image are retained. In this manner, different portions or combinations of portions of a retinal image may be ablated to form an ablated image.
• step 404 further includes shuffling the pixel values within whatever portion of the original retinal image is retained to form an ablated image.
• This shuffling may be a randomly- performed repositioning of the pixel values amongst the pixels included in the retained portion of the retinal image.
• an ablated image may be a lesion shuffled image in which the portion of the retinal image identified as the GA lesion is retained and the pixel values of pixels within this portion are shuffled. This shuffling retains the intensity information associated with this portion of the retinal image but removes textural information (e.g., what areas of this portion are brighter than others).
• An ablated image may be a rim shuffled image in which the portion of the retinal image identified as the rim of the GA lesion is retained with the pixel values of this portion being shuffled.
• An ablated image may be a background shuffled image in which the portion of the retinal image identified as background is retained with the pixel values of this portion being shuffled.
• the plurality of ablated image groups may include a lesion shuffled image group, a rim shuffled image group, a background shuffled image group, a lesion and rim shuffled image group, a lesion and background shuffled image group, a rim and background shuffled image group, or a combination thereof.
• Step 406 includes training an initial model to predict growth rate of a GA lesion using each of the plurality of ablated image groups to form a plurality of trained models.
• the initial model may be, for example, a deep learning model and may include one or more neural networks.
• a trained model may be formed for each ablated image group of the plurality of ablated image groups.
• the initial model may be trained and tested using a first ablated image group of the plurality of ablated image groups to form a first trained model.
• the initial model may be trained and tested using a second ablated image group of the plurality of ablated image groups to form a second trained model.
• Step 408 includes evaluating a performance of the plurality of trained models.
• Performance may be analyzed with respect to any number of metrics including, for example, without limitation, accuracy, precision, a coefficient of determination (r 2 ), a time spent by the trained model to analyze a single ablated image to predict growth rate as compared to a time spent by the initial model to analyze the corresponding original retinal image, an amount of computing resources spent by the trained model to analyze a single ablated image to predict growth rate as compared to the amount of computing resources spent by the initial model to analyze the corresponding original retinal image, one or more other types of metrics, or a combination thereof.
• metrics including, for example, without limitation, accuracy, precision, a coefficient of determination (r 2 ), a time spent by the trained model to analyze a single ablated image to predict growth rate as compared to a time spent by the initial model to analyze the corresponding original retinal image, an amount of computing resources spent by the trained model to analyze a single ablated image to predict growth rate as compared to the amount of computing resources spent by the initial model to analyze the corresponding original retinal image, one
• Step 410 includes generating an output for use in improving performance of the initial model based on the performance of the plurality of trained models.
• Step 410 may be performed in various ways.
• the output may be an identification of the ablated image group corresponding to the trained model that had the best performance.
• the trained model corresponding to a rim retained image group may be identified as having the best performance.
• the output may identify the rim of the GA lesion as most relevant to the prediction of growth rate.
• the output may further identify one or more biomarkers associated with this rim region with an indication that focusing on these one or more biomarkers may improve model performance with respect to speed and computing resources utilized. In this manner, identifying the rim of the GA lesion as most relevant to the prediction of growth rate may help localize the biomarkers of interest.
• Process 400 may optionally include step 412, which includes adjusting the initial model based on the output to form a new model.
• adjusting the initial model includes narrowing the biomarkers analyzed by the initial model to those associated with the region (e.g., GA lesion, rim of the GA lesion, or background) of the retinal identified as being most relevant to predicting growth rate.
• adjusting the initial model includes integrating a supplemental model (which may itself include one or more algorithms or models) as part of the initial model to form thew model or combining the supplemental model with the initial model to form the new model.
• the supplemental model may, for example, be used to segment an input retinal image and form an ablated image based on this segmentation.
• the new model uses the ablated image to predict growth rate, which may be faster and/or consume fewer computing resources than using the non-ablated retinal image to predict growth rate.
• A. Exemplary Visualization Outputs Generated for Two Deep Learning Models [0067] An experiment was conducted in which two different deep learning models were trained and tested using retinal images for a plurality of subjects. These retinal images were baseline FAF images, each of which captured a GA lesion which may have been a unifocal GA lesion or a multifocal lesion. The first deep learning model and the second deep learning model were both used to predict lesion growth rate for the GA lesions. Visualization outputs were generated for these two deep learning models. Specifically, gradient activation maps were generated for the two deep learning models to provide information about what portions of each retinal image were ultimately relevant to the growth rate predicted by the corresponding deep learning model.
• Figure 5 is a chart comparing the gradient activation maps generated for the two deep learning models in accordance with one or more embodiments.
• first group of gradient activation maps 502 which was generated for the first deep learning model (1 st DL Model) is one example of an implementation for at least a portion of set of gradient activation maps 124 in Figure 1.
• first group of gradient activation maps 502 and the five gradient activation maps in second group of gradient activation maps 504 were generated for the same group of five retinal images, each of which captured a unifocal GA lesion. Comparing first group of gradient activation maps 502 with second group of gradient activation maps 504 reveals that for retinal images of unifocal GA lesions, different portions of these retinal images were relevant to the first deep learning model as compared to the second deep learning model.
• Figure 6 is a chart comparing the gradient activation maps generated for the two deep learning models in accordance with one or more embodiments.
• first group of gradient activation maps 602 which was generated for the first deep learning model (1 st DL Model) is one example of an implementation for at least a portion of set of gradient activation maps 124 in Figure 1.
• FIG. 7 is a chart depicting exemplary ablated images in accordance with one or more embodiments.
• Each of ablated images 700 may be one example of an implementation for an ablated image of ablated images 130 in Figure 1.
• Ablated images 700 include a lesion ablated image 702, a rim ablated image 704, a background ablated image 706, a lesion retained image 708, a rim retained image 710, a background retained image 712, a lesion shuffled image 714, a rim shuffled image 716, and a background shuffled image 718.
• FIG 8 is a block diagram of a computer system in accordance with various embodiments.
• Computer system 800 may be an example of one implementation for computing platform 102 described above in Figure 1.
• computer system 800 can include a bus 802 or other communication mechanism for communicating information, and a processor 804 coupled with bus 802 for processing information.
• computer system 800 can also include a memory, which can be a random-access memory (RAM) 806 or other dynamic storage device, coupled to bus 802 for determining instructions to be executed by processor 804.
• RAM random-access memory
• Memory also can be used for storing temporary variables or other intermediate information during execution of instructions to be executed by processor 804.
• computer system 800 can further include a read only memory (ROM) 808 or other static storage device coupled to bus 802 for storing static information and instructions for processor 804.
• ROM read only memory
• a storage device 810 such as a magnetic disk or optical disk, can be provided and coupled to bus 802 for storing information and instructions.
• computer system 800 can be coupled via bus 802 to a display 812, such as a cathode ray tube (CRT) or liquid crystal display (LCD), for displaying information to a computer user.
• a display 812 such as a cathode ray tube (CRT) or liquid crystal display (LCD)
• An input device 814 can be coupled to bus 802 for communicating information and command selections to processor 804.
• a cursor control 816 such as a mouse, a joystick, a trackball, a gesture input device, a gaze-based input device, or cursor direction keys for communicating direction information and command selections to processor 804 and for controlling cursor movement on display 812.
• results can be provided by computer system 800 in response to processor 804 executing one or more sequences of one or more instructions contained in RAM 806.
• Such instructions can be read into RAM 806 from another computer- readable medium or computer-readable storage medium, such as storage device 810.
• Execution of the sequences of instructions contained in RAM 806 can cause processor 804 to perform the processes described herein.
• hard-wired circuitry can be used in place of or in combination with software instructions to implement the present teachings.
• implementations of the present teachings are not limited to any specific combination of hardware circuitry and software.
• computer-readable medium e.g., data store, data storage, storage device, data storage device, etc.
• computer-readable storage medium refers to any media that participates in providing instructions to processor 804 for execution.
• Such a medium can take many forms, including but not limited to, non-volatile media, volatile media, and transmission media.
• non volatile media can include, but are not limited to, optical, solid state, magnetic disks, such as storage device 810.
• volatile media can include, but are not limited to, dynamic memory, such as RAM 806.
• transmission media can include, but are not limited to, coaxial cables, copper wire, and fiber optics, including the wires that comprise bus 802.
• Computer-readable media include, for example, a floppy disk, a flexible disk, hard disk, magnetic tape, or any other magnetic medium, a CD-ROM, any other optical medium, punch cards, paper tape, any other physical medium with patterns of holes, a RAM, PROM, and EPROM, a FLASH-EPROM, any other memory chip or cartridge, or any other tangible medium from which a computer can read.
• the methods of the present teachings may be implemented as firmware and/or a software program and applications written in conventional programming languages such as C, C++, Python, etc. If implemented as firmware and/or software, the embodiments described herein can be implemented on a non-transitory computer-readable medium in which a program is stored for causing a computer to perform the methods described above. It should be understood that the various engines described herein can be provided on a computer system, such as computer system 800, whereby processor 804 would execute the analyses and determinations provided by these engines, subject to instructions provided by any one of, or a combination of, the memory components RAM 806, ROM, 808, or storage device 810 and user input provided via input device 814.
• one element e.g., a component, a material, a layer, a substrate, etc.
• one element may be “on,” “attached to,” “connected to,” or “coupled to” another element regardless of whether the one element is directly on, attached to, connected to, or coupled to the other element or there are one or more intervening elements between the one element and the other element.
• subject may refer to a subject of a clinical trial, a person undergoing treatment, a person undergoing anti-cancer therapies, a person being monitored for remission or recovery, a person undergoing a preventative health analysis (e.g., due to their medical history), or any other person or patient of interest.
• subject and patient may be used interchangeably herein.
• the term “plurality” can be 2, 3, 4, 5, 6, 7, 8, 9, 10, or more.
• a set of means one or more.
• a set of items includes one or more items.
• the phrase “at least one of,” when used with a list of items, may mean different combinations of one or more of the listed items may be used and only one of the items in the list may be needed.
• the item may be a particular object, thing, step, operation, process, or category.
• “at least one of’ means any combination of items or number of items may be used from the list, but not all of the items in the list may be required.
• “at least one of item A, item B, or item C” means item A; item A and item B; item B; item A, item B, and item C; item B and item C; or item A and C.
• “at least one of item A, item B, or item C” means, but is not limited to, two of item A, one of item B, and ten of item C; four of item B and seven of item C; or some other suitable combination.
• a “model” may include one or more algorithms, one or more mathematical techniques, one or more machine learning algorithms, or a combination thereof.
• a neural network may process information in two ways; when it is being trained it is in training mode and when it puts what it has learned into practice it is in inference (or prediction) mode.
• Neural networks learn through a feedback process (e.g., backpropagation) which allows the network to adjust the weight factors (modifying its behavior) of the individual nodes in the intermediate hidden layers so that the output matches the outputs of the training data.
• a neural network learns by being fed training data (learning examples) and eventually learns how to reach the correct output, even when it is presented with a new range or set of inputs.
• a neural network may include, for example, without limitation, at least one of a Feedforward Neural Network (FNN), a Recurrent Neural Network (RNN), a Modular Neural Network (MNN), a Convolutional Neural Network (CNN), a Residual Neural Network (ResNet), an Ordinary Differential Equations Neural Networks (neural-ODE), or another type of neural network.
• FNN Feedforward Neural Network
• RNN Recurrent Neural Network
• MNN Modular Neural Network
• CNN Convolutional Neural Network
• Residual Neural Network Residual Neural Network
• Neural-ODE Ordinary Differential Equations Neural Networks
• a “lesion” may be a region in an organ or tissue that has suffered damage via injury or disease. This region may be a continuous or discontinuous region.
• a lesion may include multiple regions.
• a geographic atrophy (GA) lesion may be a region of the retina that has suffered chronic progressive degeneration.
• a GA lesion may include one lesion (e.g., one continuous lesion region) or multiple lesions (e.g., discontinuous lesion region comprised of multiple, separate lesions).
• a “lesion area” may mean the total area covered by a lesion, whether that lesion be a continuous region or a discontinuous region.
• longitudinal may mean over a period of time.
• the period of time may be in days, weeks, months, years, or some other measure of time.
• Embodiment 1 A method for evaluating geographic atrophy, the method comprising: receiving a set of retinal images; training each model of a plurality of models to predict a set of geographic atrophy (GA) progression parameters for a geographic atrophy (GA) lesion using the set of retinal images; and generating a visualization output for each model of the plurality of models, wherein the visualization output for a corresponding model of the plurality of models provides information about how the corresponding model uses the set of retinal images to predict the set of GA progression parameters.
• GA geographic atrophy
• GA geographic atrophy
• Embodiment 2 The method of embodiment 1, wherein the generating comprises: generating a gradient activation map for a corresponding retinal image in the set of retinal images for the corresponding model, wherein the gradient activation map indicates a set of regions in the corresponding retinal image that contributed to the set of GA progression parameters predicted by the corresponding model for the GA lesion.
• Embodiment 3 The method of embodiment 1 or embodiment 2, wherein the plurality of models includes a deep learning model and further comprising: validating the deep learning model using the visualization output generated for the deep learning model.
• Embodiment 5 The method of embodiment 4, further comprising: selecting either the first deep learning model or the second deep learning model as a best model for predicting the set of GA progression parameters based on the comparison.
• Embodiment 6 The method of any one of embodiments 1-5, further comprising: modifying a model of the plurality of models to form a new model based on the visualization output generated for the model to improve a performance of the model.
• Embodiment 7 The method of any one of embodiments 1-6, wherein the set of GA progression parameters comprises at least one of a growth rate for the GA lesion or a baseline lesion area for the GA lesion.
• Embodiment 9 The method of embodiment 8, wherein the set of fundus autofluorescence (FAF) images is a set of baseline FAF images and wherein the set of optical coherence tomography (OCT) images is a set of baseline OCT images.
• FAF fundus autofluorescence
• OCT optical coherence tomography
• a method for evaluating geographic atrophy in a retina comprising: receiving a set of retinal images; predicting a set of geographic atrophy (GA) progression parameters for a geographic atrophy (GA) lesion in the retina using the set of retinal images and a deep learning model; and generating a set of gradient activation maps corresponding to the set of retinal images for the deep learning model, wherein a gradient activation map in the set of gradient activation maps for a corresponding retinal image of the set of retinal images identifies a set of regions in the corresponding retinal image that is relevant to predicting the set of GA progression parameters by the deep learning model.
• Embodiment 11 Embodiment 11.
• Embodiment 12 The method of embodiment 10 or embodiment 11, wherein the set of GA progression parameters comprises at least one of a growth rate for the GA lesion or a baseline lesion area for the GA lesion.
• Embodiment 13 The method of any one of embodiments 10-12, wherein the set of retinal images comprises at least one of a set of fundus autofluorescence (FAF) images or a set of optical coherence tomography (OCT) images.
• FAF fundus autofluorescence
• OCT optical coherence tomography
• Embodiment 14 A system for evaluating geographic atrophy, the system comprising: a memory containing machine readable medium comprising machine executable code; and a processor coupled to the memory, the processor configured to execute the machine executable code to cause the processor to: receive a set of retinal images; train each model of a plurality of models to predict a set of geographic atrophy (GA) progression parameters for a geographic atrophy (GA) lesion using the set of retinal images; and generate a visualization output for each model of the plurality of models, wherein the visualization output for a corresponding model of the plurality of models provides information about how the corresponding model uses the set of retinal images to predict the set of GA progression parameters.
• Embodiment 15 Embodiment 15.
• the visualization output includes a gradient activation map for a corresponding retinal image in the set of retinal images for the corresponding model, and wherein the gradient activation map indicates a set of regions in the corresponding retinal image that contributed to the set of GA progression parameters predicted by the corresponding model for the GA lesion.
• Embodiment 17 The system of any one of embodiments 14-16, wherein the plurality of models includes a first deep learning model and a second deep learning model and wherein the processor is configured to execute the machine executable code to cause the processor to: perform a comparison of the visualization output generated for the first deep learning model with the visualization output generated for the second deep learning model; and select either the first deep learning model or the second deep learning model as a best model for predicting the set of GA progression parameters based on the comparison. [0116] Embodiment 18.
• processor is configured to execute the machine executable code to cause the processor to modify a model of the plurality of models to form a new model based on the visualization output generated for the model to improve a performance of the model.
• Embodiment 19 The system of any one of embodiments 14-18, wherein the set of GA progression parameters comprises at least one of a growth rate for the GA lesion or a baseline lesion area for the GA lesion.
• Some embodiments of the present disclosure include a system including one or more data processors.
• the system includes a non-transitory computer readable storage medium containing instructions which, when executed on the one or more data processors, cause the one or more data processors to perform part or all of one or more methods and/or part or all of one or more processes disclosed herein.
• Some embodiments of the present disclosure include a computer-program product tangibly embodied in a non-transitory machine -readable storage medium, including instructions configured to cause one or more data processors to perform part or all of one or more methods and/or part or all of one or more processes disclosed herein.

Landscapes

• Engineering & Computer Science (AREA)
• Health & Medical Sciences (AREA)
• Medical Informatics (AREA)
• Public Health (AREA)
• General Health & Medical Sciences (AREA)
• Biomedical Technology (AREA)
• Epidemiology (AREA)
• Primary Health Care (AREA)
• Data Mining & Analysis (AREA)
• Radiology & Medical Imaging (AREA)
• Pathology (AREA)
• Databases & Information Systems (AREA)
• Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
• Theoretical Computer Science (AREA)
• General Physics & Mathematics (AREA)
• Physics & Mathematics (AREA)
• Computer Vision & Pattern Recognition (AREA)
• Quality & Reliability (AREA)
• Eye Examination Apparatus (AREA)
• Software Systems (AREA)
• Image Analysis (AREA)
• Medical Treatment And Welfare Office Work (AREA)
• Artificial Intelligence (AREA)
• Evolutionary Computation (AREA)
• Computing Systems (AREA)
• General Engineering & Computer Science (AREA)
• Mathematical Physics (AREA)

Priority Applications (5)

Applications Claiming Priority (2)

Related Child Applications (1)

Publications (1)

Family

ID=81975322

Family Applications (1)

Country Status (6)

Families Citing this family (2)

Citations (3)

Family Cites Families (2)

• 2022
  • 2022-05-17 CN CN202280035158.5A patent/CN117296104A/zh active Pending
  • 2022-05-17 JP JP2023571163A patent/JP2024521070A/ja active Pending
  • 2022-05-17 EP EP22728740.6A patent/EP4341951A1/en active Pending
  • 2022-05-17 KR KR1020237039344A patent/KR20240011140A/ko active Pending
  • 2022-05-17 WO PCT/US2022/029699 patent/WO2022245873A1/en not_active Ceased
• 2023
  • 2023-11-17 US US18/513,106 patent/US20240087120A1/en active Pending

Patent Citations (3)

Non-Patent Citations (5)

Also Published As

Similar Documents

Legal Events