DE102014014814A1 - Method for the detection and classification of AMD based on at least two modalities of the respective eye - Google Patents

Abstract

The present invention relates to a method with which age-related macular degeneration can be detected and classified on the basis of at least two modalities of the relevant eye. The method according to the invention for the detection and classification of AMD is based on modalities of the relevant eye which were recorded at different times and / or under different conditions by means of different recording methods. In the process, signal-based features which are suitable for describing the sought-after biomarkers are extracted from all recorded modalities registered with respect to a common coordinate system. These extracted, signal-based features are statistically classified and combined or first combined and then classified to determine a probability map for the disease in AMD. Although the proposed method is intended for the detection and classification of AMD, it can also be used for the detection and classification of diabetic retinopathy or Alzheimer's disease, depending on the biomarkers to be detected.

Description

The present invention relates to a method with the age-related macular degeneration (AMD short) can be detected and classified using at least two modalities of the eye in question.

Modalities are to be understood below as meaning the image data recorded by imaging methods used in medical diagnostics.

The finding of AMD in the available image data represents a general challenge in medical diagnostics. In particular, the detection or the follow-up of pathological changes in the retina for the medical professional, for example, a doctor or a so-called "Grader" in a reading center, often very difficult and leads to inadequate quality of diagnosis.

These complex and usually not always objectively defined relationships lead on the one hand to a considerable amount of time for the diagnosis of AMD and on the other hand to variability between the findings.

For a diagnosis, a finder has to manually record and evaluate the multimodal complementary information and its complex relationships. Due to the complexity of the multimodal data, there is the danger that important relationships will not be recognized or misallocated in different modalities.

Particularly challenging in the diagnosis of AMD is the need for the simultaneous detection of several clinical symptoms (for example, drusen and hyperpigmentation for the AREDS classification) during image data diagnosis. In addition, the symptoms usually have to be spatially and temporally related to each other or to anatomical landmarks (eg macula). By way of example, for a classification according to AREDS, the covered area of hyperpigmentation within macular segments and the presence of drusen with a certain minimum area are decisive for the severity of the clinical picture.

Drusen are deposits of extracellular material below the retina. Although they occur in all age groups, they increase in size and number with increasing age. They are considered as an early form of age-related macular degeneration.

Since a clinical picture can consist of several clinical symptoms, which appear differently in different modalities, image recordings of several modalities and / or different temporal recordings can be present for a patient.

For example, some hyperpigmentations can not only be detected in color fundus photography, but are also visible as bright spots in autofluorescent images. This image modality can thus additionally support the detection of hyperpigmentation in color fundus images.

The situation is similar for drusen, which are better recognizable in OCT volume images than in the flat color fundus images.

For a diagnosis, a finder must now manually record and evaluate this multimodal complementary information and its complex relationships. Due to the complexity of the multimodal data (eg multiple channels and multiple layers), there is the danger that important connections will not be recognized or misallocated in different modalities. Therefore, a multimodal disease symptom detection system (so-called lesions) will be presented below to support AMD diagnosis (eg, according to the AREDS Score) of multimodal retinal data.

The automatic recognition of anatomical landmarks, in particular of the retinal vascular tree [2], as well as the uni- and multimodal detection and classification of drusen, [3], [4], [5], [6 ] or hyperpigmentation in color fundus images or OCT images known.

There are also solutions in the patent literature. So will in the US 7,474,775 B2 the automatic detection of lesions fundus recordings described. The fundus pictures with a very high Resolution is first preprocessed, objects that can represent lesions are extracted, and in the last step, the probability of each object for a lesion is estimated using a classifier.

Another system for the analysis of fundus images is in the US Pat. No. 7,856,135 B1 described. With the aid of an image analysis system, fundus images are imported, normalized and registered in order to analyze them and determine differences between the images. Sequences of these fundus images show changes in the eye over time. The information is mainly used for diagnostic purposes.

Also in the US 2012/0237096 A1 describes a method for the diagnosis of eye diseases based on fundus recordings and pathologies. In the fundus image of a patient, regions or structures of interest are identified by means of image processing. These are then described using a variety of feature vectors. For a clear diagnosis, the current fundus image is compared with at least one fundus image from an archive containing similar regions or structures. In addition to the fundus images, the corresponding clinical pictures are stored in the archive. In an at least partial agreement of the compared recordings the patient can be made a reliable diagnosis.

Retmarker from Coimbra (Portugal) has developed a history monitoring tool for age-related macular degeneration. However, it has the disadvantage that anatomical landmarks (eg optical nerve) and also lesions have to be marked manually before a statistical evaluation (size, area,.

• • Jeder Klassifikator liefert in der Regel auch Falschalarme für Symptome/Läsionen. Unabhängige Klassifikatoren würden zu einer Aufsummierung der Falschalarme führen. Ist die Menge der Falschalarme jedoch zu groß, würde das System das Vertrauen des Arztes verlieren und dadurch inpraktikabel werden. Ferner würde das Reviewen von zu vielen Falschalarmen den zeitlichen Aufwand des Arztes nicht reduzieren.
• • Kontextinformationen, d. h. räumliche und/oder zeitliche Abhängigkeiten zwischen verschiedenen Läsionstypen werden nicht berücksichtigt, obwohl es klinisch häufig vorkommt, dass die Präsenz von Läsionstyp A auch eine höhere Auftrittswahrscheinlichkeit für Läsionstyp B impliziert.
• • Strategien zur Datenfusion der Ergebnisse zweier oder mehrerer Klassifikatoren für unterschiedliche Läsionen sind nicht vorhanden.

As mentioned in the introduction, however, it is usually not enough to detect only one symptom for a comprehensive diagnosis. Therefore, an obvious approach would be to apply the existing classifiers independently for the different symptoms / lesions. However, this approach has the following serious disadvantages:

• • Each classifier also usually provides false alerts for symptoms / lesions. Independent classifiers would lead to a summation of the false alarms. However, if the amount of false alarms is too large, the system would lose the physician's confidence and thus become impractical. Furthermore, reviewing too many false alarms would not reduce the physician's time.
• • Contextual information, ie spatial and / or temporal dependencies between different lesion types are not considered, although it is clinically common that the presence of lesion type A also implies a higher probability of occurrence of lesion type B.
• • There are no strategies for data fusion of the results of two or more classifiers for different lesions.

Publications on the AREDS classification [1] of an image are also known. However, such approaches have the disadvantage that it is not possible to monitor the object level. Furthermore, the classification is difficult for the physician to understand because it can be badly substantiated by objective criteria.

Literature:

• [1] Frederick L. Ferris and others; A Simplified Severity Scale for Age-Related Macular Degeneration: Age-Related Eye Disease Study Research Group - AREDS "; Arch Ophthalmol. 2004; 122: 723-724
• [2] Zhihong Hu; "Multimodal Retinal Vessel Segmentation from Spectral Domain Optical Coherence Tomography and Fundus Photography", IEEE Transactions on Medical Imaging, VOL. 31, no. 10, OCTOBER 2012
• [3] Daisuke Iwama, et al. "Automated Assessment of Drusen Using Three Dimensional Spectral Domain Optical Coherence Tomography", Investigative Ophthalmology & Visual Science, March 2012, Vol. 3, pp 1576-1583
• [4] Richard F. Spaide and Christine A. Curcio; "Drusen Characterization with Multimodal Imaging"; Author manuscript; available in PMC 2011 October 1
• [5] Yehoshua Z, et al. "Comparison of drusen area detected by spectral domain optical coherence tomography and color fundus imaging"; Invest Ophthalmol Vis Sci. 2013
• [6] Francisco A. Folgar, et al. "Spatial Correlation between Hyperpigmentary Changes on Color Fundus Photography and Hyperreflective Foci on SDOCT in Intermediate AMD"; Investigative Ophthalmology & Visual Science, July 2012, Vol. 53, no. 8th

The object of the present invention is to further develop the method for the diagnosis of AMD on the basis of image data in such a way that the detection and classification can take place as simply and quickly as possible and a high quality of the diagnosis is achieved. The procedure is intended to assist the diagnosis in its diagnosis, thereby reducing the variability between the individual findings.

According to the invention, this object is achieved by the method for the automated or semi-automated detection and classification of AMD, based on modalities of the eye in question with at least partially identical areas, which were recorded at different times and / or under different conditions by means of different recording methods using different ophthalmological examination devices solved that all recorded modalities are registered with respect to a common coordinate system, that extracted from all recorded and registered modalities signal-based characteristics that are suitable to describe the biomarkers sought, that the extracted, signal-based characteristics of the recorded modalities by statistical methods classified and before or are combined according to their classification and from this the classification result in the form of a probability map for the Disease on AMD is determined. This probability map can then be used to derive further key figures that serve to describe the clinical picture and the course of the disease.

According to the invention the object is solved by the features of the independent claims. Preferred developments and refinements are the subject of the dependent claims.

The proposed method is intended for the automated or semi-automated detection and classification of AMD. Depending on the biomarkers to be detected, the proposed method can also be used for the detection and classification of diabetic retinopathy (DR for short) or Alzheimer's disease (MA for short). The results of the procedure, regardless of the disease being diagnosed, can be used both for the quantitative description of the disease and for the quantitative characterization of the course of the disease.

In medicine or biology refers to measurable products of organisms that are used as indicators z. B. are used for environmental pollution or disease, as a biomarker.

Diabetic retinopathy is a disease of the retina of the eye caused by the diabetes mellitus. The increasing damage to small blood vessels causes an initially unnoticed damage to the retina. It can lead to blindness in the course.

Alzheimer's disease (Latin: Alzheimer's disease) is a disease that occurs in its most common form in people over the age of 65 and accounts for approximately 60 percent of dementias.

The invention will be described in more detail below with reference to exemplary embodiments.

The method proposed here for the automated or semi-automated detection and classification of AMD is based on modalities of the relevant eye with at least partially identical areas, which were recorded at different times and / or under different conditions by means of different recording methods using different ophthalmological examination devices.

According to the invention, all recorded modalities are registered with respect to a common coordinate system, signal-based features are extracted from all recorded and registered modalities that are suitable for describing the searched biomarkers, classifying the extracted, signal-based features of the recorded modalities by statistical methods and combining them before or after their classification in order to determine the classification result in the form of a probability map for the disease of AMD.

For the proposed method and in particular for the processing and corresponding evaluation of the modalities taken at different times and / or under different conditions by means of different recording methods using different ophthalmological examination devices, preferably a PC, laptop, tablet PC or smart phone is used. To display the results, a display should be available.

• • Drusenbildung,
• • Hyperpigmentierung,
• • atrophische Areale,
• • Hyper-/Hypofluoreszenzen,
• • Choroidale Neovaskularisationen (CNV);
• • frühe entzündliche Prozesse,
• • Expression vom Vascular Endothelial Growth Factor (VEGF),
• • Bildung von ersten „Feeder Vessel” (FV) und
• • retinalen Einblutungen.

Age-related macular degeneration (AMD) can be diagnosed by the following features:

• • drusen formation,
• • hyperpigmentation,
• • atrophic areas,
• • hyper- / hypofluorescences,
• • Choroidal neovascularization (CNV);
• • early inflammatory processes,
• Expression of Vascular Endothelial Growth Factor (VEGF),
• • Formation of first "Feeder Vessel" (FV) and
• • retinal bleeding.

• • „Block-matching via cross-correlation”;
• • „Keypoint matching with robust outlier rejection”;
• • „Optimization with gradient descent methods”;
• • „Optimization with variational methods”;
• • „Vessel segmentation, matching via cross-correlation”.

For the registration of all recorded modalities with respect to a common coordinate system, it is necessary that in the individual modalities automatically characteristic points are recognized, which can be detected with high probability in the other modalities. This makes it possible to produce a transformation between the image data of the various modalities. The type of registration algorithm used may depend on the image modality (s) used. Thus, rigid and / or non-rigid approaches can be used in combination with any distance measures, such as:

• • "Block-matching via cross-correlation";
• • "Keypoint matching with robust outlier rejection";
• • "Optimization with gradient descent methods";
• • "Optimization with variational methods";
• • "Vessel segmentation, matching via cross-correlation".

• • einer Farbe,
• • einer Form,
• • einer Textur,
• • einer Orientierung,
• • einer Gradientenverteilung (Histogram of Oriented Gradients) oder
• • Filtern, die datengetrieben aus einer Trainingsmenge bestimmt werden.

After registering all recorded modalities, signal-based features are extracted which are suitable for describing the biomarkers sought. For this purpose, algorithms are used for the extraction of low-level features, which are based for example on:

• • one color,
• • a shape
• • a texture,
• • an orientation,
• • a gradient distribution (Histogram of Oriented Gradients) or
• • Filters that are data-driven from a training set.

• • Nearest Neighbor,
• • Support Vector Machine,
• • Random Forests,
• • Boosting (AdaBoost, GentleBoost),
• • Convolutional Neural Networks oder
• • Logistic Regression.

To classify the extracted, signal-based characteristics, statistical classifiers are used, such as:

• • Nearest Neighbor,
• • Support Vector Machine,
• • Random Forests,
• • Boosting (AdaBoost, GentleBoost),
• • Convolutional Neural Networks or
• • Logistic regression.

• • GraphCut,
• • Belief Propagation in unterschiedlichen Ausprägungen,
• • Mean Field Inferenz oder
• • Gibbs sampling.

For the combination of the locally extracted and classified features of the individual modalities, algorithms for global inference in graphical models can optionally be used, such as:

• • GraphCut,
• • Belief propagation in different forms,
• • Mean Field Inference or
• • Gibbs sampling.

The finding underlying the proposed solution is that different imaging techniques are able to represent and resolve different information. For example, optical coherence tomography (OCT) allows cross-sectional images of the various layers of the retina and three-dimensional images composed of them. In contrast, a fundus camera can realize photographic and fluorescence images of the retina with a high resolution. It is thus possible to obtain complementary information through the use of multiple imaging techniques.

In particular, the modalities that can be used can be based on optical methods as well as X-ray radiation, ultrasound, magnetic fields, nuclear radiation (nuclear medicine) or else endoscopy or microscopy.

• • Colorfundusphotografie (CFP),
• • Fluoreszenzangiographie/-bildgebung,
• • Molekulare Bildgebung mit endogenen oder exogenen Markern,
• • Fundusautofluoreszenz (FAF),
• • Polarisationsbildgebung,
• • Fluoreszenzlebensdauer,
• • Hyperspektrale Bildgebung,
• • Optische Kohärenztomographie (OCT),
• • Infrarotbildgebung oder
• • Photoakkustik.

Preferred modalities here are all known retinal imaging methods, such as, for example:

• Color photographic photography (CFP),
• Fluorescence angiography / imaging,
• Molecular imaging with endogenous or exogenous markers,
• Fundus autofluorescence (FAF),
• • polarization imaging,
• Fluorescence lifetime,
• • Hyperspectral imaging,
• Optical coherence tomography (OCT),
• • Infrared imaging or
• • Photoakkustik.

• • Computertomographie (kurz: CT),
• • Magnetresonanztomographie (kurz: MRT),
• • Positronen-Emissions-Tomographie (kurz: PET),
• • Single-Photonen-Emissions-Computertomographie (kurz: SPECT) oder
• • Ultraschall, Endoskopie, sowie mikroskopische Aufnahmen jeder Art.

However, other imaging techniques are also conceivable, such as, for example:

• • computed tomography (short: CT),
• • magnetic resonance imaging (MRT for short),
• • positron emission tomography (short: PET),
• • single photon emission computed tomography (short: SPECT) or
• • Ultrasound, endoscopy and microscopic images of all kinds.

According to the invention, at least two, preferably three and particularly preferably four or more modalities can be recorded, their extracted, signal-based features being classified by means of a one-stage or multistage statistical method. Preferably, both the classification result and the modalities classified with regard to the presence of the searched biomarker are output.

According to a first embodiment variant, two or more recorded modalities are registered with respect to a common coordinate system in which recorded and registered modalities extract signal-based features capable of describing the sought-after biomarkers that independently classify extracted signal-based features of the modalities by statistical methods which extracted and classified, signal-based features of the modalities combined and determined therefrom the classification result in the form of a probability map for the disease in AMD.

In this embodiment, the extracted features from each input modality are first independently classified. Subsequently, these individual results are combined in a further step to obtain a final classification result. The combination is done, for example, by weighting all the individual results.

According to a second embodiment variant, two or more recorded modalities will be registered with respect to a common coordinate system in which recorded and registered modalities extract signal-based features capable of describing the searched biomarkers combining extracted and classified signal-based features of the modalities extracted and Combined, signal-based features of the modalities are classified by statistical methods and used to determine the classification result in the form of a probability map for the disease of AMD.

The use of different imaging methods has, in addition to the already mentioned advantage that thereby complementary information on existing lesions are available, a further advantage that has not yet been considered in previously described solutions.

So exist between the occurrence of lesions spatial and / or temporal relationships. For example, in a disease of AMD hyperpigmentations often occur in close proximity to drusen. In diabetic retinopathy, microaneurysms often occur near blood vessels.

According to the invention, the extracted, signal-based features of the recorded modalities before or after their classification are therefore combined using known context information.

There are different possibilities for modeling context. In one embodiment, this may be based on low-level probability maps that are used in a first classification step be calculated. A probability map indicates for each image or spatial point a probability for the occurrence of a lesion to be detected.

From this, features are calculated which can be used as additional input modality in a further classification or optimization step.

According to a third embodiment variant, the extracted, signal-based features of two or more recorded and registered modalities are combined on the basis of known context information and classified by statistical methods in order to determine therefrom the classification result in the form of a probability map for the disease at AMD. It is irrelevant whether the extracted, signal-based features of the two modalities are combined before or after their classification on the basis of known context information.

For the combination of different modalities, in one embodiment, the extracted features are used by means of a one- or multistage statistical classifier to determine a decision function which classifies the image content according to the presence or absence of the biomarkers sought. From each input modality features are extracted and combined in a common classifier.

In this case, the decision function can also be determined independently of the statistical occurrence in a learning quantity manually, for example with the aid of a physical model. Irrespective of the above characteristics, the decision function can classify the image as a whole or else output decisions for subregions.

According to a third embodiment variant, two or more recorded modalities are registered with respect to a common coordinate system, extracted in a first, recorded and registered modality signal-based features that are suitable for describing the searched biomarkers, which classifies the extracted, signal-based features of this first modality by means of statistical methods, the classified features of this first modality used to further classify particularly interesting subareas using further recorded and registered modalities by additionally extracting signal-based features in the further modalities and re-classifying them by statistical methods together with the features of the preceding modalities or even without them to determine the classification result in the form of a probability map for the disease of AMD.

In this embodiment, conspicuous regions are determined by means of the features from an input modality, which are then further classified based on the features of other modalities. The different input modalities are processed sequentially. Conspicuous regions are calculated and passed on to the following modality. Also this sequential approach is independent of the way of determining the decision function.

In a further embodiment, based on the determined classification result in the form of a probability map on the basis of a scale, a phenotypic severity of a disease is determined on AMD.

Preferably, the determined phenotypic severity of the disease in AMD is used to more accurately determine its individual risk of disease in conjunction with genetic information of the particular patient.

The phenotype or appearance in genetics is the set of all the characteristics of an organism. It does not only refer to morphological but also to physiological and psychological characteristics. The phenotype is determined by the interaction of hereditary factors and environmental factors. The phenotype is thus the sum of all observable features of the organism.

According to a first embodiment, the accommodated modalities are based on the retinal imaging methods CFP and OCT As described above, the recorded modalities are registered with respect to a common coordinate system extracted from the CFP-based modality size and number of drusen as well as hyperpigmentation, from the OCT-based modality Extracting size, number and volume of drusen as well as hyperpigmentation, the extracted, signal-based features of the two modalities are classified by statistical methods and before or afterwards using known Context information combined to determine the classification result as the severity of the disease in AMD.

In accordance with a particularly preferred embodiment, a third modality based on the retinal imaging method FAF is additionally recorded and extracted with respect to the common coordinate system, extracted in this hyperpigmentation, the extracted signal-based features of the three modalities are classified by statistical methods and combined beforehand or thereafter using known context information to determine the classification result as the severity of the disease in AMD.

For the classification of AMD, z. For example, according to the Simplified Severity Scale [4], it is necessary to detect the large drusen and pigment changes, such as hyper- and hypopigmentations.

However, each classifier usually also provides false alarms for symptoms / lesions, i. H. the system detects symptoms / lesions, which are not. Independent classifiers would therefore lead to a summation of the false alarms. However, if the amount of false alarms is too large, the system would lose the physician's confidence and thus become impractical. Furthermore, reviewing too many false alarms would not reduce the physician's time.

This is remedied by the contextual combination of several lesion probability maps. If, for example, correct hyperpigmentation occurs predominantly on drusen, but false positives between drusen, an improved and more robust detection result can be achieved by combining the probability maps for drusen and hyperpigmentation. Both biomarkers (drusen and pigmentation) can be determined both uni- and multimodal, with the multimodal classification proving to be advantageous, especially in the detection of pigment changes.

According to a second embodiment, the accommodated modalities are based on the retinal imaging methods CFP and OCT As described above, the recorded modalities are registered with respect to a common coordinate system. From the CFP-based modality, hyperpigmentation is extracted and classified by statistical methods. The classified features of this first modality are used here to further classify regions of particular interest with the aid of the second recorded and registered modality by extracting hyperpigmentation from the OCT-based modality and classifying it by statistical methods to derive the classification result in the form of a probability map for the disease to determine AMD.

According to a particularly preferred embodiment, hyperpigmentation is extracted from the CFP-based modality and classified by means of statistical methods in terms of size and number. In the most interesting areas of the OCT-based modality, the distance d of hyperpigmentation to the retinal pigment epithelium is determined and evaluated with respect to healthy tissue to determine therefrom the classification result in the form of a probability map for the disease of AMD.

In this exemplary embodiment, the lesion is detected in the image data of the CFP-based modality and its position in the image data of the OCT-based modality is determined absolutely or relatively in the retinal tissue. This additional information allows a better assessment of the severity of the disease.

In this case, a first assessment of the severity of the disease in AMD may already be due to the situation, i. H. Size and number of hyperpigmentation in the retina take place. This evaluation is significantly improved by taking into account the measurements in the areas of interest of the second modality.

For this purpose, the distance d of the hyperpigmentation to the retinal pigment epithelium (RPE) is determined in the OCT-based modality and evaluated with respect to healthy tissue.

In healthy tissue, the distance d to the retinal pigment epithelium is between 0-12 μm. In contrast, the distance d in diseased tissue may be between 12 μm to 100 μm (foveal) or up to 230 μm (parafoveal), with the degree of the disease correlating with the distance d.

In a further embodiment, the results of different examination times can be compared in a multimodal manner, wherein the changes in the distance Δd are evaluated as follows: Δd = ± 10 μm quasi-stable state Δd <10 μm Improvement of the disease Δd> 10 μm Progression of the disease

In another embodiment, graphical models can be used to model neighborhood relationships. The information from different probability maps is used as input for the graph nodes. The probabilities of a neighborhood are modeled using the edge potentials. The graphical model can be defined both on a regular grid over the CFP or an OCT volume as well as on an object-related representation in which interesting image areas are grouped into objects.

The modeling of the context can be done both along spatial coordinates, ie by using information in the spatial neighborhood, as well as along the temporal dimension, where the course over any number of times can be considered. With the help of the temporal context, it is possible, for example, to model the occurrence of certain lesions in a disease stage by reusing already obtained information.

The finally obtained classification result can be used for the quantitative description of the disease (for example area of diseased tissue). If the modalities are recorded at different times, their description can also be used for the quantitative characterization of the course of the disease (for example, change in the area of the diseased tissue).

In one embodiment, the classification result may be used for attention control to direct the user's focus to particularly prominent areas of the input signal.

In a further embodiment, it is also possible to use the classification result for automated diagnosis or for therapeutic recommendations.

The proposed method shows how these different information / signals can be combined while exploiting spatial and temporal context information with the aim of improving automated recognition performance over using a single modality without context modeling.

Automated or semi-automated detected biomarkers of a disease can be used to improve the quality of diagnosis in a variety of forms. On the one hand, the doctor can be made aware of potentially pathological areas in the retina. Furthermore, the degree of disease can be automatically determined with appropriate robustness of the procedures and checked by the doctor. It is also possible to link the thus determined phenotype with genetic information (genotype) to assess the individual risk to the patient, eg. B. to get wet AMD, to be able to determine more precisely.

In contrast to the phenotype, the genotype or genetic image of an organism represents its exact genetic endowment, ie the individual set of genes that it carries in the cell nucleus and thus determines its morphological and physiological phenotype. The genotype does not change during the lifetime of an organism.

Further assistance functions, for example in eye surgery and the control of pharmacological interventions, are conceivable. Thus, an eye surgeon could easily find a pathological place through a navigational aid to perform therapeutic interventions there.

In the case of a pharmaceutical intervention, the efficacy and dosage can be controlled.

The proposed method initially provides for the multimodal registration of all recorded modalities in a common coordinate system. The procedure is independent of the specific modality chosen.

Subsequently, signal-based features are extracted from the registered input modalities that are suitable for describing the biomarkers sought. These depend on the disease to be detected and can also vary from input modality to input modality.

In particular, it is possible to extract a superset of suitable features in order to select them automatically in the detection step, for example automatically using methods of automatic feature selection. It is also possible to extract the features at different times in order to characterize the temporal course of the input modalities. For this purpose, it is possible in particular to extract features which are suitable for reproducing the temporal signal profile, such as features based on the so-called optical flow. For the classification of AMD in OCT volumes, for example, the change of the retinal layer thicknesses or the spatial "migration" of lesions could be an important feature.

With the solution according to the invention, a method is provided for the diagnosis of AMD on the basis of image data, with which the detection and classification can take place simply and quickly. The procedure achieves a high quality of diagnostics and reduces the variability between the individual findings.

Although the proposed method of AMD detection is intended, it can also be used to detect and classify DR or MA, depending on the biomarkers to be detected. While in DR DR as biomarkers aneurysms, edema, CNV or exudates are possible, these are in MA amyloid plaques or PHF.

While known disease feature detection techniques rely on features and classification schemes derived from a single input modality, the inventive method is based on features and classification schemes of multiple input modalities that preferentially exploited context information to improve the overall classification result.

The proposed method shows different ways to combine complementary information from different input modalities. Furthermore, possibilities for the explicit modeling of context information are provided. In particular, in clinical pictures that have several biomarkers, which must be recognized in different modalities and are in a certain spatial and temporal relation, it is thereby possible to achieve an improved automated recognition performance.

• – Die automatische Erkennung von Landmarken (z. B. Makularegion)
• – Die automatische Erkennung von unterschiedlichen Läsionen unter Verwendung von – Multimodalen Informationen, – Kontextspezifischen Informationen mit entsprechender Datenfusionsstrategie, – Dem zeitlichen Verlauf,
• – Die automatische Verlaufskontrolle (beispielhaft Größe, Fläche, Intensität, ...) auf Läsionsebene
• – Die automatische Klassifikation/Einstufung des Krankheitsbildes (bei mehr als einem Läsionstyp).

The present invention describes a method for multimodal diagnosis of AMD, which overcomes the disadvantages of the prior art solutions. These are summarized as follows:

• - Automatic recognition of landmarks (eg macular region)
• - The automatic detection of different lesions using - Multimodal information, - Context-specific information with appropriate data fusion strategy, - The time course,
• - The automatic progress control (example size, area, intensity, ...) on lesion level
• - The automatic classification / classification of the disease (in more than one lesion type).

QUOTES INCLUDE IN THE DESCRIPTION

This list of the documents listed by the applicant has been generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Cited patent literature

• US 7474775 B2 [0013]
• US 7856135 B1 [0014]
• US 2012/0237096 A1 [0015]

Cited non-patent literature

• Frederick L. Ferris and others; A Simplified Severity Scale for Age-Related Macular Degeneration: Age-Related Eye Disease Study Research Group - AREDS "; Arch Ophthalmol. 2004; 122: 723-724 [0020]
• Zhihong Hu; "Multimodal Retinal Vessel Segmentation from Spectral Domain Optical Coherence Tomography and Fundus Photography", IEEE Transactions on Medical Imaging, VOL. 31, no. 10, OCTOBER 2012 [0020]
• Daisuke Iwama, et al. "Automated Assessment of Drusen Using Three Dimensional Spectral Domain Optical Coherence Tomography", Investigative Ophthalmology & Visual Science, March 2012, Vol. 3, pp 1576-1583 [0020]
• Richard F. Spaide and Christine A. Curcio; "Drusen Characterization with Multimodal Imaging"; Author manuscript; available in PMC 2011 October 1 [0020]
• Yehoshua Z, et al. "Comparison of drusen area detected by spectral domain optical coherence tomography and color fundus imaging"; Invest Ophthalmol Vis Sci. 2013 [0020]
• Francisco A. Folgar, et al. "Spatial Correlation between Hyperpigmentary Changes on Color Fundus Photography and Hyperreflective Foci on SDOCT in Intermediate AMD"; Investigative Ophthalmology & Visual Science, July 2012, Vol. 53, no. 8 [0020]

Claims (20)

Method for the automated or semi-automated detection and classification of AMD, based on modalities of the eye concerned with at least partially identical areas, which were recorded at different times and / or under different conditions by means of different recording methods using different ophthalmological examination apparatus, characterized in that all recorded Modalities are registered with respect to a common coordinate system that extracted from all recorded and registered modalities signal-based characteristics that are suitable to describe the biomarker sought, that the extracted signal-based characteristics of the recorded modalities are classified by statistical methods and combined before or after their classification and from this the classification result in the form of a probability map for the disease is determined on AMD. A method according to claim 1, characterized in that the usable modalities can be based on optical methods as well as X-rays, ultrasound, magnetic fields, nuclear radiation or endoscopy or microscopy. A method according to claim 1, characterized in that at least two, preferably three and more preferably four or more modalities are included. A method according to claim 1, characterized in that the classification of the extracted, signal-based features by means of a one- or multi-stage statistical method. Method according to claim 1, characterized in that both the classification result and the modalities classified with regard to the presence of the searched biomarkers are output. Method according to claims 1 to 4, characterized in that two or more recorded modalities are registered with respect to a common coordinate system, that in the recorded and registered modalities signal-based features are extracted which are suitable for describing the searched biomarkers that the extracted, signal-based Characteristics of the modalities are statistically independently classified, that the extracted and classified, signal-based features of the modalities are combined and used to determine the classification result in the form of a probability map for the disease at AMD. Method according to claims 1 to 4, characterized in that two or more recorded modalities are registered with respect to a common coordinate system, that in the recorded and registered modalities signal-based features are extracted which are suitable for describing the searched biomarkers that the extracted and classified , signal-based features of the modalities are combined, that the extracted and combined, signal-based features of the modalities are classified by statistical methods and from this the classification result in the form of a probability map for the disease is determined at AMD. A method according to claim 1, characterized in that the extracted signal-based characteristics of the recorded modalities are combined before or after their classification on the basis of known context information. Method according to claims 1 to 8, characterized in that two or more recorded modalities are registered with respect to a common coordinate system, that in the recorded and registered modalities signal-based features are extracted which are suitable for describing the searched biomarkers that the extracted, signal-based Characteristics of the modalities are combined on the basis of known context information and classified by means of statistical methods and from this the classification result in the form of a probability map for the disease is determined on AMD. Method according to claims 1 to 8, characterized in that two or more recorded modalities are registered with respect to a common coordinate system, that in the recorded and registered modalities signal-based features are extracted which are suitable for describing the searched biomarkers that the extracted, signal-based Characteristics of the modalities are classified by means of statistical methods and then combined using known context information and from the classification result in the form of a probability map for the disease is determined on AMD. Method according to claims 1 to 8, characterized in that two or more recorded modalities are registered with respect to a common coordinate system, that in a first, recorded and registered modality signal-based features are extracted which are suitable for describing the searched biomarkers that the extracted ones , Signal-based features of this first modality are classified by statistical methods that the classified features of this first modality are used to further classify particularly interesting sections using the other recorded and registered modalities, by extracting additional signal-based features in the other modalities and statistical Methods are classified together with the features of the preceding modalities or even without them and from this the classification result in the form of a probability map for the disease determined on AMD. A method according to claim 1, characterized in that based on the determined classification result based on a scale, a phenotypic severity of a disease is determined on AMD. A method according to claim 12, characterized in that the determined phenotypic severity of the disease is used to AMD in order to more accurately determine in connection with genetic information of the respective patient's individual risk for a disease. Method according to one of claims 9 and 10, characterized in that the recorded modalities are based on retinal imaging methods CFP and OCT, that the recorded modalities are registered with respect to a common coordinate system that extracts size and number of drusen as well as hyperpigmentation from the CFP-based modality be extracted from the OCT-based modality size, number and volume of drusen and hyperpigmentation that the extracted signal-based features of the two modalities are classified by statistical methods and before or after combined using known context information and from it the classification result as severity of Disease on AMD is determined. Method according to claim 14, characterized in that additionally a third modality based on the retinal imaging method FAF is registered and registered with respect to the common coordinate system, that in this FAF-based modality hyperpigmentation is extracted, the extracted, signal-based features of the three modalities are classified by means of statistical methods and combined beforehand or thereafter on the basis of known context information, and from this the classification result is determined as the severity of the disease in AMD. A method according to claim 11, characterized in that the recorded modalities are based on retinal imaging methods CFP and OCT, that the recorded modalities are registered with respect to a common coordinate system, that extracts characteristics for describing hyperpigmentation and / or drusen from the CFP-based modality statistical methods are classified, that the classified features of this first modality are used to further classify particularly interesting sections using the second recorded and registered modality by additionally extracting features for describing hyperpigmentation and / or drusen from the OCT-based modality and are classified by statistical methods, and from this the classification result in the form of a probability map for the disease is determined on AMD. A method according to claim 16, characterized in that extracted from the CFP-based modality hyperpigmentation and classified by means of statistical methods in terms of size and number that determined in the particularly interesting areas of the OCT-based modality of the distance d of hyperpigmentation to the retinal pigment epithelium and in Healthy tissue is evaluated and the classification result is determined from this in the form of a probability map for the disease in AMD. Method for the automated or semi-automated detection and classification of DR, based on modalities of the respective eye with at least partially identical areas, which were recorded at different times and / or under different conditions by means of different recording methods using different ophthalmological examination apparatuses, characterized in that all recorded Modalities are registered with respect to a common coordinate system that extracted from all recorded and registered modalities signal-based features that are suitable to describe the biomarkers sought, that the extracted signal-based characteristics of the recorded modalities by statistical methods classified and before or after whose classification is combined and from this the classification result in the form of a probability map for the disease is determined on DR. Method for automated or semi-automated detection and classification of MA, based on modalities of the respective eye with at least partially identical areas, which were recorded at different times and / or under different conditions by means of different recording methods using different ophthalmological examination apparatus, characterized in that all recorded Modalities are registered with respect to a common coordinate system that extracted from all recorded and registered modalities signal-based characteristics that are suitable to describe the biomarker sought, that the extracted signal-based characteristics of the recorded modalities are classified by statistical methods and combined before or after their classification and from this the classification result in the form of a probability map for the disease is determined at MA. A method according to claims 18 and 19, characterized in that the claims 2 to 13 are mutatis mutandis applicable to the method according to claims 18 and 19.