JP5740403B2 - System and method for detecting retinal abnormalities - Google Patents

Description

(Citation of related application)
This application claims the benefit of US Provisional Application No. 61 / 279,156 (filed Oct. 15, 2009, entitled “Systems and Methods for Retinal Image Analysis”). This application is a continuation-in-part of US patent application Ser. No. 12 / 355,567 (filed Jan. 16, 2009, entitled “Systems and Methods for Analyzing Image Data Using Adaptive Neighborhooding”), Application No. 61 / 021,513 (filed on Jan. 16, 2008, entitled “An Energy Functional Framework for Simulative Learning, Smoothing, Segmentation, and Grouping in United States”). 011,456 (filed on January 16, 2008, name "Image-based Me""Hods for Measuring Global Nuclear Patterns as Epigenetic Markers of Cell Differentiation") To do. Each of these applications is hereby incorporated by reference in its entirety.

(Announcement of federal funding research and development)
The research and development described herein, in whole or in part, is a grant number from the National Institutes of Health (NIH / NIBIB). RO1-EB006161-01A2, NINDS / NIH grant from National Neurological and Stroke Research Institute. Funded by 5-R01-NS34189-10. In the present invention, the US government has certain rights.

  Diabetic retinopathy (DR) is one of several systemic diseases that damage by attacking the retina and causing lesions. The disease often has no early signs, is relatively advanced, and the lesions can cause blindness. Diabetes currently affects over 194 million individuals worldwide, is on the rise, and is expected to acquire 333 million patients by 2025. DR affects up to 80% of patients with diabetes for more than 10 years. DR is currently the leading cause of blindness in developed countries, is on the rise, and is expected to affect 275 million individuals worldwide.

  The severity of blindness from diabetes can be reduced if patients are timely and rigorously evaluated. The maintenance of vision requires lifelong care, a periodic and accurate diagnosis of DR severity. Each patient should have DR severity assessed at least once a year and requires a global minimum throughput of 24 eyes per second.

  There are a number of challenges that hinder meeting this need. First, many diabetics, including 25% of the US diabetic population, lack health awareness. Secondly, the majority of patients, especially within developing countries, do not reside near clinical facilities. Third, DR diagnosis requires special training, and many clinics do not have such trained clinicians. Finally, even if care is available and within reach of a conscious patient, ability is limited, diagnostic procedures are inconvenient, and subject to personal discrepancies and errors.

  The methods and systems described herein were inspired by these and other issues of retinal imaging.

  Described herein are systems, methods, and non-transitory computer readable media for detecting retinal abnormalities. These methods are preferably implemented by a computer or other suitably configured processing device.

  In a first aspect, a computer receives an image of the retina and smoothes and segments the image. Lubrication and segmentation can be interdependent. The computer identifies at least one retinal feature in the smoothed and segmented image and analyzes at least one retinal feature identified in the smoothed and segmented image to identify retinal abnormalities . In some embodiments, analyzing the identified at least one retinal feature includes analyzing the texture of the image. In some embodiments, analyzing the identified at least one blood vessel determines at least one route associated with the at least one blood vessel in the image and the angle of the orientation of the route at a location along the route. And determining a variation in at least one of the diameters of at least one blood vessel at a location along the path. If the variation exceeds the threshold, the computer identifies a retinal abnormality at the location. The abnormality can be, for example, an abnormality in the venous inner diameter, a new blood vessel distribution, a non-vascular abnormality such as fluffy vitiligo, or any other abnormality.

  In some embodiments, segmenting and smoothing includes determining edge field strengths at multiple locations in the image, and at least one path is determined based at least on the edge field strengths. Determining at least one path may include determining at least one centerline associated with at least one blood vessel in the image. Determining the orientation angle variation may include determining a change in the orientation angle between successive locations.

  In some embodiments, the computer filters multiple variations. The filter may be a moving window average. In some embodiments, the variation exceeds the threshold when at least one of the filtered variations exceeds the threshold. In some embodiments, if the variation at a location exceeds a threshold, the computer determines whether a vascular bifurcation is present at the location, and if the vascular bifurcation is present at the location, the computer The variation is filtered out by the computer.

In some embodiments, smoothing and segmenting may define shapes and orientations that define neighborhoods associated with multiple locations in the image (eg, to reduce the value of the energy function associated with the error metric). Adaptively adjusting at least one of them. In some embodiments, smoothing and segmenting includes reducing the values of the first and second energy functions, the first energy function using a first error metric, The second energy function uses a second error metric that is different from the first error metric. In such embodiments, the computer may combine information about the first and second sets of smoothing and partitioning parameters obtained by reducing the values of the first and second energy functions, respectively.
This specification also provides the following items, for example.
(Item 1)
A computerized method for detecting retinal abnormalities,
Receiving a retina image by a computer;
Smoothing and segmenting the image by the computer, wherein the smoothing and segmentation is interdependent;
Identifying at least one retinal feature in the smoothed and segmented image by the computer;
Analyzing, by the computer, at least one retinal feature identified in the smoothed and segmented image to identify retinal abnormalities;
Including a method.
(Item 2)
The method of claim 1, wherein the at least one retinal feature comprises a blood vessel.
(Item 3)
Analyzing the identified at least one blood vessel comprises:
Determining at least one path associated with the at least one blood vessel in the image;
Determining an angle of orientation of the path at a location along the path and a variation in at least one of the diameters of the at least one blood vessel at the location along the path;
Identifying a retinal abnormality at the location when the variation exceeds a threshold;
The method according to item 2, comprising:
(Item 4)
The segmenting and smoothing includes determining edge field strengths at a plurality of locations in the image, wherein the at least one path is determined based at least in part on the edge field strengths. 3. The method according to 3.
(Item 5)
4. The method of item 3, wherein determining the at least one path includes determining at least one centerline associated with the at least one blood vessel in the image.
(Item 6)
4. The method of item 3, wherein determining the orientation angle variation comprises determining a change in orientation angle between a plurality of consecutive locations.
(Item 7)
4. The method of item 3, further comprising filtering a plurality of variations by the computer.
(Item 8)
The method of claim 7, wherein filtering the plurality of variations comprises applying a moving window average.
(Item 9)
8. The method of item 7, wherein the variation exceeds the threshold when at least one of a plurality of variations that are filtered exceeds the threshold.
(Item 10)
The method according to item 1, wherein the abnormality is an abnormality of a venous inner diameter.
(Item 11)
The method according to item 1, wherein the abnormality is a new blood vessel distribution.
(Item 12)
Determining if a blood vessel bifurcation is present at the location by the computer if the variation at a location exceeds a threshold;
Removing a variation by a filter by the computer when a blood vessel bifurcation is present at the location;
The method according to item 3, further comprising:
(Item 13)
In item 1, the smoothing and segmentation includes adaptively adjusting, by the computer, at least one of a shape and orientation that defines a neighborhood associated with a plurality of locations in the image. The method described.
(Item 14)
14. The method of item 13, wherein adaptively adjusting at least one of the neighboring shape and orientation includes reducing an energy function value associated with an error metric.
(Item 15)
The smoothing and segmentation is:
Reducing the values of the first and second energy functions, wherein the first energy function uses a first error metric and the second energy function is the first error measurement. Using a second error metric different from the reference;
Combining information on the first and second sets of smoothing and partitioning parameters obtained by reducing the values of the first and second energy functions, respectively.
The method according to item 1, comprising:
(Item 16)
The method of item 1, wherein analyzing the identified at least one retinal feature comprises analyzing a texture of the image.
(Item 17)
A system for detecting retinal abnormalities,
A processor, the processor comprising:
Receiving an image of the retina;
Smoothing and segmenting the image, the smoothing and segmentation being interdependent;
Identifying at least one retinal feature in the smoothed and segmented image;
Analyzing at least one retinal feature identified in the smoothed and segmented image to identify retinal abnormalities;
Configured to do the system.
(Item 18)
The system of claim 17, wherein the at least one retinal feature comprises a blood vessel.
(Item 19)
Analyzing the identified at least one blood vessel comprises:
Determining at least one path associated with the at least one blood vessel in the image;
Determining an angle of orientation of the path at a location along the path and a variation in at least one of the diameters of the at least one blood vessel at the location along the path;
Identifying a retinal abnormality at the location when the variation exceeds a threshold;
19. The system according to item 18, comprising:
(Item 20)
Item 18. The segmenting and smoothing includes determining edge field strength at a plurality of locations in the image, and the at least one path is determined based on at least the edge field strength. System.
(Item 21)
The system of claim 19, wherein determining the at least one path comprises determining at least one centerline associated with the at least one blood vessel in the image.
(Item 22)
20. The system of item 19, wherein determining the orientation angle variation comprises determining a change in orientation angle between a plurality of consecutive locations.
(Item 23)
The system of item 19, wherein the processor is further configured to filter a plurality of variations.
(Item 24)
24. The system of item 23, wherein filtering the plurality of variations includes applying a moving window average.
(Item 25)
24. The system of item 23, wherein the variation exceeds the threshold when at least one of a plurality of filtered variations exceeds the threshold.
(Item 26)
18. The system according to item 17, wherein the abnormality is an abnormality in a venous inner diameter.
(Item 27)
The system according to item 17, wherein the abnormality is a new blood vessel distribution.
(Item 28)
The processor is
Determining whether a vascular bifurcation is present at the location when the variation at a location exceeds a threshold;
Item 20. The system of item 19, further configured to filter the variation by a computer when a vascular bifurcation is present at the location.
(Item 29)
The item of claim 17, wherein the smoothing and segmenting includes adaptively adjusting, by a computer, at least one of a shape and orientation that defines a neighborhood associated with multiple locations in the image. system.
(Item 30)
30. The system of item 29, wherein adaptively adjusting at least one of the neighboring shape and orientation includes reducing an energy function value associated with an error metric.
(Item 31)
The smoothing and segmentation is:
Reducing the values of the first and second energy functions, wherein the first energy function uses a first error metric and the second energy function is the first error measurement. Using a second error metric different from the reference;
Combining information on the first and second sets of smoothing and partitioning parameters obtained by reducing the values of the first and second energy functions, respectively.
The system according to item 17, comprising:
(Item 32)
18. The system of item 17, wherein analyzing the identified at least one retinal feature comprises analyzing the texture of the image.
(Item 33)
A non-transitory computer readable medium storing instructions executable by a computer, wherein the instructions are executed,
Receiving an image of the retina;
Smoothing and segmenting the image, wherein the smoothing and segmentation is interdependent;
Identifying at least one retinal feature in the smoothed and segmented image;
Analyzing at least one retinal feature identified in the smoothed and segmented image to identify retinal abnormalities;
Causing the computer to execute a method comprising:
A non-transitory computer readable medium.
(Item 34)
34. The non-transitory computer readable medium of item 33, wherein the at least one retinal feature comprises a blood vessel.

  This application is also incorporated by reference herein in its entirety, U.S. Patent Application No. 12 / 321,360, “Image-based Methods for Measuring Global Patterns as Epigenetic Markers of January 16, 2009, which is incorporated herein by reference in its entirety. It also relates to “Cell Differentiation”.

This patent or application file contains at least one drawing executed in color. Copies of this patent or patent application publication with colored drawings will be provided to the Patent Office upon request and payment of the necessary fee. The above and other features of the invention, its nature and various advantages will be apparent upon consideration of the following detailed description, taken in conjunction with the accompanying drawings.
FIG. 1 is a schematic diagram of a system for image analysis, according to an illustrative embodiment of the invention. FIG. 2 is a diagram of an illustrative process for generating smoothed data, partitioning, and attribute estimates according to an embodiment of the present invention. FIG. 3 illustrates an information extraction method as applied to an example image data according to an embodiment of the present invention. 4A-4C depict three different approaches to neighborhood fitting, according to an illustrative embodiment of the invention. 4A-4C depict three different approaches to neighborhood fitting, according to an illustrative embodiment of the invention. 4A-4C depict three different approaches to neighborhood fitting, according to an illustrative embodiment of the invention. FIG. 5 demonstrates the improvement achievable by the present invention in the detection of image elements in the presence of noise. FIG. 6 illustrates a method for generating smoothed data, partitioning, and attribute estimates by minimizing the energy function, according to an illustrative embodiment of the invention. FIG. 7 is a flow diagram of a technique for detecting retinal abnormalities, according to an embodiment. FIG. 8 is a flow diagram of an anomaly detection technique 800, according to an embodiment. 9A-9C illustrate edge detection using different energy function formulations, according to embodiments. 10A-10C illustrate retinal image processing according to an embodiment. 11A-11C illustrate a vessel identification technique according to an embodiment. 12A-12D illustrate a vascular path detection technique according to an embodiment. 13A-13C illustrate the operation and intermediate results of the blood vessel identification technique, according to an embodiment. FIGS. 14A-14C illustrate an embodiment for determining variations in vessel path slope. 15A-15C illustrate an embodiment for determining the variation in the angle of blood vessel orientation. 16A-16C illustrate an embodiment for determining the variation in vessel radius. FIGS. 17A-17F illustrate an analysis of a vascular bifurcation and each of its two branches for retinal abnormalities, according to an embodiment.

  In order to provide a general understanding of the present invention, certain illustrative embodiments are described, including systems and methods for extracting information from image data. However, the systems and methods described herein may be adapted and modified for other suitable applications, and such other additions and modifications may not depart from the scope of the specification. It will be understood by those skilled in the art.

(A. Image analysis system)
FIG. 1 is a schematic diagram of an image analysis system, according to an illustrative embodiment of the invention. The system 100 includes an image capture device 110, an image database 120, an information extraction processor 130, a display 140, a results database 150, and a classification processor 160.

  The system 100 includes at least one image capture device 110 for capturing an image of a scene. Exemplary image capture devices 110 include visible light cameras and video recorders, PET, SPECT, MRI, X-ray, CT scanners, and other medical imaging devices, bright field, phase contrast, atomic force, and scanning electron microscopes. Including satellite radar, thermographic cameras, seismometers, and sonar and electromagnetic wave detectors. Each of the image capture devices 110 may generate an analog or digital image. An image captured by a single image capture device 110 may be a scalar, vector, or matrix value and may vary as a function of time. The imaged scene can include any physical object of interest, collection of physical objects, or physical phenomenon for which at least one property measurement can be obtained by the image capture device. For example, the fetal environment of a fetus is a scene that can be measured with an ultrasound image capture device. In another example, atmospheric moisture location and movement is a scene that can be measured with a satellite radar image capture device.

  The image database 120 is used to store images captured by the image capture device 110 as a set of image data. The image database 120 may comprise any suitable combination of magnetic, optical, and / or semiconductor memory, and may include, for example, RAM, ROM, optical disks such as flash drives, compact disks, and / or hard disks or drives. Those skilled in the art will be familiar with several examples for the image database 120 in the system 100, along with exemplary embodiments including a database server designed to communicate with the processor 130, a local storage unit, or a removable computer readable medium. Those preferred implementations will be recognized.

  The information extraction processor 130 and the database 120 may be integrated into the same physical unit, or a USB port, serial port cable, coaxial cable, Ethernet type cable, telephone line, radio frequency transceiver, or other similar Can be housed as separate devices that are connected to each other by a wireless or wired medium, or a communication medium such as a combination of the foregoing. The information extraction processor 130 queries the database 120 to obtain a non-empty portion of the set of image data. The information extraction processor 130 also performs the information extraction process described below. Exemplary embodiments of information extraction processor 130 include software programmable processors found in general purpose computers, as well as specialized processors that can be incorporated into larger devices. The information extraction processor 130 performs the methods described herein by executing instructions stored on a computer readable medium, and those of ordinary skill in the art will recognize, without limitation, such media as solid, magnetic, It will be appreciated that includes holographic, magneto-optic, and optical memory units. Additional embodiments of the information extraction processor 130 and the remaining elements of FIG. 1 are discussed below.

  Upon completion of the information extraction method or simultaneously with the method, the information extraction processor 130 outputs a set of processing data. A display 140 visually presents the processing data to the user. Exemplary embodiments include a computer monitor or other electronic screen, a physical printout generated by an electronic printer in communication with the information extraction processor 130, or a three-dimensional projection or model. The results database 150 is a data storage device where processed data is stored for further analysis. Exemplary embodiments include the structures and devices described for the image database 120, as well as others known to those skilled in the art. The classification processor 160 optionally extracts the processing data from the database 150 to classify the processing data, i.e. identify the meaning and content of the elements in the captured scene, and the architecture and information described for the information extraction processor 130. A data processing device that may be embodied by a device.

  Although system components 110-160 are depicted in FIG. 1 as separate units, one of ordinary skill in the art would practically combine two or more of these units into an integrated device that performs the same overall function. You will immediately recognize that they can be combined. For example, a single physical camera has both visible and infrared imaging capabilities and can therefore represent two image capture devices. A single image processing device may also include a database 120 for image data that can be transmitted directly to the processor 130. Similarly, database 120 and processor 130 can be configured in a single general purpose computer so that processors 130 and 160 can be configured. Many combinations of system components in hardware are possible and still remain within the claimed system. System components 110-160 can be coupled using a communication protocol over a physical connection or can be coupled using a wireless communication protocol. In one exemplary embodiment, image data is transmitted from an image capture device to a data processing facility where it is stored and processed wirelessly or via an electronic network. In another exemplary embodiment, the system of FIG. 1 is deployed in an automobile or fleet that can use processing data to make decisions regarding the behavior of the automobile or fleet.

  Returning to FIG. 1, those skilled in the art will recognize that many different embodiments of the system components 110-160 are possible, as are many different settings for the system as a whole. In one embodiment, the system of FIG. 1 is present in a laboratory or medical research environment, for example, image data from perfusion imaging, fMRI, multispectral or hyperspectral imaging, bright field microscopy, or phase contrast microscopy. Used to improve patient diagnosis. In another embodiment, the system of FIG. 1 resides within a monitoring station and is used to assess conditions in a particular geographic region by combining data from at least one imaging device. These devices may include satellite radar, aerial photography or thermography, seismometers, sonar or electromagnetic wave detectors. In another embodiment, the system of FIG. 1 can be configured for any general purpose computer to meet the hardware requirements and extract information from the image data resulting from the user's particular application.

(B. Information extraction)
The information extraction processor 130 uses adaptively adjusted neighborhoods to smooth out the image data to improve the representation of the scene and distinguish these elements by determining edges between elements in the scene. As described above, the image data is segmented and the attribute of the element in the scene is estimated to extract information on the element in the captured scene. FIG. 2 depicts one illustrative embodiment of an information extraction method performed by the information extraction processor 130. The input to the information extraction processor 130 includes image data 210 comprising images 1, 2,..., N, prior knowledge 220 of image data characteristics, and prior knowledge 230 of attributes within the captured scene. Image data characteristics prior knowledge 220 includes noise intensity and distribution information, captured scene model, environmental factors, and imaging device properties. The prior knowledge 230 of the attributes in the captured scene includes the location in the scene with known attributes, the knowledge of the presence or absence of elements in the captured scene, the real-world experience of the captured scene, or the content of the captured scene Including a probabilistic evaluation of The smoothing 240, segmentation 250, and attribute estimation 260 processes are interdependent in the sense that the processor considers the results of each of these processes when performing other processes. Neighbor adaptive adjustments 265 are discussed in more detail below. In addition, the processes are performed simultaneously or substantially simultaneously. At the end of these processes, the information extraction processor 130 includes a set of smoothed data 270, a set of partitions 280 that divide the captured scene into consistent elements, and a set of estimated attributes 290 that exist within the scene. Output a set of data. Each of processes 240, 250, and 260 are discussed in more detail below.

  The smoothing process 240 generates a set of smoothed data 270 from the image data. Smoothed data 270 represents the most accurate estimate of the true characteristics of the imaged scene. Images are often corrupted by noise and by distortion from imaging equipment, so that image data is by no means a perfect representation of a true scene. When performing the smoothing 240, the processor 130 may, among other factors, characterize noise that occurs at all points between the image data, the physical model of the captured scene, and the captured scene and the database 120. And the results of the segmentation process 250 and the attribute estimation process 260 are taken into account.

  The segmentation process 250 demarcates distinct elements in the captured scene by drawing edges that distinguish one element from another. For example, the segmentation process may distinguish between an object and its background, several objects that overlap within the captured scene, or an area within the captured scene that exhibits different attributes. The segmentation process results in a set of edges that define the segment 280. These edges may be scalar, vector, or matrix values, or may represent other data types. When performing segmentation 250, the information extraction processor 130, among other factors, includes the image data 210, the physical model of the captured scene, and all points between the captured scene and the image database 120. The characteristics of the resulting noise and the results of the smoothing process 240 and the attribute estimation process 260 are taken into account.

  The attribute estimation process 260 identifies the nature of the elements in the captured scene. An attribute is any property of an object whose image data contains some information. The set of available attributes depends on the imaging modality represented in the image data. For example, a thermographic camera generates images from infrared radiation, and these images contain information about the temperature of the object in the captured scene. Additional examples of attributes include texture, radioactivity, moisture content, color, and material composition, among others. For example, the surface of a pineapple may be identified by the processor as having a texture (attribute) that is rough (attribute value). In one embodiment, the attribute of interest is the parameter on which the parameterized model group representing the image data is based. In another embodiment, the attribute of interest is the parametric model itself. When performing attribute measurements, the information extraction processor 130 occurs at all points between the image data 210, the physical model of the captured scene, and the captured scene and the image database 120, among other factors. The noise characteristics and the results of the smoothing process 240 and the segmentation process 250 are taken into account.

  If more than one image is represented in the image data, the information extraction processor 130 may also determine the relative amount of state included in each image for a particular attribute. In estimating this attribute, the information extraction processor 130 may then utilize each image according to its information content regarding the attribute. For example, multispectral imaging returns multiple images, each generated by a camera operating in a specific wavelength band. Different attributes may be better represented at one frequency than at another frequency. For example, satellites use a wavelength range of 450-520 nm to image the deep sea, but use a wavelength range of 1550-1750 nm to image terrestrial plants. In addition, the information extraction processor 130 may use image data statistics to identify images that are particularly relevant to the attribute of interest. For example, one or more different weighted combinations of image data can be identified as having more information content compared to other combinations of any particular attribute. The present invention allows the attribute estimation process to selectively use data from different images, interdependently with the smoothing and segmentation processes.

  In addition, the information extraction processor 130 may selectively utilize data in different ways at different locations within the captured scene for any of the smoothing, segmentation, and attribute estimation processes. For example, if each image in the data set corresponds to a personal picture taken at a different angle, only a portion of these images contain information about the personal facial features. Thus, these images are selectively used by the information extraction processor 130 to extract information about the facial region in the captured scene. The information extraction method presented herein selectively uses image data to decompose elements in captured scenes at different locations, interdependently with the smoothing, segmentation, and attribute estimation processes. It is possible.

  It is important to note that the number of attributes of interest and the number of available images can be independent. For example, several attributes can be estimated within a single image, or multiple images can be combined to estimate a single attribute.

(C. Application: Aerial imaging)
Examples are useful for more clearly explaining the information extraction method. FIG. 3 depicts three images 310, 320, 330 for processing by the image analysis system of FIG. 1 according to the methodology of FIG. These three images represent the same scene, image 310 is an aerial photograph, image 320 is a cloud cover satellite radar image, and image 330 is an aerial thermogram (the thermal radiation emitted by elements in the scene). Measure). These three images represent three different imaging modalities, i.e. how to capture information about the scene, and therefore it is possible to detect more kinds of phenomena than a single modality. In the example, image 320 represents the presence of a cloud. The image 330 reveals areas of high thermal radiation that can show the characteristics of the dwellings in the scene. Since the dwelling is hidden by the foliage and the amount of clouds, it is difficult to detect in the image 310. One of the strengths of this information extraction method is the ability to combine multiple modalities to best reveal the underlying elements in the scene. Different modality combinations are discussed in more detail below.

  The information extraction processor 130 obtains these three images from the database 120 and then outputs a set of smoothed data 270, a set of segments 280 in the scene, and an estimate 290 of attributes in the scene. If the goal is to obtain the most accurate representation of the structure on the ground, the set of smoothed data 270 can be as depicted in the illustration 340. To generate the smoothed data 270, the information extraction processor 130 removes the obscuring leaves and generates information about the cloud cover from the image 320 and from the image 330 to generate a clearer illustration of the structure on the ground. Using the potential existence of a residence. By segmenting the image data, the set of segmented data 280 depicted in the illustration 350 can be provided in which residence and road contours are identified. Depending on the user's interest, the information extraction processor 130 may also smooth and segment additional elements in the scene, such as leaves in the image 310 or clouds in the image 320. Information extraction processor 130 may also identify materials used to build each of the residences and roads via attribute estimation process 260, as depicted in illustration 360. As discussed in more detail below, the attribute estimation process 260 uses different neighborhoods via the adaptive neighborhood adjustment process 265 to identify “asphalt” and “aluminum” surfaces in the captured scene. Can do. Once processing data is available, a classification processor 160 may be used to classify elements within the captured scene, for example, the illustration 370 may be used for smoothing, segmentation, and attribute estimation processes 240-260. All results are used to identify “roads” and “residences” in the captured scene. Additional embodiments of the smoothing, segmentation, and attribute estimation processes 240-260 are described below.

(D. Adaptive neighborhood)
When generating a set of smoothed data 270 from a noisy image, or when classifying segments according to their attribute values, distinguish which locations in the captured scene correspond to edges and which do not It is desirable to be able to do it. Once an edge has been identified, the information extraction processor 130 then treats both sides of the edge and locations on the edge itself separately to improve smoothing and classification performance. It is then desirable to use local information selectively during the smoothing, segmentation, and attribute estimation processes. Thus, in one embodiment, the adaptive neighborhood adjustment process 265 makes a determination at each location based on the neighborhood of the surrounding locations. One embodiment of the present invention associates a neighborhood with each specific location in the scene that was imaged. Each neighborhood includes a number of other locations near a particular location. The information extraction processor 130 can then use the neighborhood of each location to focus more on the smoothing, segmentation, and attribute estimation processes 240-260 to better extract information about the location. In its simplest form, the neighborhood associated with each location can have a fixed size, shape, and orientation, eg, a circle with a fixed radius. However, there are several drawbacks to using inflexible neighborhood sizes and shapes. For example, if the location is located above an edge, a smoothing and attribute estimation process that relies on a fixed neighborhood uses information from the scene elements on both sides of the edge, leading to incorrect results. One improvement is to adjust the neighborhood size of each location based on local information. Further improvements include adjusting the size, shape, and orientation of the neighborhood in the adaptive neighborhood adjustment process 265 to better match the local characteristics. These embodiments are described in more detail below.

  In one embodiment, the information extraction processor 130 performs the information extraction method while adjusting the size, shape, and orientation characteristics in the vicinity of the surrounding location in the captured scene. Specifically, processor 130 adapts the neighborhood characteristics associated with each location in an interdependent manner with the smoothing, segmentation, and attribute estimation processes 240-260. In another embodiment, information extraction processor 130 utilizes a separate, independently adapted neighborhood for each attribute analyzed by information extraction processor 130.

  The benefits of utilizing adaptive neighborhood sizes, shapes, and orientations can be seen in FIGS. 4A-4C and FIG. 4A-4C illustrate three different neighborhood-based approaches. FIGS. 4A-4C for each example depict edges and some illustrative neighborhoods 410-430 at corresponding locations. The first example illustrates an approach where the neighborhood 410 associated with each location in the captured scene is the same. In FIG. 4A, all neighbors 410 are circles centered on the location with a fixed radius. In FIG. 4B, all neighbors 420 are circular, but with a radius that is allowed to change to avoid the neighbors 420 overlapping the edges. In FIG. 4C, in an exemplary embodiment of the invention, neighborhoods 430 are ellipses that are allowed to change their size, shape, and orientation to better match the characteristics of the local region.

  An exemplary implementation of an information extraction method that includes an averaging step within the smoothing process 240 to reduce noise present in the raw image data to demonstrate improvements that can provide such a fit. Consider the form. The averaging step generates a smoothed data value at each location (with associated neighborhood) by replacing the image data value at that location with the average of the image data values at each location that falls within the relevant neighborhood.

  Referring to FIGS. 4A-4C, this averaging is performed on the neighborhoods 410-430 shown. In FIG. 4A, averaging occurs across edge values and across partitions, blurring the distinction between partitions. The mathematical formulation according to neighborhood 410 is given by:

In the equation, g is image data, u is smoothed data, α and β are adjustable parameters, and integration is obtained over all locations X in the region R.

  In FIG. 4B, the location near the edge has an associated neighborhood 420 that is necessarily small so as to avoid overlapping the edge, and thus more susceptible to noise. The mathematical formulation according to the neighborhood 420 is given by:

In the equation, g is image data, u is smoothed data, v is an edge value, and α, β, and ρ are adjustable parameters. A method related to that illustrated in FIG. 4B is incorporated herein by reference in its entirety, “Model-based variational smoothing and segmentation for tenor imaging in the brain,” Neuroinformatics. 4 Used in 2006 by Desai et al. To analyze diffusion tensor imaging data of the human brain.

  In FIG. 4C, where the size, shape, and orientation are allowed to change, it is possible to selectively identify the neighborhood 430 that each location averages, and the average across the edges while improving noise reduction performance. Is prevented. The mathematical formulation according to the neighborhood 430 is given by:

g is image data, u is smoothed data, V is a symmetric positive definite 2 × 2 matrix representing the neighborhood, w weights the data fidelity term, F and G is a function, and α, β, ρ, and ρ _w are adjustable parameters. Information extraction processor 130 may also use information resulting from smoothing and attribute estimation processes 150-160 to adjust the size, shape, and orientation of the neighborhood.

(E. Example: Noisy image)
FIG. 5 demonstrates the performance enhancement achievable in a compatible neighborhood, such as the third neighborhood 430 as illustrated in FIG. 4C. The original scene 510 of white squares on a black background is corrupted by noise during the imaging process, resulting in a noisy image 520. In the noisy image 520, it is difficult for human eyes to distinguish the square from the background. The smoothing method of the first embodiment 410 is applied to generate a first smoothed image 530. This method does not produce a set of edges and the resulting smoothed data blurs the white rectangular border. The smoothing method of the second embodiment 420 is applied to generate a second smoothed image 540 and a set of scalar edges 550. To the human eye, the second smoothed image 540 is as unclear as the first smoothed image 530 and the scalar edge 550 is diffuse.

  The smoothing method of the third example 430, which is an exemplary embodiment of the present invention, results in a third smoothed image 560 and matrix value edges associated with each pixel. Since the edges are matrix values, it is not possible to represent them in the same way as the scalar edge 550. One meaningful scalar value associated with the edge matrix is its trace, and thus the third example 410 can be associated with a set of traces 570 of matrix value edges. Additional embodiments of the invention include examining eigenvalues and eigenvector functions of matrix value edges for boundary information. The third smoothed image 560 is much sharper than the first or second smoothed images 530 and 540 and the boundary is defined by a set of traces 570 of matrix value edges rather than a set of scalar edges 550. Much more clearly drawn. The ability of the third embodiment to adapt neighborhood size, shape, and orientation, interdependently with the smoothing process, allows for improved information extraction performance as demonstrated in FIG.

(F. Energy function approach)
One particular embodiment of the information extraction method is illustrated in FIG. As discussed above, a neighborhood matching process can be performed for each attribute of interest. At each location, a different neighborhood can be determined for each attribute, which allows identification of attribute values and attribute edge values for each attribute. FIG. 6 shows as input image data, attribute background knowledge, segment (and associated edges) 610 in image data, smoothed image data 620, segment (and associated edges) 630 in attribute estimates, and attribute estimates. Describes an iterative process that captures the value 640. To begin applying the iterative process of FIG. 6, initial values for inputs 610, 620, 630, and 640 can be specified by the user or automatically selected by processor 130. The adaptation process seeks to minimize the energy function, including penalties for undesirable implementations. Some example penalties that can be included in the energy function are depicted in the energy function element block 650. These include penalties for discrepancies between image data and smoothed data, penalties for specifying excessive edges in the data, penalties for specifying excessive edges in attributes, discontinuities or non-smooth edges in the data Includes penalties for gender, penalties for edge discontinuities or non-smoothness in attributes, discontinuities or abrupt changes in smoothed data, and discontinuities or abrupt changes in attribute estimates. The input to the energy function can be used to calculate the energy value, and then the inputs 610, 620, 630, and 640 are adaptively adjusted to achieve a lower energy value.

  In one implementation of this embodiment, the energy value determination is calculated according to the following equation:

In the equation, e ₁ , e ₂ , e ₃ , e ₄ , e ₅ are error terms as described below. The values for smoothed data u, segment edge υ _u , attribute θ, and attribute segment edge υ _θ are selected for each (x, y) coordinate to minimize the expression contained within the square brackets. And integrated over the entire plane. This equation depends on the image data g, the data function T (θ) with the attribute θ, the parameters λ _u , α _u , ρ _u , λ _θ , α _θ , ρ _θ ,

It is.
The term e ₁ is a penalty for discrepancies between the image data and the smoothed data, the term e ₂ is a penalty for discontinuities in the smoothed data, and the term e ₃ is the presence of edges and the edges Term e ₄ contains the penalty for the discontinuity in the attribute estimate, and term e ₅ contains the penalty for the presence of the attribute edge and the discontinuity of the attribute edge. One skilled in the art will recognize that there are many additional penalties that can be included in the energy function, and the selection of the appropriate penalty will depend on the application at hand. Equivalently, this problem can be expressed as a reward function maximization, where different reward terms correspond to different desirable performance requirements for the information extraction method. There are many standard numerical techniques that can be readily applied to this particular mathematical formulation by those skilled in the art, such as, for example, gradient descent. These techniques can be implemented in any of the embodiments described herein.

  In another embodiment, the calculation of the minimum energy value is performed according to the following equation:

In the equation, e ₁ , e ₂ , e ₃ , e ₄ , e ₅ are error terms as described below. Smoothed data u, segment edge w, measurement model parameter edge field υ _m , process model parameter edge field υ _u , measurement model parameter edge field υ _m , process parameter correlation edge field υ _c , process model parameter Values for θ _u , and measurement model parameter θ _{m are selected} for each (x ₁ , x ₂ ,..., x _N , t) coordinates to minimize the expression contained within the square brackets, It is integrated over the entire N-dimensional image data space augmented with a one-dimensional time variable. The error term is given by
e ₁ = βM (u, g, w, θ _m ),
e ₂ = α _m L _m (θ _m , υ _m ),
e ₃ = α _u C _u (u, υ _u , θ _u ),
_{e 4 = α C L C (} υ C, θ u), and _{e 5 = π (u, w} , υ m, υ n, υ C, θ u, θ m)
Where M is a function that measures data fidelity, L _m estimates measurement model parameters, C _u measures process model spatial correlation, L _c estimates process model parameters, π represents a prior distribution of unknown variables, and β, α _m , α _u , α _c are parameters that allow the process to place different emphasis on the terms e ₁ , e ₂ , e ₃ , e _4. is there.

(G. Retina imaging)
As explained above, the need for accurate, rapid and convenient diagnosis of retinal abnormalities is serious. To address this need, the present specification can analyze retinal images to help clinicians with moderate skills effectively evaluate retinal disease in a large number of patients worldwide. Thus, automatic classification and decision support methods and systems that can provide a clinically validated determination of the severity of retinopathy are described. These systems and methods have been developed to address several specific challenges of detecting retinal abnormalities, including:
1. An abnormality in a blood vessel is associated with the geometry of that particular blood vessel elsewhere along the blood vessel.
2. An unusual feature is a non-linear function of the vessel geometry.
3. Anomalies are subtle and can be easily camouflaged by noise and imaging artifacts.
4). Retinal images typically have low signal-to-noise (SNR) characteristics that vary spatially on blood vessels and the retinal background.

  Retinal abnormalities detected by the techniques disclosed herein include venous caliber abnormalities (VCAB), new vascular distribution within the optic disc (NVD), new vascular distribution elsewhere (NVE), intraretinal subtleties Includes vascular abnormalities (IRMA), microaneurysms, and anterior retina and vitreous hemorrhage. Each of these abnormalities is categorized in the diagnostic guidelines for early treatment studies of diabetic retinopathy used by practitioners. For example, VCAB is a deformation of blood vessels in the retina that is both subtle and abrupt. In some patients, VCAB exhibits symptoms as a vascular lesion in which the size and orientation of the blood vessels change rapidly or slowly. To accurately detect and classify VCAB and other abnormalities, the clinician or automated decision making system is assisted by reliable determination of vessel boundaries, size, and orientation. The systems and techniques for detection of retinal abnormalities described herein address the particular challenges of this application by building on the smoothing and segmentation approach described above.

  We now discuss systems and techniques for detection of retinal abnormalities. For retinal imaging applications, the components of the image analysis system 100 of FIG. 1 can be selected to be suitable for retinal imaging. For example, the image capture device 110 of the system 100 can be an analog or digital fundus camera that can image the interior surface of the eye (eg, retina, optic disc, plaque, and posterior pole). The image capture device 110 may be used with one or more wavelengths of light and includes an optical filter for passing or blocking different wavelengths of interest (eg, passing fluorescent colors during angiography). May be included. The fundus camera can be used in mydriatic diagnostics where the patient's eyes are dilated with eye drops prior to imaging, or in non-mydriatic diagnostics. As described above with reference to FIG. 1, the image captured by the image capture device 110 may be a scalar, vector, or matrix value and may vary as a function of time. Retinal images can be stored in the image database 120 for real-time or subsequent processing, analysis, and review by the information extraction processor 130, the display 140, and the results database 150 (FIG. 1). These components can take the form of any of the embodiments described herein. Information extraction processor 130 is configured to perform any one or more of the retinal abnormality detection techniques described herein.

  FIG. 7 is a flow diagram of illustrative steps in a technique 700 for detecting retinal abnormalities, according to an embodiment. Technique 700 may be by any suitable processing device (eg, executing a non-transitory computer readable medium) included in (or in remote communication with) image analysis system 100, such as information extraction processor 130 of FIG. Can be implemented. For ease of illustration in the following description, technique 700 is described as being performed by imaging analysis system 100. In step 702, the system 100 receives an image of the retina. The retinal image can come from any suitable source, such as the image capture device 110 described above. The retinal image received at step 702 is preferably digital, but can be an analog image that is subsequently digitized.

  At step 704, the system 100 performs a smoothing and partitioning technique that may take the form of any of the smoothing, partitioning, and attribute estimation techniques described herein. As explained above, the smoothing and partitioning operations with these techniques are interdependent and can be performed substantially simultaneously (eg, with frequent alternating iterations). In some embodiments, the segmentation and smoothing technique performed in step 704 involves determining the edge field strength at the smoothed image and / or multiple locations within the image. The edge field can be a scalar value edge field (eg, as illustrated in equation 1), a vector value edge field (eg, as illustrated in equation 2), (eg, as illustrated in equation 3) It can be a matrix value edge field, or a combination thereof. In some embodiments, the smoothing and segmentation technique performed in step 704 comprises adaptively adjusting at least one of the shapes and orientations that define neighborhoods associated with multiple locations in the image. Including. In any of these embodiments, the system 100 may perform smoothing and segmentation techniques to reduce the value of the energy function associated with the error metric, as discussed above. A detailed description of some specific embodiments of step 704 follows.

  In some embodiments, each location in the image is associated with an elliptical neighborhood where the image is smoothed and the edge field is estimated, and the size, shape, and orientation of the neighborhood vary from location to location. The neighborhood of the location identified as an edge (eg, a blood vessel edge) is sized to limit edge “blurring” by smoothing across the edge (eg, as illustrated in FIG. 4C). Reduced to This formulation can be expressed as:

Where g is retinal image data output by the image capture device 110, u is smoothed data, V is a 2 × 2 symmetric edge matrix field, and X is smoothed and segmented. An image to be performed, and α, β, and ρ are adjustable parameters. As explained above, the first term can be interpreted as a smooth fidelity term that penalizes the slope of u by IV, so that smoothing occurs mainly based on neighboring pixels. . The second term is a data fidelity term that penalizes the deviation of the smoothed image data from the input image data. While the scalar term G (V) penalizes edge strength, F (V _X ) balances selection for smooth edges while recognizing “kinks” that may be a feature of VCAB. Although any numerical technique can be used to solve Equation 6 for any particular image and parameter value, one approach involves the use of Euler-Lagrange equations that form the basis of the solution. For Equation 6, the Euler-Lagrange equation is:

  In some embodiments, the smoothing and segmentation performed by system 100 at step 704 includes combining the results of two or more different energy function formulations. For example, the first energy function formulation takes the form of Equation 2 above, and the system 100 uses any of the computational techniques described herein to determine the values of the parameters u and V. Can do that. Next, the system 100 applies a second energy function formulation that uses an error metric different from the first energy function formulation as follows:

This includes one norm on the smoothing and data fidelity terms instead of the two norm used in Equation 2. In other embodiments, any energy function formulation of Equations 1 and 3-6 may be used with different error metrics.

  The different smoothing and segmentation results produced by the different energy function formulations of equations 2 and 9 are illustrated in FIGS. 9A-9C. FIG. 9A is a part of a color fundus image. FIG. 9B depicts the edges of FIG. 9A determined according to the energy function formulation of Equation 2. FIG. 9C depicts the edges of FIG. 9A, determined according to the energy function formulation of Equation 9. FIG. 9B shows smoother and stronger main edges, while FIG. 9C shows less smoothing and retains faint edges. Once the system 100 determines the smoothing and segmentation parameters that minimize two different energy functions, the system 100 combines information about the two different extracted attributes of the smoothed data and edge data. obtain. This combination can allow for the detection of lesions based on both enhancement and differentiation of the spatial and intensity characteristics of the two sets of attributes. In some applications, detection may be aided by prior knowledge of different categories of lesions that may exist, as well as spatial variation of attributes relative to other parts of the retinal image of the same patient or patient population. In some embodiments, both formulations can be used to detect faint blood vessels, such as those resulting from certain retinal abnormalities. Specifically, high-order logic can be used to combine strong and weak contrast edges to determine neovascularization (NVD and NVE), IRMA, and the presence of candidate microaneurysms. In addition, in some embodiments, the energy function formulation implemented by the system 100 to perform the segmentation and smoothing of step 704 (FIG. 7) is different terms within a single energy function formulation ( For example, it includes two or more different error metrics for evaluating smoothing and data fidelity terms).

  After system 100 smoothes and segments in step 704, system 100 identifies at least one retinal feature (such as a blood vessel) in the smoothed and segmented image (step 706). 10A-10C are unprocessed retinal fundus images (FIG. 10A), processed images that have been smoothed and segmented by step 704 (FIG. 10B), and enlarged views of portions of the image of FIG. 10C). Several techniques may be used to identify retinal features in the smoothed and segmented images of FIGS. 10B and 10C. A preferred retinal feature identification technique is illustrated in FIGS. 11A-11C. FIG. 11A depicts a location in the processed image of FIG. 10C where the edge value exceeds the threshold. In a preferred embodiment, the threshold used is determined adaptively for the location by determining the average edge value in the local neighborhood of each location (which can be the same neighborhood used for smoothing and segmentation). The FIG. 11A provides a first approximation of the boundaries of the retinal features, but includes several unrelated boundaries (such as boundaries 1102 and 1104). The system 100 is configured to apply an “open” form technique to the image of FIG. 11A that removes or reduces extraneous boundaries, resulting in the image of FIG. 11B. An example of such a technique is http://cmm.com, which was last accessed on October 12, 2010 and is incorporated herein by reference in its entirety. ensmp. fr / ~ serra / pdf / birth_of_mm. It is described in “The Birth of Mathematical Morphology” by Matheron et al. available at pdf. The system 100 can then use the enhanced boundary of FIG. 11B to fill a portion of the image corresponding to the retinal features as shown in FIG. 11C.

  The system 100 may also be configured to determine at least one path of identified retinal features (eg, blood vessels) at step 706. The path can be an approximate centerline of the blood vessel or another representation of the geometric shape of the retinal feature. 12A-12D illustrate a preferred path detection technique. FIG. 12A is a color fundus image and FIG. 12B is an enlarged view of a portion of FIG. 12A corresponding to box 1202 after FIG. 12A has been smoothed and segmented (according to step 704). FIG. 12C is an enlarged view of a portion of FIG. 12B corresponding to box 1204 after the image of FIG. 12B has received the vessel identification described above with reference to FIG. FIG. 12D illustrates the result of applying the “skeletonization” technique to the image of FIG. 12C. Skeletalization techniques are generally known and are used in image processing to identify the structural framework or “skeleton” of an image. Many suitable skeletal techniques are described in the literature. See, for example, “Skeleton-based hierarchical shape segmentation,” Proc. By Reniers et al., Which is incorporated herein by reference in its entirety. of the IEEE Int. Conf. on Shape Modeling and Applications, pp. 179-188 (2007).

  However, as illustrated in FIG. 12D, skeletal techniques applied to blood vessels and other retinal features can result in side branches (such as side branches 1208 and 1210) arising from the main vascular pathway. In some embodiments, the system 100 removes these side branches from the skeletonized image before performing additional processing steps.

  FIGS. 13A-13C provide additional illustrations of operations and intermediate results that may be generated at step 706 when the system 100 identifies blood vessels in the smoothed and segmented image. FIG. 13A depicts a smoothed and segmented image, FIG. 13B depicts a blood vessel boundary determined by edge values, and FIG. 13C skeletonizes the image of FIG. 13B to find an approximate centerline. The path of the blood vessel obtained by doing is illustrated. In some embodiments, the blood vessel boundary may be determined by intensity values in addition to or instead of edge values.

  In step 708, the system 100 analyzes data, edges, and other attributes associated with the identified retinal features in the smoothed and segmented image to identify retinal abnormalities. System 100 may be configured with hardware or software to evaluate smoothed and segmented images for the presence of different features that identify different anomalies. Here we discuss some features and anomaly identification techniques.

  FIG. 8 is a flow diagram of an anomaly detection technique 800, according to an embodiment. Technique 800 is well suited for identifying, for example, venous caliber abnormalities (VCAB) and beaded veins, among other abnormalities. In step 802, the system 100 determines a path associated with the identified retinal feature. This step may be performed as described above with reference to step 706 of FIG. For example, the path can be obtained by skeletonizing a smoothed and segmented image of a blood vessel to determine an approximate centerline.

In step 804, the system 100 determines the path variation determined in step 802. To do this, the system 100
The angle of orientation of the path at different locations along the path,
It can be configured to detect one or more different properties of the path (and the retinal features it represents), including the diameter of the blood vessels at different locations along the path. Spatial variations of these properties along the blood vessel can be used as an indicator of retinal abnormalities. Specifically, at step 806, the system 100 determines whether the variation at any location along the path exceeds a threshold. If yes, the system 100 proceeds to step 808 and identifies a retinal abnormality at the location. Several exemplary embodiments of technique 800 will now be discussed.

  14A-14C illustrate an embodiment in which the system 100 determines the variation in the slope of the identified retinal feature path in step 804. FIG. 14A depicts the identified vessel and its centerline 1402 (as discussed above with reference to FIG. 13). FIG. 14B is a graph of the absolute value of the slope of the centerline 1402 along its length (ie, at a plurality of consecutive locations). The discrete distance between adjacent pixels discretizes the slope at any point along the path of FIG. 14B (here 0 or

Note that) FIG. 14C is a graph of the four slope moving window averages drawn in FIG. 14B, where each moving window average is a different pixel window width (here 5 (blue), 6 (red), 7 (green)). , And 8 (black)). Performing a moving window average of the slope of the centerline 1402 is one way to filter the slope results, and any other filtering action or combination of filtering actions may be applied.

  FIG. 14C shows a peak in the region of box 1404 corresponding to the sustained slope change of FIG. 14A in the region of box 1406. The system 100 may include a threshold against which the filtered slope of FIG. 14C is compared, and an anomaly is identified if the filtered slope (eg, path variation) exceeds the threshold (step 808). .

  FIGS. 15A-15C illustrate an embodiment in which the system 100 determines the variation in the angle of orientation of the identified retinal features in step 804. FIG. 15A depicts the identified vessel and its centerline 1502, and FIG. 15B is a graph of the angle of orientation of the centerline 1502 along its length (ie, at a plurality of consecutive locations). Note that the discrete angle between adjacent pixels discretizes the angle at any point along the path in FIG. 15B (here -45, 0, or 45 degrees). FIG. 15C is a graph of the four angle moving window averages drawn in FIG. 15B, where each moving window average has a different pixel window width (here 5 (blue), 6 (red), 7 (green)). , And 8 (black)). FIG. 15C shows a peak in the region of box 1504 that corresponds to the kinking of FIG. 15A in the region of box 1506. As discussed above with respect to FIGS. 14A-14C, if the filtered angle exceeds a threshold, system 100 may identify an anomaly (step 808).

  FIGS. 16A-16C illustrate an embodiment in which the system 100 determines the variation in the radius of the retinal feature at step 804. FIG. 16A depicts the identified vessel and its centerline 1602, and FIG. 16B is a graph of the radius of the vessel along the length of the centerline 1602 (ie, at a plurality of consecutive locations). FIG. 16C uses a color change to show the radius of the blood vessel at the corresponding point along the centerline 1602, and the red in the region of box 1604, corresponding to the stenosis of FIG. 16A in the region of box 1606. Indicates stenosis. As discussed above with respect to FIGS. 14A-14C, when the radius decreases below a certain radius (or decreases below a nominal or a specified percentage of the average radius), the system 100 determines that the stenosis has exceeded a threshold. An anomaly may be determined and identified (step 808). In some embodiments, the system 100 detects the significant variation in the radius of the blood vessel along its length, for example normalizing the radius by its local standard deviation and analyzing the normalized radius. Can identify the beaded vein anomaly in step 808.

  In some embodiments, characteristics other than tilt, angle, and radius are detected and analyzed by the system 100. For example, the system 100 may be configured for PRH detection and classification to identify such parameters such as aspect ratio, covered retinal area, lesion size, and inner and outer textures. Can do. In other examples, the system 100 may be configured for IRMAS detection and classification to identify characteristics such as shape, area covered by candidate IRMA vessel contours, and whether the contours are closed. In some embodiments, vessel boundaries may be analyzed separately to identify vessel characteristics. For example, a rapid change in the edge value at the blood vessel boundary may indicate an abnormality.

  In some embodiments, the system 100 can be configured to determine whether a bifurcation is present at a location identified as part of a blood vessel. In response to the determination that the change in blood vessels has exceeded a threshold (eg, as discussed above with respect to step 806 of FIG. 8), or at any other stage in the analysis of the retinal image, the system 100 A decision can be made. In some retinal imaging applications, the bifurcation location is associated with the diagnosis of retinal pathology. In other retinal imaging applications, the bifurcation is desirably filtered out or otherwise ignored during anomaly detection (eg, when it may be misinterpreted by the system 100 as a VCAB or beaded vein). Is done. Any of several bifurcation detection techniques can be used to identify vessel bifurcations in smoothed and segmented images. Examples of known techniques are described by Zana et al., “A multi-modal registration algorithm of eye funding images using detection detection and Hough transform E., incorporated herein by reference in its entirety. Med. Imaging, v. 18, n. 5, pp. 419-428 (1999).

  In some embodiments, bifurcation identification is followed by a separate analysis of each of the two “branches” for retinal abnormalities. Figures 17A-17F illustrate such an embodiment. FIG. 17A depicts a smoothed and segmented fundus image, and FIG. 17B is an enlarged view of a portion of the image of FIG. 17A indicated by box 1702. FIG. 17B clearly illustrates the vessel bifurcation. The upper branch of the bifurcation is illustrated in FIG. 17C, and the orientation angle of the upper branch is depicted in FIG. 17D. The sustained angular change is indicated by box 1704 and corresponds to a retinal abnormality indicated by box 1706. The lower branch of the bifurcation is illustrated in box 1708 of FIG. 17E, and the orientation angle of the lower branch is depicted in FIG. 17F. The plot of FIG. 17F does not show a sustained angular change, and therefore retinal abnormalities are not detected in the lower branch.

  The systems and methods described herein are usefully applied to non-vascular retinal features and abnormalities such as hard and fluffy vitiligo. For example, fluffy vitiligo is fluffy on the retina caused by nerve injury and can be identified by the disclosed smoothing, segmentation, and texture analysis techniques.

  Optionally, system 100 can include one or more pre-processing that can be applied to the retinal image prior to the anomaly detection techniques described herein (such as techniques 700 and 800 of FIGS. 7 and 8, respectively). Can be configured with techniques. Examples of these pre-processing techniques are described by Walter et al., “Segmentation of color Funds images of the human retina: detection of the optical disc and each of which is incorporated herein by reference in its entirety. vascular molecular morphological techniques, "in Crespo et al, eds. Lecture Notes in Computer Science, v. 2199, pp. 282-287 (2001); Walter et al., "A contribution of image processing to the diagnosis of the undenosed ef- ferences in the world." Med. Imaging, v. 21, pp. 1326-1243 (2002); and Walter et al., "Automatic detection of microaneurysms in color fundus images," Med. Image Analysis, v. 11, pp. 555-566 (2007), including contrast enhancement, alignment, and other morphological techniques.

  In addition, by applying one or more robust sequential detection algorithms, point spikes in intermediate data sets (eg, tilt, orientation angle, radius, etc.) can be selectively removed. One such algorithm is described in “Dual sensor failure identification using analytic redundancy,” J. Pat. Guidance and Control, v. 2, n. 3, pp. 2213-2220 (1979).

  As discussed above, the image analysis techniques described herein can identify image attributes such as texture. In some retinal imaging applications, texture analysis improves detection of abnormalities, particularly fine or high spatial frequency abnormalities. Edges provide a kind of measure of “higher frequency” features, and others can be used. Conventional algorithms for texture analysis typically require that the area to be analyzed be large enough to include a rectangular sub-region of relatively uniform sub-texture. However, in retinal imaging, the abnormal structure is small, inhomogeneous and irregular in shape. Such shape analysis requires an improved algorithm suitable for these anomalies.

  Matrix edge onion peel (MEOP) methods can be used to identify retinal abnormalities based on their texture. In some embodiments where the texture region is large enough, a texture wavelet analysis algorithm may be used, but may be combined with a MEOP algorithm for small size texture regions. This methodology is incorporated by reference herein in its entirety by Desai et al., “Noise Adaptable Matrix Edge Ignition Immersion of the Heterogeneous Nation of Heterogeneous Energy and Sr. 1386-1389. An energy function approach can be used for simultaneous smoothing and segmentation. The methodology includes two features: a matrix edge field, and adaptive weighting of the measurement to the smoothing process model. The matrix edge function adaptively and implicitly modulates the shape, size, and orientation near the smoothing over different regions of the texture. Thus, this provides directional information about textures that are not available with the more traditional scalar edge field-based approach. Adaptive measurement weighting changes the weighting between measurements at each pixel.

  In some embodiments, non-parametric methods for identifying retinal abnormalities may be used. These methods may be based on combining level set methods, multi-resolution wavelet analysis, and non-parametric estimation of the density function of wavelet coefficients from decomposition. In addition, the multi-resolution of elongate and irregularly shaped nuclei to deal with small size textures, where the largest inscribed rectangular window may not contain a sufficient number of pixels for multi-resolution analysis We propose an adjustable window opening method that enables analysis. In some exemplary embodiments, an adjustable window opening approach combined with a non-parametric density model provides a better classification for cases where parametric density modeling of wavelet coefficients may not be applicable.

  Such a method also allows multiscale qualitative monitoring of retinal images over time at multiple spatiotemporal resolutions. Statistical multi-resolution wavelet texture analysis has been shown to be effective when combined with a parametric statistical model, such as the generalized Gaussian density (GGD), used to represent wavelet coefficients within the detailed subbands. However, parametric statistical multi-resolution wavelet analysis as already implemented has the following limitations. 1) Requires the user to manually select a sufficiently sized rectangle and texture homogeneous region to allow texture analysis. 2) Assuming that the coefficient distribution is symmetric, single mode, and unbiased, this may not be the case for some textures. As described above, in some applications, the matrix edge onion peel algorithm is a small size that exhibits a “onion layer” texture change (ie, a texture characteristic that changes as a function of the radius from the center of the structure). Can be used for irregularly shaped structures.

  In some embodiments, an algorithm is used to automatically segment the retinal features to reduce the number of coefficients available from multi-resolution decomposition of small irregularly shaped (ie, non-rectangular) regions. To maximize, an adjustable window opening method can be used. These steps allow for automatic analysis of images using multiple retinal features, eliminating the need for a human to manually select a window to perform texture analysis. Finally, non-parametric statistical analysis can be applied to cases where parametric GGD models are not applicable. This analysis can provide better performance than the parametric model when the latter is not applicable.

  Wavelet-based texture models, coefficient extraction, PDF, and adaptive window creation for texture dissimilarity estimation, density models such as generalized Gaussian and symmetric alpha stability, and KLD estimators such as Ahmad-Linand and Loftsgaarden-Quesenbergry Several additional image processing techniques are suitable for use with the retinal imaging systems and methods disclosed herein.

  In some embodiments, more than one of the techniques described herein may be used in combination, eg, in parallel, sequentially, or a support vector machine or stochastic It can be fused using non-linear classifiers such as methods. By using multiple techniques for each retinal abnormality, accuracy can be improved without significant loss of speed. In addition, any of the techniques described herein can be used to diabetic retinopathy or diabetic macular according to the classification scale of early treatment studies of diabetic retinopathy commonly used in the art. Useful for determining the level of edema.

  The present invention may be embodied in other specific forms without departing from its spirit or essential characteristics. The foregoing embodiments are therefore considered to be illustrative in all aspects rather than limiting of the invention.

Claims (32)

A computerized method for detecting retinal abnormalities,
Receiving a retina image by a computer;
By the computer, the method comprising: smoothing and partitioning the image, to the smoothing and partitioning, interdependent der is, furthermore,
Reducing the values of the first and second energy functions, wherein the first energy function uses a first error metric and the second energy function is the first error measurement. Using a second error metric different from the reference;
Combining information on the first and second sets of smoothing and partitioning parameters obtained by reducing the values of the first and second energy functions, respectively.
Including , and
By the computer, identifying at least one retinal features in the smoothed and segmented image,
Analyzing, by the computer, at least one retinal feature identified in the smoothed and segmented image to identify retinal abnormalities. The method of claim 1, wherein the at least one retinal feature comprises a blood vessel. Analyzing the identified at least one blood vessel by the computer ;
Determining , by the computer, at least one path associated with the at least one blood vessel in the image;
Determining , by the computer, a variation in at least one of an angle of orientation of the path at a location along the path and a diameter of the at least one blood vessel at the location along the path;
3. The method of claim 2, comprising identifying , by the computer, retinal abnormalities at the location when the variation exceeds a threshold. The segmenting and smoothing further includes determining edge field strength at a plurality of locations in the image by the computer , wherein the at least one path is based at least in part on the edge field strength. 4. The method of claim 3, wherein the method is determined. 4. The method of claim 3 , wherein determining the at least one path by the computer includes determining at least one centerline associated with the at least one blood vessel in the image by the computer . 4. The method of claim 3, wherein determining the orientation angle variation by the computer comprises determining a change in orientation angle between a plurality of consecutive locations by the computer . 4. The method of claim 3, further comprising filtering a plurality of variations by the computer. The method of claim 7, wherein filtering the plurality of variations by the computer comprises applying a moving window average by the computer . The variation, when more than at least one threshold value of a plurality of fluctuations exerted on the filter, exceeds the threshold value, The method of claim 7. The method according to claim 1, wherein the abnormality is an abnormality of a vein inner diameter. The method of claim 1, wherein the abnormality is a new blood vessel distribution. Determining if a blood vessel bifurcation is present at the location by the computer if the variation at a location exceeds a threshold;
The method of claim 3, further comprising: filtering out the variation by the computer when a vascular bifurcation is present at the location. The smoothing and segmenting further includes adaptively adjusting, by the computer, at least one of shapes and orientations that define neighborhoods associated with multiple locations in the image. Item 2. The method according to Item 1. By the computer, adjusting at least one adaptively among shape and orientation of the vicinity, by the computer, including reducing the value of the energy function associated with the error metric, in claim 13 The method described. The method of claim 1, wherein analyzing the identified at least one retinal feature comprises analyzing a texture of the image. A system for detecting retinal abnormalities,
A processor, the processor comprising:
Receiving an image of the retina;
The method comprising: smoothing and partitioning the image, to the smoothing and partitioning, interdependent der is, furthermore,
Reducing the values of the first and second energy functions, wherein the first energy function uses a first error metric and the second energy function is the first error measurement. Using a second error metric different from the reference;
Combining information on the first and second sets of smoothing and partitioning parameters obtained by reducing the values of the first and second energy functions, respectively.
Including , and
Identifying at least one retinal features in the smoothed and segmented image,
Analyzing at least one retinal feature identified in the smoothed and segmented image to identify retinal abnormalities. The system of claim 16 , wherein the at least one retinal feature comprises a blood vessel. Analyzing the identified at least one blood vessel comprises:
Determining at least one path associated with the at least one blood vessel in the image;
Determining an angle of orientation of the path at a location along the path and a variation in at least one of the diameters of the at least one blood vessel at the location along the path;
18. The system of claim 17 , comprising identifying a retinal abnormality at the location if the variation exceeds a threshold. The segmenting and smoothing further includes determining edge field strengths at a plurality of locations in the image, wherein the at least one path is determined based at least in part on the edge field strengths. The system of claim 17 . The system of claim 18 , wherein determining the at least one path includes determining at least one centerline associated with the at least one blood vessel in the image. 19. The system of claim 18 , wherein determining the orientation angle variation comprises determining a change in orientation angle between a plurality of consecutive locations. The system of claim 18 , wherein the processor is further configured to filter a plurality of variations. 23. The system of claim 22 , wherein filtering the plurality of variations includes applying a moving window average. The variation, when more than at least one threshold value of a plurality of fluctuations exerted on the filter, exceeds the threshold value, the system according to claim 22. The system according to claim 16 , wherein the abnormality is an abnormality of a vein inner diameter. The system of claim 16 , wherein the abnormality is a new blood vessel distribution. The processor is
Determining whether a vascular bifurcation is present at the location when the variation at a location exceeds a threshold;
If the vessel bifurcation is present in the location, by the computer, the is further configured to perform the be removed by filtering the variation system of claim 18. To the smoothing and partitioning, further by a computer, comprising adaptively adjusting at least one of specified shape and orient the neighborhood associated with a plurality of locations in the image, according to claim 16 The system described in. 29. The system of claim 28 , wherein adaptively adjusting at least one of the neighboring shapes and orientations includes reducing an energy function value associated with an error metric. The system of claim 16 , wherein analyzing the identified at least one blood vessel comprises analyzing a texture of the image. A non-transitory computer readable medium storing instructions executable by a computer , wherein the instructions are executed by the computer ,
Receiving an image of the retina;
The method comprising: smoothing and partitioning the image, to the smoothing and partitioning, interdependent der is, furthermore,
Reducing the values of the first and second energy functions, wherein the first energy function uses a first error metric and the second energy function is the first error measurement. Using a second error metric different from the reference;
Combining information on the first and second sets of smoothing and partitioning parameters obtained by reducing the values of the first and second energy functions, respectively.
Including , and
Identifying at least one retinal features in the smoothed and segmented image,
A non-transitory computer-readable medium that causes the computer to perform a method comprising: analyzing at least one retinal feature identified in the smoothed and segmented image to identify retinal abnormalities. 32. The non-transitory computer readable medium of claim 31 , wherein the at least one retinal feature comprises a blood vessel.